[PATCH v10 00/38] v4l: subdev internal routing and streams

[PATCH v10 00/38] v4l: subdev internal routing and streams

[Tomi Valkeinen]

Hi,

This is v10 of the multiplexed streams series. v8 can be found from:

https://lore.kernel.org/all/20211005085750.138151-1-tomi.valkeinen@ideasonboard.com/

I have pushed my work branch to:

git://git.kernel.org/pub/scm/linux/kernel/git/tomba/linux.git multistream/work-v10

which contains the patches in this series, along with subdev drivers
using multiplexed streams.

I have also pushed v4l-utils changes to:

https://github.com/tomba/v4l-utils.git streams-review-v1

Changes to v9:

- Add V4L2_SUBDEV_CAP_MPLEXED flag
- Use standard kmalloc and kmemdup for routes
- Allow NULL as pad/stream param for v4l2_state_find_opposite_end
- Add for_each_active_route
- Use _BITUL() in uapi header
- Rearrange struct v4l2_subdev_routing members to align on 64 bit
- Renames:
	sd->state -> sd->active_state
	v4l2_state_find_opposite_end -> v4l2_subdev_routing_find_opposite_end
	v4l2_state_get_opposite_stream_format -> v4l2_subdev_state_get_opposite_stream_format
	v4l2_routing_simple_verify -> v4l2_subdev_routing_validate_1_to_1
	v4l2_subdev_validate_and_lock_state -> v4l2_subdev_lock_and_return_state
- Doc & comment updates

 Tomi

Jacopo Mondi (3):
  media: entity: Add iterator helper for entity pads
  media: Documentation: Add GS_ROUTING documentation
  media: subdev: Add for_each_active_route() macro

Laurent Pinchart (4):
  media: entity: Add has_route entity operation
  media: entity: Add media_entity_has_route() function
  media: entity: Use routing information during graph traversal
  media: subdev: Add [GS]_ROUTING subdev ioctls and operations

Sakari Ailus (13):
  media: entity: Use pad as a starting point for graph walk
  media: entity: Use pads instead of entities in the media graph walk
    stack
  media: entity: Walk the graph based on pads
  media: mc: Start walk from a specific pad in use count calculation
  media: entity: Move the pipeline from entity to pads
  media: entity: Use pad as the starting point for a pipeline
  media: entity: Skip link validation for pads to which there is no
    route
  media: entity: Add an iterator helper for connected pads
  media: entity: Add only connected pads to the pipeline
  media: entity: Add debug information in graph walk route check
  media: Add bus type to frame descriptors
  media: Add CSI-2 bus configuration to frame descriptors
  media: Add stream to frame descriptor

Tomi Valkeinen (18):
  media: subdev: rename subdev-state alloc & free
  media: subdev: add active state to struct v4l2_subdev
  media: subdev: pass also the active state to subdevs from ioctls
  media: subdev: add subdev state locking
  media: subdev: Add v4l2_subdev_lock_and_return_state()
  media: Documentation: add documentation about subdev state
  media: subdev: increase V4L2_FRAME_DESC_ENTRY_MAX to 8
  media: add V4L2_SUBDEV_FL_MULTIPLEXED
  media: add V4L2_SUBDEV_CAP_MPLEXED
  media: subdev: add v4l2_subdev_has_route()
  media: subdev: add v4l2_subdev_set_routing helper()
  media: Documentation: add multiplexed streams documentation
  media: subdev: add stream based configuration
  media: subdev: use streams in v4l2_subdev_link_validate()
  media: subdev: add "opposite" stream helper funcs
  media: subdev: add v4l2_subdev_get_fmt() helper function
  media: subdev: add v4l2_subdev_set_routing_with_fmt() helper
  media: subdev: add v4l2_subdev_routing_validate_1_to_1 helper

 Documentation/driver-api/media/mc-core.rst    |  18 +-
 .../driver-api/media/v4l2-subdev.rst          |  36 +
 .../userspace-api/media/v4l/dev-subdev.rst    | 167 ++++
 .../userspace-api/media/v4l/user-func.rst     |   1 +
 .../v4l/vidioc-subdev-enum-frame-interval.rst |   5 +-
 .../v4l/vidioc-subdev-enum-frame-size.rst     |   5 +-
 .../v4l/vidioc-subdev-enum-mbus-code.rst      |   5 +-
 .../media/v4l/vidioc-subdev-g-crop.rst        |   5 +-
 .../media/v4l/vidioc-subdev-g-fmt.rst         |   5 +-
 .../v4l/vidioc-subdev-g-frame-interval.rst    |   5 +-
 .../media/v4l/vidioc-subdev-g-routing.rst     | 150 ++++
 .../media/v4l/vidioc-subdev-g-selection.rst   |   5 +-
 drivers/media/mc/mc-device.c                  |  13 +-
 drivers/media/mc/mc-entity.c                  | 257 +++---
 drivers/media/pci/intel/ipu3/ipu3-cio2-main.c |   6 +-
 .../media/platform/exynos4-is/fimc-capture.c  |   8 +-
 .../platform/exynos4-is/fimc-isp-video.c      |   8 +-
 drivers/media/platform/exynos4-is/fimc-isp.c  |   2 +-
 drivers/media/platform/exynos4-is/fimc-lite.c |  10 +-
 drivers/media/platform/exynos4-is/media-dev.c |  20 +-
 drivers/media/platform/omap3isp/isp.c         |   2 +-
 drivers/media/platform/omap3isp/ispvideo.c    |  25 +-
 drivers/media/platform/omap3isp/ispvideo.h    |   2 +-
 .../media/platform/qcom/camss/camss-video.c   |   6 +-
 drivers/media/platform/rcar-vin/rcar-core.c   |  16 +-
 drivers/media/platform/rcar-vin/rcar-dma.c    |   8 +-
 drivers/media/platform/rcar-vin/rcar-v4l2.c   |   5 +-
 .../platform/rockchip/rkisp1/rkisp1-capture.c |   6 +-
 .../media/platform/s3c-camif/camif-capture.c  |   6 +-
 drivers/media/platform/stm32/stm32-dcmi.c     |   6 +-
 .../platform/sunxi/sun4i-csi/sun4i_dma.c      |   6 +-
 .../platform/sunxi/sun6i-csi/sun6i_video.c    |   6 +-
 drivers/media/platform/ti-vpe/cal-video.c     |   6 +-
 drivers/media/platform/vsp1/vsp1_entity.c     |   6 +-
 drivers/media/platform/vsp1/vsp1_video.c      |  18 +-
 drivers/media/platform/xilinx/xilinx-dma.c    |  20 +-
 drivers/media/platform/xilinx/xilinx-dma.h    |   2 +-
 .../media/test-drivers/vimc/vimc-capture.c    |   6 +-
 drivers/media/usb/au0828/au0828-core.c        |   8 +-
 drivers/media/v4l2-core/v4l2-ioctl.c          |  25 +-
 drivers/media/v4l2-core/v4l2-mc.c             |  43 +-
 drivers/media/v4l2-core/v4l2-subdev.c         | 764 +++++++++++++++++-
 drivers/staging/media/imx/imx-media-utils.c   |   8 +-
 drivers/staging/media/ipu3/ipu3-v4l2.c        |   6 +-
 drivers/staging/media/omap4iss/iss.c          |   2 +-
 drivers/staging/media/omap4iss/iss_video.c    |  40 +-
 drivers/staging/media/omap4iss/iss_video.h    |   2 +-
 drivers/staging/media/tegra-video/tegra210.c  |   6 +-
 drivers/staging/media/tegra-video/vi.c        |   6 +-
 include/media/media-entity.h                  | 143 +++-
 include/media/v4l2-subdev.h                   | 394 ++++++++-
 include/uapi/linux/v4l2-subdev.h              |  88 +-
 52 files changed, 2048 insertions(+), 370 deletions(-)
 create mode 100644 Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst

-- 
2.25.1

[PATCH v10 01/38] media: subdev: rename subdev-state alloc & free

[Tomi Valkeinen]

v4l2_subdev_alloc_state() and v4l2_subdev_free_state() are not supposed
to be used by the drivers. However, we do have a few drivers that use
those at the moment, so we need to expose these functions for the time
being.

Prefix the functions with __ to mark the functions as internal.

At the same time, rename them to v4l2_subdev_state_alloc and
v4l2_subdev_state_free to match the style used for other functions like
video_device_alloc() and media_request_alloc().

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
---
 drivers/media/platform/rcar-vin/rcar-v4l2.c |  4 ++--
 drivers/media/platform/vsp1/vsp1_entity.c   |  4 ++--
 drivers/media/v4l2-core/v4l2-subdev.c       | 12 ++++++------
 drivers/staging/media/tegra-video/vi.c      |  4 ++--
 include/media/v4l2-subdev.h                 | 10 +++++-----
 5 files changed, 17 insertions(+), 17 deletions(-)

diff --git a/drivers/media/platform/rcar-vin/rcar-v4l2.c b/drivers/media/platform/rcar-vin/rcar-v4l2.c
index 0d141155f0e3..ba1d16ab1651 100644
--- a/drivers/media/platform/rcar-vin/rcar-v4l2.c
+++ b/drivers/media/platform/rcar-vin/rcar-v4l2.c
@@ -252,7 +252,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
 	u32 width, height;
 	int ret;
 
-	sd_state = v4l2_subdev_alloc_state(sd);
+	sd_state = __v4l2_subdev_state_alloc(sd);
 	if (IS_ERR(sd_state))
 		return PTR_ERR(sd_state);
 
@@ -288,7 +288,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
 
 	rvin_format_align(vin, pix);
 done:
-	v4l2_subdev_free_state(sd_state);
+	__v4l2_subdev_state_free(sd_state);
 
 	return ret;
 }
diff --git a/drivers/media/platform/vsp1/vsp1_entity.c b/drivers/media/platform/vsp1/vsp1_entity.c
index 823c15facd1b..869cadc1468d 100644
--- a/drivers/media/platform/vsp1/vsp1_entity.c
+++ b/drivers/media/platform/vsp1/vsp1_entity.c
@@ -675,7 +675,7 @@ int vsp1_entity_init(struct vsp1_device *vsp1, struct vsp1_entity *entity,
 	 * Allocate the pad configuration to store formats and selection
 	 * rectangles.
 	 */
-	entity->config = v4l2_subdev_alloc_state(&entity->subdev);
+	entity->config = __v4l2_subdev_state_alloc(&entity->subdev);
 	if (IS_ERR(entity->config)) {
 		media_entity_cleanup(&entity->subdev.entity);
 		return PTR_ERR(entity->config);
@@ -690,6 +690,6 @@ void vsp1_entity_destroy(struct vsp1_entity *entity)
 		entity->ops->destroy(entity);
 	if (entity->subdev.ctrl_handler)
 		v4l2_ctrl_handler_free(entity->subdev.ctrl_handler);
-	v4l2_subdev_free_state(entity->config);
+	__v4l2_subdev_state_free(entity->config);
 	media_entity_cleanup(&entity->subdev.entity);
 }
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 5d27a27cc2f2..fe49c86a9b02 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -28,7 +28,7 @@ static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
 {
 	struct v4l2_subdev_state *state;
 
-	state = v4l2_subdev_alloc_state(sd);
+	state = __v4l2_subdev_state_alloc(sd);
 	if (IS_ERR(state))
 		return PTR_ERR(state);
 
@@ -39,7 +39,7 @@ static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
 
 static void subdev_fh_free(struct v4l2_subdev_fh *fh)
 {
-	v4l2_subdev_free_state(fh->state);
+	__v4l2_subdev_state_free(fh->state);
 	fh->state = NULL;
 }
 
@@ -870,7 +870,7 @@ int v4l2_subdev_link_validate(struct media_link *link)
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
 
-struct v4l2_subdev_state *v4l2_subdev_alloc_state(struct v4l2_subdev *sd)
+struct v4l2_subdev_state *__v4l2_subdev_state_alloc(struct v4l2_subdev *sd)
 {
 	struct v4l2_subdev_state *state;
 	int ret;
@@ -903,9 +903,9 @@ struct v4l2_subdev_state *v4l2_subdev_alloc_state(struct v4l2_subdev *sd)
 
 	return ERR_PTR(ret);
 }
-EXPORT_SYMBOL_GPL(v4l2_subdev_alloc_state);
+EXPORT_SYMBOL_GPL(__v4l2_subdev_state_alloc);
 
-void v4l2_subdev_free_state(struct v4l2_subdev_state *state)
+void __v4l2_subdev_state_free(struct v4l2_subdev_state *state)
 {
 	if (!state)
 		return;
@@ -913,7 +913,7 @@ void v4l2_subdev_free_state(struct v4l2_subdev_state *state)
 	kvfree(state->pads);
 	kfree(state);
 }
-EXPORT_SYMBOL_GPL(v4l2_subdev_free_state);
+EXPORT_SYMBOL_GPL(__v4l2_subdev_state_free);
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
 
diff --git a/drivers/staging/media/tegra-video/vi.c b/drivers/staging/media/tegra-video/vi.c
index d321790b07d9..66b9ce160472 100644
--- a/drivers/staging/media/tegra-video/vi.c
+++ b/drivers/staging/media/tegra-video/vi.c
@@ -507,7 +507,7 @@ static int __tegra_channel_try_format(struct tegra_vi_channel *chan,
 	if (!subdev)
 		return -ENODEV;
 
-	sd_state = v4l2_subdev_alloc_state(subdev);
+	sd_state = __v4l2_subdev_state_alloc(subdev);
 	if (IS_ERR(sd_state))
 		return PTR_ERR(sd_state);
 	/*
@@ -558,7 +558,7 @@ static int __tegra_channel_try_format(struct tegra_vi_channel *chan,
 	v4l2_fill_pix_format(pix, &fmt.format);
 	tegra_channel_fmt_align(chan, pix, fmtinfo->bpp);
 
-	v4l2_subdev_free_state(sd_state);
+	__v4l2_subdev_state_free(sd_state);
 
 	return 0;
 }
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 95ec18c2f49c..e52bf508c75b 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1135,20 +1135,20 @@ int v4l2_subdev_link_validate_default(struct v4l2_subdev *sd,
 int v4l2_subdev_link_validate(struct media_link *link);
 
 /**
- * v4l2_subdev_alloc_state - allocate v4l2_subdev_state
+ * __v4l2_subdev_state_alloc - allocate v4l2_subdev_state
  *
  * @sd: pointer to &struct v4l2_subdev for which the state is being allocated.
  *
- * Must call v4l2_subdev_free_state() when state is no longer needed.
+ * Must call __v4l2_subdev_state_free() when state is no longer needed.
  */
-struct v4l2_subdev_state *v4l2_subdev_alloc_state(struct v4l2_subdev *sd);
+struct v4l2_subdev_state *__v4l2_subdev_state_alloc(struct v4l2_subdev *sd);
 
 /**
- * v4l2_subdev_free_state - free a v4l2_subdev_state
+ * __v4l2_subdev_state_free - free a v4l2_subdev_state
  *
  * @state: v4l2_subdev_state to be freed.
  */
-void v4l2_subdev_free_state(struct v4l2_subdev_state *state);
+void __v4l2_subdev_state_free(struct v4l2_subdev_state *state);
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
 
-- 
2.25.1

[PATCH v10 02/38] media: subdev: add active state to struct v4l2_subdev

[Tomi Valkeinen]

Add a new 'active_state' field to struct v4l2_subdev to which we can
store the active state of a subdev. This will place the subdev
configuration into a known place, allowing us to use the state directly
from the v4l2 framework, thus simplifying the drivers.

Also add functions v4l2_subdev_init_finalize() and
v4l2_subdev_cleanup(), which will allocate and free the active state.
The functions are named in a generic way so that they can be also used
for other subdev initialization work.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 21 +++++++++++
 include/media/v4l2-subdev.h           | 54 +++++++++++++++++++++++++++
 2 files changed, 75 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index fe49c86a9b02..de160140d63b 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -943,3 +943,24 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
 	v4l2_subdev_notify(sd, V4L2_DEVICE_NOTIFY_EVENT, (void *)ev);
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
+
+int v4l2_subdev_init_finalize(struct v4l2_subdev *sd)
+{
+	struct v4l2_subdev_state *state;
+
+	state = __v4l2_subdev_state_alloc(sd);
+	if (IS_ERR(state))
+		return PTR_ERR(state);
+
+	sd->active_state = state;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_init_finalize);
+
+void v4l2_subdev_cleanup(struct v4l2_subdev *sd)
+{
+	__v4l2_subdev_state_free(sd->active_state);
+	sd->active_state = NULL;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_cleanup);
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index e52bf508c75b..87c3126b2754 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -645,6 +645,9 @@ struct v4l2_subdev_ir_ops {
  * This structure only needs to be passed to the pad op if the 'which' field
  * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
  * %V4L2_SUBDEV_FORMAT_ACTIVE it is safe to pass %NULL.
+ *
+ * Note: This struct is also used in active state, and the try_ prefix is
+ * historical and to be removed.
  */
 struct v4l2_subdev_pad_config {
 	struct v4l2_mbus_framefmt try_fmt;
@@ -898,6 +901,8 @@ struct v4l2_subdev_platform_data {
  * @subdev_notifier: A sub-device notifier implicitly registered for the sub-
  *		     device using v4l2_async_register_subdev_sensor().
  * @pdata: common part of subdevice platform data
+ * @active_state: active state for the subdev (NULL for subdevs tracking the
+ *                state internally)
  *
  * Each instance of a subdev driver should create this struct, either
  * stand-alone or embedded in a larger struct.
@@ -929,6 +934,19 @@ struct v4l2_subdev {
 	struct v4l2_async_notifier *notifier;
 	struct v4l2_async_notifier *subdev_notifier;
 	struct v4l2_subdev_platform_data *pdata;
+
+	/*
+	 * The fields below are private, and should only be accessed via
+	 * appropriate functions.
+	 */
+
+	/*
+	 * TODO: state should most likely be changed from a pointer to an
+	 * embedded field. For the time being it's kept as a pointer to more
+	 * easily catch uses of state in the cases where the driver doesn't
+	 * support it.
+	 */
+	struct v4l2_subdev_state *active_state;
 };
 
 
@@ -1217,4 +1235,40 @@ extern const struct v4l2_subdev_ops v4l2_subdev_call_wrappers;
 void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
 			      const struct v4l2_event *ev);
 
+/**
+ * v4l2_subdev_init_finalize() - Finalize the initialization of the subdevice
+ * @sd: The subdev
+ *
+ * This finalizes the initialization of the subdev, including allocation of
+ * the active state for the subdev.
+ *
+ * This must be called by the subdev drivers that use the centralized active
+ * state, after the subdev struct has been initialized and
+ * media_entity_pads_init() has been called.
+ *
+ * Must call v4l2_subdev_cleanup() when the subdev is being removed.
+ */
+int v4l2_subdev_init_finalize(struct v4l2_subdev *sd);
+
+/**
+ * v4l2_subdev_cleanup() - Release the resources needed by the subdevice
+ * @sd: The subdevice
+ *
+ * This will release the resources allocated in v4l2_subdev_init_finalize.
+ */
+void v4l2_subdev_cleanup(struct v4l2_subdev *sd);
+
+/**
+ * v4l2_subdev_get_active_state() - Return the active subdev state for subdevice
+ * @sd: The subdevice
+ *
+ * Return the active state for the subdevice, or NULL if the subdev does not
+ * support active state.
+ */
+static inline struct v4l2_subdev_state *
+v4l2_subdev_get_active_state(struct v4l2_subdev *sd)
+{
+	return sd->active_state;
+}
+
 #endif
-- 
2.25.1

[PATCH v10 03/38] media: subdev: pass also the active state to subdevs from ioctls

[Tomi Valkeinen]

At the moment when a subdev op is called, the TRY subdev state
(subdev_fh->state) is passed as a parameter even for the ACTIVE case, or
alternatively a NULL can be passed for ACTIVE case. This used to make
sense, as the ACTIVE state was handled internally by the subdev drivers.

We now have a state for the ACTIVE case in a standard place, and can
pass that also to the drivers. This patch changes the subdev ioctls to
either pass the TRY or ACTIVE state to the subdev.

Unfortunately many drivers call ops from other subdevs, and implicitly
pass NULL as the state, so this is just a partial solution. A coccinelle
spatch could perhaps be created which fixes the drivers' subdev calls.

For all current upstream drivers this doesn't matter, as they do not
expect to get a valid state for ACTIVE case. But future drivers which
support multiplexed streaming and routing will depend on getting a state
for both active and try cases.

For new drivers we can mandate that the pipelines where the drivers are
used need to pass the state properly, or preferably, not call such
subdev ops at all.

However, if an existing subdev driver is changed to support multiplexed
streams, the driver has to consider cases where its ops will be called
with NULL state. The problem can easily be solved by using the
v4l2_subdev_validate_and_lock_state() helper, introduced in a follow up
patch.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 73 +++++++++++++++++++++++----
 1 file changed, 63 insertions(+), 10 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index de160140d63b..3289875d9ec1 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -353,6 +353,53 @@ const struct v4l2_subdev_ops v4l2_subdev_call_wrappers = {
 EXPORT_SYMBOL(v4l2_subdev_call_wrappers);
 
 #if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
+
+static struct v4l2_subdev_state *
+subdev_ioctl_get_state(struct v4l2_subdev *sd, struct v4l2_subdev_fh *subdev_fh,
+		       unsigned int cmd, void *arg)
+{
+	u32 which;
+
+	switch (cmd) {
+	default:
+		return NULL;
+
+	case VIDIOC_SUBDEV_G_FMT:
+	case VIDIOC_SUBDEV_S_FMT: {
+		which = ((struct v4l2_subdev_format *)arg)->which;
+		break;
+	}
+	case VIDIOC_SUBDEV_G_CROP:
+	case VIDIOC_SUBDEV_S_CROP: {
+		which = ((struct v4l2_subdev_crop *)arg)->which;
+		break;
+	}
+	case VIDIOC_SUBDEV_ENUM_MBUS_CODE: {
+		which = ((struct v4l2_subdev_mbus_code_enum *)arg)->which;
+		break;
+	}
+	case VIDIOC_SUBDEV_ENUM_FRAME_SIZE: {
+		which = ((struct v4l2_subdev_frame_size_enum *)arg)->which;
+		break;
+	}
+
+	case VIDIOC_SUBDEV_ENUM_FRAME_INTERVAL: {
+		which = ((struct v4l2_subdev_frame_interval_enum *)arg)->which;
+		break;
+	}
+
+	case VIDIOC_SUBDEV_G_SELECTION:
+	case VIDIOC_SUBDEV_S_SELECTION: {
+		which = ((struct v4l2_subdev_selection *)arg)->which;
+		break;
+	}
+	}
+
+	return which == V4L2_SUBDEV_FORMAT_TRY ?
+			     subdev_fh->state :
+			     v4l2_subdev_get_active_state(sd);
+}
+
 static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 {
 	struct video_device *vdev = video_devdata(file);
@@ -360,8 +407,11 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 	struct v4l2_fh *vfh = file->private_data;
 	struct v4l2_subdev_fh *subdev_fh = to_v4l2_subdev_fh(vfh);
 	bool ro_subdev = test_bit(V4L2_FL_SUBDEV_RO_DEVNODE, &vdev->flags);
+	struct v4l2_subdev_state *state;
 	int rval;
 
+	state = subdev_ioctl_get_state(sd, subdev_fh, cmd, arg);
+
 	switch (cmd) {
 	case VIDIOC_SUBDEV_QUERYCAP: {
 		struct v4l2_subdev_capability *cap = arg;
@@ -484,7 +534,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 
 		memset(format->reserved, 0, sizeof(format->reserved));
 		memset(format->format.reserved, 0, sizeof(format->format.reserved));
-		return v4l2_subdev_call(sd, pad, get_fmt, subdev_fh->state, format);
+		return v4l2_subdev_call(sd, pad, get_fmt, state, format);
 	}
 
 	case VIDIOC_SUBDEV_S_FMT: {
@@ -495,7 +545,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 
 		memset(format->reserved, 0, sizeof(format->reserved));
 		memset(format->format.reserved, 0, sizeof(format->format.reserved));
-		return v4l2_subdev_call(sd, pad, set_fmt, subdev_fh->state, format);
+		return v4l2_subdev_call(sd, pad, set_fmt, state, format);
 	}
 
 	case VIDIOC_SUBDEV_G_CROP: {
@@ -509,7 +559,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 		sel.target = V4L2_SEL_TGT_CROP;
 
 		rval = v4l2_subdev_call(
-			sd, pad, get_selection, subdev_fh->state, &sel);
+			sd, pad, get_selection, state, &sel);
 
 		crop->rect = sel.r;
 
@@ -531,7 +581,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 		sel.r = crop->rect;
 
 		rval = v4l2_subdev_call(
-			sd, pad, set_selection, subdev_fh->state, &sel);
+			sd, pad, set_selection, state, &sel);
 
 		crop->rect = sel.r;
 
@@ -542,7 +592,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 		struct v4l2_subdev_mbus_code_enum *code = arg;
 
 		memset(code->reserved, 0, sizeof(code->reserved));
-		return v4l2_subdev_call(sd, pad, enum_mbus_code, subdev_fh->state,
+		return v4l2_subdev_call(sd, pad, enum_mbus_code, state,
 					code);
 	}
 
@@ -550,7 +600,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 		struct v4l2_subdev_frame_size_enum *fse = arg;
 
 		memset(fse->reserved, 0, sizeof(fse->reserved));
-		return v4l2_subdev_call(sd, pad, enum_frame_size, subdev_fh->state,
+		return v4l2_subdev_call(sd, pad, enum_frame_size, state,
 					fse);
 	}
 
@@ -575,7 +625,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 		struct v4l2_subdev_frame_interval_enum *fie = arg;
 
 		memset(fie->reserved, 0, sizeof(fie->reserved));
-		return v4l2_subdev_call(sd, pad, enum_frame_interval, subdev_fh->state,
+		return v4l2_subdev_call(sd, pad, enum_frame_interval, state,
 					fie);
 	}
 
@@ -584,7 +634,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 
 		memset(sel->reserved, 0, sizeof(sel->reserved));
 		return v4l2_subdev_call(
-			sd, pad, get_selection, subdev_fh->state, sel);
+			sd, pad, get_selection, state, sel);
 	}
 
 	case VIDIOC_SUBDEV_S_SELECTION: {
@@ -595,7 +645,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 
 		memset(sel->reserved, 0, sizeof(sel->reserved));
 		return v4l2_subdev_call(
-			sd, pad, set_selection, subdev_fh->state, sel);
+			sd, pad, set_selection, state, sel);
 	}
 
 	case VIDIOC_G_EDID: {
@@ -829,10 +879,13 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
 	if (is_media_entity_v4l2_subdev(pad->entity)) {
 		struct v4l2_subdev *sd =
 			media_entity_to_v4l2_subdev(pad->entity);
+		struct v4l2_subdev_state *state;
+
+		state = v4l2_subdev_get_active_state(sd);
 
 		fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
 		fmt->pad = pad->index;
-		return v4l2_subdev_call(sd, pad, get_fmt, NULL, fmt);
+		return v4l2_subdev_call(sd, pad, get_fmt, state, fmt);
 	}
 
 	WARN(pad->entity->function != MEDIA_ENT_F_IO_V4L,
-- 
2.25.1

[PATCH v10 04/38] media: subdev: add subdev state locking

[Tomi Valkeinen]

The V4L2 subdevs have managed without centralized locking for the state
(previously pad_config), as the TRY state is supposedly safe (although I
believe two TRY ioctls for the same fd would race), and the ACTIVE
state, and its locking, is managed by the drivers internally.

We now have ACTIVE state in a centralized position, and need locking.
Strictly speaking the locking is only needed for new drivers that use
the new state, as the current drivers continue behaving as they used to.

Add a mutex to the struct v4l2_subdev_state, along with a few helper
functions for locking/unlocking.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/platform/rcar-vin/rcar-v4l2.c |  3 +-
 drivers/media/platform/vsp1/vsp1_entity.c   |  4 +-
 drivers/media/v4l2-core/v4l2-subdev.c       | 38 +++++++++++++---
 drivers/staging/media/tegra-video/vi.c      |  4 +-
 include/media/v4l2-subdev.h                 | 50 ++++++++++++++++++++-
 5 files changed, 89 insertions(+), 10 deletions(-)

diff --git a/drivers/media/platform/rcar-vin/rcar-v4l2.c b/drivers/media/platform/rcar-vin/rcar-v4l2.c
index ba1d16ab1651..e6bd94d63e4f 100644
--- a/drivers/media/platform/rcar-vin/rcar-v4l2.c
+++ b/drivers/media/platform/rcar-vin/rcar-v4l2.c
@@ -244,6 +244,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
 {
 	struct v4l2_subdev *sd = vin_to_source(vin);
 	struct v4l2_subdev_state *sd_state;
+	static struct lock_class_key key;
 	struct v4l2_subdev_format format = {
 		.which = which,
 		.pad = vin->parallel.source_pad,
@@ -252,7 +253,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
 	u32 width, height;
 	int ret;
 
-	sd_state = __v4l2_subdev_state_alloc(sd);
+	sd_state = __v4l2_subdev_state_alloc(sd, "rvin:state->lock", &key);
 	if (IS_ERR(sd_state))
 		return PTR_ERR(sd_state);
 
diff --git a/drivers/media/platform/vsp1/vsp1_entity.c b/drivers/media/platform/vsp1/vsp1_entity.c
index 869cadc1468d..e607c3ae2520 100644
--- a/drivers/media/platform/vsp1/vsp1_entity.c
+++ b/drivers/media/platform/vsp1/vsp1_entity.c
@@ -613,6 +613,7 @@ int vsp1_entity_init(struct vsp1_device *vsp1, struct vsp1_entity *entity,
 		     const char *name, unsigned int num_pads,
 		     const struct v4l2_subdev_ops *ops, u32 function)
 {
+	static struct lock_class_key key;
 	struct v4l2_subdev *subdev;
 	unsigned int i;
 	int ret;
@@ -675,7 +676,8 @@ int vsp1_entity_init(struct vsp1_device *vsp1, struct vsp1_entity *entity,
 	 * Allocate the pad configuration to store formats and selection
 	 * rectangles.
 	 */
-	entity->config = __v4l2_subdev_state_alloc(&entity->subdev);
+	entity->config = __v4l2_subdev_state_alloc(&entity->subdev,
+						   "vsp1:config->lock", &key);
 	if (IS_ERR(entity->config)) {
 		media_entity_cleanup(&entity->subdev.entity);
 		return PTR_ERR(entity->config);
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 3289875d9ec1..2053fe1cd67d 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -27,8 +27,9 @@
 static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
 {
 	struct v4l2_subdev_state *state;
+	static struct lock_class_key key;
 
-	state = __v4l2_subdev_state_alloc(sd);
+	state = __v4l2_subdev_state_alloc(sd, "fh->state->lock", &key);
 	if (IS_ERR(state))
 		return PTR_ERR(state);
 
@@ -923,7 +924,9 @@ int v4l2_subdev_link_validate(struct media_link *link)
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
 
-struct v4l2_subdev_state *__v4l2_subdev_state_alloc(struct v4l2_subdev *sd)
+struct v4l2_subdev_state *
+__v4l2_subdev_state_alloc(struct v4l2_subdev *sd, const char *lock_name,
+			  struct lock_class_key *lock_key)
 {
 	struct v4l2_subdev_state *state;
 	int ret;
@@ -932,6 +935,8 @@ struct v4l2_subdev_state *__v4l2_subdev_state_alloc(struct v4l2_subdev *sd)
 	if (!state)
 		return ERR_PTR(-ENOMEM);
 
+	__mutex_init(&state->lock, lock_name, lock_key);
+
 	if (sd->entity.num_pads) {
 		state->pads = kvmalloc_array(sd->entity.num_pads,
 					     sizeof(*state->pads),
@@ -963,6 +968,8 @@ void __v4l2_subdev_state_free(struct v4l2_subdev_state *state)
 	if (!state)
 		return;
 
+	mutex_destroy(&state->lock);
+
 	kvfree(state->pads);
 	kfree(state);
 }
@@ -997,11 +1004,12 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
 
-int v4l2_subdev_init_finalize(struct v4l2_subdev *sd)
+int __v4l2_subdev_init_finalize(struct v4l2_subdev *sd, const char *name,
+				struct lock_class_key *key)
 {
 	struct v4l2_subdev_state *state;
 
-	state = __v4l2_subdev_state_alloc(sd);
+	state = __v4l2_subdev_state_alloc(sd, name, key);
 	if (IS_ERR(state))
 		return PTR_ERR(state);
 
@@ -1009,7 +1017,7 @@ int v4l2_subdev_init_finalize(struct v4l2_subdev *sd)
 
 	return 0;
 }
-EXPORT_SYMBOL_GPL(v4l2_subdev_init_finalize);
+EXPORT_SYMBOL_GPL(__v4l2_subdev_init_finalize);
 
 void v4l2_subdev_cleanup(struct v4l2_subdev *sd)
 {
@@ -1017,3 +1025,23 @@ void v4l2_subdev_cleanup(struct v4l2_subdev *sd)
 	sd->active_state = NULL;
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_cleanup);
+
+struct v4l2_subdev_state *v4l2_subdev_lock_active_state(struct v4l2_subdev *sd)
+{
+	mutex_lock(&sd->active_state->lock);
+
+	return sd->active_state;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_lock_active_state);
+
+void v4l2_subdev_lock_state(struct v4l2_subdev_state *state)
+{
+	mutex_lock(&state->lock);
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_lock_state);
+
+void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state)
+{
+	mutex_unlock(&state->lock);
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_unlock_state);
diff --git a/drivers/staging/media/tegra-video/vi.c b/drivers/staging/media/tegra-video/vi.c
index 66b9ce160472..3516735f0cad 100644
--- a/drivers/staging/media/tegra-video/vi.c
+++ b/drivers/staging/media/tegra-video/vi.c
@@ -491,6 +491,7 @@ static int __tegra_channel_try_format(struct tegra_vi_channel *chan,
 				      struct v4l2_pix_format *pix)
 {
 	const struct tegra_video_format *fmtinfo;
+	static struct lock_class_key key;
 	struct v4l2_subdev *subdev;
 	struct v4l2_subdev_format fmt;
 	struct v4l2_subdev_state *sd_state;
@@ -507,7 +508,8 @@ static int __tegra_channel_try_format(struct tegra_vi_channel *chan,
 	if (!subdev)
 		return -ENODEV;
 
-	sd_state = __v4l2_subdev_state_alloc(subdev);
+	sd_state = __v4l2_subdev_state_alloc(subdev, "tegra:state->lock",
+					     &key);
 	if (IS_ERR(sd_state))
 		return PTR_ERR(sd_state);
 	/*
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 87c3126b2754..1810dde9c7fc 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -658,6 +658,7 @@ struct v4l2_subdev_pad_config {
 /**
  * struct v4l2_subdev_state - Used for storing subdev state information.
  *
+ * @lock: mutex for the state
  * @pads: &struct v4l2_subdev_pad_config array
  *
  * This structure only needs to be passed to the pad op if the 'which' field
@@ -665,6 +666,8 @@ struct v4l2_subdev_pad_config {
  * %V4L2_SUBDEV_FORMAT_ACTIVE it is safe to pass %NULL.
  */
 struct v4l2_subdev_state {
+	/* lock for the struct v4l2_subdev_state fields */
+	struct mutex lock;
 	struct v4l2_subdev_pad_config *pads;
 };
 
@@ -1156,10 +1159,14 @@ int v4l2_subdev_link_validate(struct media_link *link);
  * __v4l2_subdev_state_alloc - allocate v4l2_subdev_state
  *
  * @sd: pointer to &struct v4l2_subdev for which the state is being allocated.
+ * @lock_name: name of the state lock
+ * @key: lock_class_key for the lock
  *
  * Must call __v4l2_subdev_state_free() when state is no longer needed.
  */
-struct v4l2_subdev_state *__v4l2_subdev_state_alloc(struct v4l2_subdev *sd);
+struct v4l2_subdev_state *__v4l2_subdev_state_alloc(struct v4l2_subdev *sd,
+						    const char *lock_name,
+						    struct lock_class_key *key);
 
 /**
  * __v4l2_subdev_state_free - free a v4l2_subdev_state
@@ -1248,7 +1255,16 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
  *
  * Must call v4l2_subdev_cleanup() when the subdev is being removed.
  */
-int v4l2_subdev_init_finalize(struct v4l2_subdev *sd);
+#define v4l2_subdev_init_finalize(sd)                                          \
+	({                                                                     \
+		static struct lock_class_key __key;                            \
+		const char *name = KBUILD_BASENAME                             \
+			":" __stringify(__LINE__) ":subdev->state->lock";      \
+		__v4l2_subdev_init_finalize(sd, name, &__key);                 \
+	})
+
+int __v4l2_subdev_init_finalize(struct v4l2_subdev *sd, const char *name,
+				struct lock_class_key *key);
 
 /**
  * v4l2_subdev_cleanup() - Release the resources needed by the subdevice
@@ -1271,4 +1287,34 @@ v4l2_subdev_get_active_state(struct v4l2_subdev *sd)
 	return sd->active_state;
 }
 
+/**
+ * v4l2_subdev_lock_active_state() - Lock and return the active subdev state for
+ *				     subdevice
+ * @sd: The subdevice
+ *
+ * Return the locked active state for the subdevice, or NULL if the subdev
+ * does not support active state.
+ *
+ * Must be unlocked with v4l2_subdev_unlock_state() after use.
+ */
+struct v4l2_subdev_state *v4l2_subdev_lock_active_state(struct v4l2_subdev *sd);
+
+/**
+ * v4l2_subdev_lock_state() - Lock the subdev state
+ * @state: The subdevice state
+ *
+ * Lock the given subdev state.
+ *
+ * Must be unlocked with v4l2_subdev_unlock_state() after use.
+ */
+void v4l2_subdev_lock_state(struct v4l2_subdev_state *state);
+
+/**
+ * v4l2_subdev_unlock_state() - Unlock the subdev state
+ * @state: The subdevice state
+ *
+ * Unlock the given subdev state.
+ */
+void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state);
+
 #endif
-- 
2.25.1

[PATCH v10 05/38] media: subdev: Add v4l2_subdev_lock_and_return_state()

[Tomi Valkeinen]

All suitable subdev ops are now passed either the TRY or the ACTIVE
state by the v4l2 core. However, other subdev drivers can still call the
ops passing NULL as the state, implying the active case.

For all current upstream drivers this doesn't matter, as they do not
expect to get a valid state for ACTIVE case. But future drivers which
support multiplexed streaming and routing will depend on getting a state
for both active and try cases.

For new drivers we can mandate that the pipelines where the drivers are
used need to pass the state properly, or preferably, not call such
subdev ops at all.

However, if an existing subdev driver is changed to support multiplexed
streams, the driver has to consider cases where its ops will be called
with NULL state. The problem can easily be solved by using the
v4l2_subdev_lock_and_return_state() helper, introduced here.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 include/media/v4l2-subdev.h | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 1810dde9c7fc..873bbe0686e3 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1317,4 +1317,35 @@ void v4l2_subdev_lock_state(struct v4l2_subdev_state *state);
  */
 void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state);
 
+/**
+ * v4l2_subdev_lock_and_return_state() - Gets locked TRY or ACTIVE subdev state
+ * @sd: subdevice
+ * @state: subdevice state as passed to the subdev op
+ *
+ * Due to legacy reasons, when subdev drivers call ops in other subdevs they use
+ * NULL as the state parameter, as subdevs always used to have their active
+ * state stored privately.
+ *
+ * However, newer state-aware subdev drivers, which store their active state in
+ * a common place, subdev->active_state, expect to always get a proper state as
+ * a parameter.
+ *
+ * These state-aware drivers can use v4l2_subdev_lock_and_return_state() instead
+ * of v4l2_subdev_lock_state(). v4l2_subdev_lock_and_return_state() solves the
+ * issue by using subdev->state in case the passed state is NULL.
+ *
+ * This is a temporary helper function, and should be removed when we can ensure
+ * that all drivers pass proper state when calling other subdevs.
+ */
+static inline struct v4l2_subdev_state *
+v4l2_subdev_lock_and_return_state(struct v4l2_subdev *sd,
+				  struct v4l2_subdev_state *state)
+{
+	state = state ? state : sd->active_state;
+
+	v4l2_subdev_lock_state(state);
+
+	return state;
+}
+
 #endif
-- 
2.25.1

[PATCH v10 06/38] media: Documentation: add documentation about subdev state

[Tomi Valkeinen]

Add documentation about centrally managed subdev state.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 .../driver-api/media/v4l2-subdev.rst          | 28 +++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/Documentation/driver-api/media/v4l2-subdev.rst b/Documentation/driver-api/media/v4l2-subdev.rst
index 7736da077fb8..db012729f8e3 100644
--- a/Documentation/driver-api/media/v4l2-subdev.rst
+++ b/Documentation/driver-api/media/v4l2-subdev.rst
@@ -518,6 +518,34 @@ The :c:func:`v4l2_i2c_new_subdev` function will call
 :c:type:`i2c_board_info` structure using the ``client_type`` and the
 ``addr`` to fill it.
 
+Centrally managed subdev active state
+-------------------------------------
+
+Traditionally V4L2 subdev drivers maintained internal state for the active
+configuration for the subdev. This is often implemented as an array of struct
+v4l2_mbus_framefmt, one entry for each pad.
+
+In addition to the active configuration, each subdev filehandle has an array of
+struct v4l2_subdev_pad_config, managed by V4L2 core, which contains the TRY
+configuration.
+
+To simplify the subdev drivers the V4L2 subdev API now optionally supports a
+centrally managed active configuration. A subdev driver must use
+v4l2_subdev_init_finalize() to initialize the active state between calls to
+media_entity_pads_init() and v4l2_*_register_subdev(), and must call
+v4l2_subdev_cleanup() to free the state.
+
+The active state must be locked before access, and that can be done with
+v4l2_subdev_lock_state() or v4l2_subdev_lock_active_state().
+
+The V4L2 core will pass either the TRY or ACTIVE state to various subdev ops.
+Unfortunately not all the subdev drivers comply with this yet, and may pass
+NULL for the ACTIVE case. This is only a problem for subdev drivers which use
+the centrally managed active state and are used in media pipelines with older
+subdev drivers. In these cases the called subdev ops must also handle the NULL
+case. This can be easily managed by the use of
+v4l2_subdev_validate_and_lock_state() helper.
+
 V4L2 sub-device functions and data structures
 ---------------------------------------------
 
-- 
2.25.1

[PATCH v10 07/38] media: entity: Use pad as a starting point for graph walk

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

With the upcoming use of the recently added has_route() media entity op,
all the pads in an entity will no longer be considered interconnected.
This has an effect where the media graph is traversed: the starting pad
does make a difference.

Prepare for this change by using pad instead of the entity as an argument
for the graph walk operations. The actual graph traversal algorithm change
is in further patches.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 Documentation/driver-api/media/mc-core.rst    |  2 +-
 drivers/media/mc/mc-entity.c                  | 17 ++++++++---------
 drivers/media/platform/exynos4-is/media-dev.c |  4 ++--
 drivers/media/platform/omap3isp/ispvideo.c    |  2 +-
 drivers/media/platform/vsp1/vsp1_video.c      |  2 +-
 drivers/media/platform/xilinx/xilinx-dma.c    |  2 +-
 drivers/media/v4l2-core/v4l2-mc.c             |  6 +++---
 drivers/staging/media/omap4iss/iss_video.c    |  4 ++--
 include/media/media-entity.h                  | 10 ++++------
 9 files changed, 23 insertions(+), 26 deletions(-)

diff --git a/Documentation/driver-api/media/mc-core.rst b/Documentation/driver-api/media/mc-core.rst
index 57b5bbba944e..ba0aee982124 100644
--- a/Documentation/driver-api/media/mc-core.rst
+++ b/Documentation/driver-api/media/mc-core.rst
@@ -167,7 +167,7 @@ Drivers initiate a graph traversal by calling
 :c:func:`media_graph_walk_start()`
 
 The graph structure, provided by the caller, is initialized to start graph
-traversal at the given entity.
+traversal at the given pad in an entity.
 
 Drivers can then retrieve the next entity by calling
 :c:func:`media_graph_walk_next()`
diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index f40f41977142..47cdaa301838 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -291,17 +291,16 @@ void media_graph_walk_cleanup(struct media_graph *graph)
 }
 EXPORT_SYMBOL_GPL(media_graph_walk_cleanup);
 
-void media_graph_walk_start(struct media_graph *graph,
-			    struct media_entity *entity)
+void media_graph_walk_start(struct media_graph *graph, struct media_pad *pad)
 {
 	media_entity_enum_zero(&graph->ent_enum);
-	media_entity_enum_set(&graph->ent_enum, entity);
+	media_entity_enum_set(&graph->ent_enum, pad->entity);
 
 	graph->top = 0;
 	graph->stack[graph->top].entity = NULL;
-	stack_push(graph, entity);
-	dev_dbg(entity->graph_obj.mdev->dev,
-		"begin graph walk at '%s'\n", entity->name);
+	stack_push(graph, pad->entity);
+	dev_dbg(pad->graph_obj.mdev->dev,
+		"begin graph walk at '%s':%u\n", pad->entity->name, pad->index);
 }
 EXPORT_SYMBOL_GPL(media_graph_walk_start);
 
@@ -420,7 +419,7 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 			goto error_graph_walk_start;
 	}
 
-	media_graph_walk_start(&pipe->graph, entity);
+	media_graph_walk_start(&pipe->graph, entity->pads);
 
 	while ((entity = media_graph_walk_next(graph))) {
 		DECLARE_BITMAP(active, MEDIA_ENTITY_MAX_PADS);
@@ -504,7 +503,7 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 	 * Link validation on graph failed. We revert what we did and
 	 * return the error.
 	 */
-	media_graph_walk_start(graph, entity_err);
+	media_graph_walk_start(graph, entity_err->pads);
 
 	while ((entity_err = media_graph_walk_next(graph))) {
 		/* Sanity check for negative stream_count */
@@ -555,7 +554,7 @@ void __media_pipeline_stop(struct media_entity *entity)
 	if (WARN_ON(!pipe))
 		return;
 
-	media_graph_walk_start(graph, entity);
+	media_graph_walk_start(graph, entity->pads);
 
 	while ((entity = media_graph_walk_next(graph))) {
 		/* Sanity check for negative stream_count */
diff --git a/drivers/media/platform/exynos4-is/media-dev.c b/drivers/media/platform/exynos4-is/media-dev.c
index fa648721eaab..4ec7f7a6b0f6 100644
--- a/drivers/media/platform/exynos4-is/media-dev.c
+++ b/drivers/media/platform/exynos4-is/media-dev.c
@@ -1173,7 +1173,7 @@ static int __fimc_md_modify_pipelines(struct media_entity *entity, bool enable,
 	 * through active links. This is needed as we cannot power on/off the
 	 * subdevs in random order.
 	 */
-	media_graph_walk_start(graph, entity);
+	media_graph_walk_start(graph, entity->pads);
 
 	while ((entity = media_graph_walk_next(graph))) {
 		if (!is_media_entity_v4l2_video_device(entity))
@@ -1188,7 +1188,7 @@ static int __fimc_md_modify_pipelines(struct media_entity *entity, bool enable,
 	return 0;
 
 err:
-	media_graph_walk_start(graph, entity_err);
+	media_graph_walk_start(graph, entity_err->pads);
 
 	while ((entity_err = media_graph_walk_next(graph))) {
 		if (!is_media_entity_v4l2_video_device(entity_err))
diff --git a/drivers/media/platform/omap3isp/ispvideo.c b/drivers/media/platform/omap3isp/ispvideo.c
index 8811d6dd4ee7..3c1485d59404 100644
--- a/drivers/media/platform/omap3isp/ispvideo.c
+++ b/drivers/media/platform/omap3isp/ispvideo.c
@@ -234,7 +234,7 @@ static int isp_video_get_graph_data(struct isp_video *video,
 		return ret;
 	}
 
-	media_graph_walk_start(&graph, entity);
+	media_graph_walk_start(&graph, entity->pads);
 
 	while ((entity = media_graph_walk_next(&graph))) {
 		struct isp_video *__video;
diff --git a/drivers/media/platform/vsp1/vsp1_video.c b/drivers/media/platform/vsp1/vsp1_video.c
index 044eb5778820..61e4fbaba7b7 100644
--- a/drivers/media/platform/vsp1/vsp1_video.c
+++ b/drivers/media/platform/vsp1/vsp1_video.c
@@ -569,7 +569,7 @@ static int vsp1_video_pipeline_build(struct vsp1_pipeline *pipe,
 	if (ret)
 		return ret;
 
-	media_graph_walk_start(&graph, entity);
+	media_graph_walk_start(&graph, entity->pads);
 
 	while ((entity = media_graph_walk_next(&graph))) {
 		struct v4l2_subdev *subdev;
diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
index 338c3661d809..af0d55ab6c15 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.c
+++ b/drivers/media/platform/xilinx/xilinx-dma.c
@@ -189,7 +189,7 @@ static int xvip_pipeline_validate(struct xvip_pipeline *pipe,
 		return ret;
 	}
 
-	media_graph_walk_start(&graph, entity);
+	media_graph_walk_start(&graph, entity->pads);
 
 	while ((entity = media_graph_walk_next(&graph))) {
 		struct xvip_dma *dma;
diff --git a/drivers/media/v4l2-core/v4l2-mc.c b/drivers/media/v4l2-core/v4l2-mc.c
index b01474717dca..d215fe31b9a2 100644
--- a/drivers/media/v4l2-core/v4l2-mc.c
+++ b/drivers/media/v4l2-core/v4l2-mc.c
@@ -436,7 +436,7 @@ static int pipeline_pm_use_count(struct media_entity *entity,
 {
 	int use = 0;
 
-	media_graph_walk_start(graph, entity);
+	media_graph_walk_start(graph, entity->pads);
 
 	while ((entity = media_graph_walk_next(graph))) {
 		if (is_media_entity_v4l2_video_device(entity))
@@ -499,7 +499,7 @@ static int pipeline_pm_power(struct media_entity *entity, int change,
 	if (!change)
 		return 0;
 
-	media_graph_walk_start(graph, entity);
+	media_graph_walk_start(graph, entity->pads);
 
 	while (!ret && (entity = media_graph_walk_next(graph)))
 		if (is_media_entity_v4l2_subdev(entity))
@@ -508,7 +508,7 @@ static int pipeline_pm_power(struct media_entity *entity, int change,
 	if (!ret)
 		return ret;
 
-	media_graph_walk_start(graph, first);
+	media_graph_walk_start(graph, first->pads);
 
 	while ((first = media_graph_walk_next(graph))
 	       && first != entity)
diff --git a/drivers/staging/media/omap4iss/iss_video.c b/drivers/staging/media/omap4iss/iss_video.c
index d0da083deed5..760cd0ab1feb 100644
--- a/drivers/staging/media/omap4iss/iss_video.c
+++ b/drivers/staging/media/omap4iss/iss_video.c
@@ -217,7 +217,7 @@ iss_video_far_end(struct iss_video *video)
 		return NULL;
 	}
 
-	media_graph_walk_start(&graph, entity);
+	media_graph_walk_start(&graph, entity->pads);
 
 	while ((entity = media_graph_walk_next(&graph))) {
 		if (entity == &video->video.entity)
@@ -892,7 +892,7 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 		goto err_media_pipeline_start;
 
 	mutex_lock(&mdev->graph_mutex);
-	media_graph_walk_start(&graph, entity);
+	media_graph_walk_start(&graph, entity->pads);
 	while ((entity = media_graph_walk_next(&graph)))
 		media_entity_enum_set(&pipe->ent_enum, entity);
 	mutex_unlock(&mdev->graph_mutex);
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 09737b47881f..b9bfcf34eb0a 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -902,22 +902,20 @@ __must_check int media_graph_walk_init(
 void media_graph_walk_cleanup(struct media_graph *graph);
 
 /**
- * media_graph_walk_start - Start walking the media graph at a
- *	given entity
+ * media_graph_walk_start - Start walking the media graph at a given pad
  *
  * @graph: Media graph structure that will be used to walk the graph
- * @entity: Starting entity
+ * @pad: Starting pad
  *
  * Before using this function, media_graph_walk_init() must be
  * used to allocate resources used for walking the graph. This
  * function initializes the graph traversal structure to walk the
- * entities graph starting at the given entity. The traversal
+ * entities graph starting at the given pad. The traversal
  * structure must not be modified by the caller during graph
  * traversal. After the graph walk, the resources must be released
  * using media_graph_walk_cleanup().
  */
-void media_graph_walk_start(struct media_graph *graph,
-			    struct media_entity *entity);
+void media_graph_walk_start(struct media_graph *graph, struct media_pad *pad);
 
 /**
  * media_graph_walk_next - Get the next entity in the graph
-- 
2.25.1

[PATCH v10 08/38] media: entity: Use pads instead of entities in the media graph walk stack

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

Change the media graph walk stack structure to use media pads instead of
using media entities. In addition to the entity, the pad contains the
information which pad in the entity are being dealt with.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 55 ++++++++++++++++++------------------
 include/media/media-entity.h |  8 +++---
 2 files changed, 31 insertions(+), 32 deletions(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 47cdaa301838..e3a932ab9abd 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -228,40 +228,39 @@ EXPORT_SYMBOL_GPL(media_entity_pads_init);
  * Graph traversal
  */
 
-static struct media_entity *
-media_entity_other(struct media_entity *entity, struct media_link *link)
+static struct media_pad *
+media_pad_other(struct media_pad *pad, struct media_link *link)
 {
-	if (link->source->entity == entity)
-		return link->sink->entity;
+	if (link->source == pad)
+		return link->sink;
 	else
-		return link->source->entity;
+		return link->source;
 }
 
 /* push an entity to traversal stack */
-static void stack_push(struct media_graph *graph,
-		       struct media_entity *entity)
+static void stack_push(struct media_graph *graph, struct media_pad *pad)
 {
 	if (graph->top == MEDIA_ENTITY_ENUM_MAX_DEPTH - 1) {
 		WARN_ON(1);
 		return;
 	}
 	graph->top++;
-	graph->stack[graph->top].link = entity->links.next;
-	graph->stack[graph->top].entity = entity;
+	graph->stack[graph->top].link = pad->entity->links.next;
+	graph->stack[graph->top].pad = pad;
 }
 
-static struct media_entity *stack_pop(struct media_graph *graph)
+static struct media_pad *stack_pop(struct media_graph *graph)
 {
-	struct media_entity *entity;
+	struct media_pad *pad;
 
-	entity = graph->stack[graph->top].entity;
+	pad = graph->stack[graph->top].pad;
 	graph->top--;
 
-	return entity;
+	return pad;
 }
 
 #define link_top(en)	((en)->stack[(en)->top].link)
-#define stack_top(en)	((en)->stack[(en)->top].entity)
+#define stack_top(en)	((en)->stack[(en)->top].pad)
 
 /**
  * media_graph_walk_init - Allocate resources for graph walk
@@ -297,8 +296,8 @@ void media_graph_walk_start(struct media_graph *graph, struct media_pad *pad)
 	media_entity_enum_set(&graph->ent_enum, pad->entity);
 
 	graph->top = 0;
-	graph->stack[graph->top].entity = NULL;
-	stack_push(graph, pad->entity);
+	graph->stack[graph->top].pad = NULL;
+	stack_push(graph, pad);
 	dev_dbg(pad->graph_obj.mdev->dev,
 		"begin graph walk at '%s':%u\n", pad->entity->name, pad->index);
 }
@@ -306,16 +305,16 @@ EXPORT_SYMBOL_GPL(media_graph_walk_start);
 
 static void media_graph_walk_iter(struct media_graph *graph)
 {
-	struct media_entity *entity = stack_top(graph);
+	struct media_pad *pad = stack_top(graph);
 	struct media_link *link;
-	struct media_entity *next;
+	struct media_pad *next;
 
 	link = list_entry(link_top(graph), typeof(*link), list);
 
 	/* The link is not enabled so we do not follow. */
 	if (!(link->flags & MEDIA_LNK_FL_ENABLED)) {
 		link_top(graph) = link_top(graph)->next;
-		dev_dbg(entity->graph_obj.mdev->dev,
+		dev_dbg(pad->graph_obj.mdev->dev,
 			"walk: skipping disabled link '%s':%u -> '%s':%u\n",
 			link->source->entity->name, link->source->index,
 			link->sink->entity->name, link->sink->index);
@@ -323,23 +322,23 @@ static void media_graph_walk_iter(struct media_graph *graph)
 	}
 
 	/* Get the entity at the other end of the link. */
-	next = media_entity_other(entity, link);
+	next = media_pad_other(pad, link);
 
 	/* Has the entity already been visited? */
-	if (media_entity_enum_test_and_set(&graph->ent_enum, next)) {
+	if (media_entity_enum_test_and_set(&graph->ent_enum, next->entity)) {
 		link_top(graph) = link_top(graph)->next;
-		dev_dbg(entity->graph_obj.mdev->dev,
+		dev_dbg(pad->graph_obj.mdev->dev,
 			"walk: skipping entity '%s' (already seen)\n",
-			next->name);
+			next->entity->name);
 		return;
 	}
 
 	/* Push the new entity to stack and start over. */
 	link_top(graph) = link_top(graph)->next;
 	stack_push(graph, next);
-	dev_dbg(entity->graph_obj.mdev->dev, "walk: pushing '%s' on stack\n",
-		next->name);
-	lockdep_assert_held(&entity->graph_obj.mdev->graph_mutex);
+	dev_dbg(next->graph_obj.mdev->dev, "walk: pushing '%s':%u on stack\n",
+		next->entity->name, next->index);
+	lockdep_assert_held(&next->graph_obj.mdev->graph_mutex);
 }
 
 struct media_entity *media_graph_walk_next(struct media_graph *graph)
@@ -354,10 +353,10 @@ struct media_entity *media_graph_walk_next(struct media_graph *graph)
 	 * top of the stack until no more entities on the level can be
 	 * found.
 	 */
-	while (link_top(graph) != &stack_top(graph)->links)
+	while (link_top(graph) != &stack_top(graph)->entity->links)
 		media_graph_walk_iter(graph);
 
-	entity = stack_pop(graph);
+	entity = stack_pop(graph)->entity;
 	dev_dbg(entity->graph_obj.mdev->dev,
 		"walk: returning entity '%s'\n", entity->name);
 
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index b9bfcf34eb0a..5b55d6179e13 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -78,16 +78,16 @@ struct media_entity_enum {
  * struct media_graph - Media graph traversal state
  *
  * @stack:		Graph traversal stack; the stack contains information
- *			on the path the media entities to be walked and the
- *			links through which they were reached.
- * @stack.entity:	pointer to &struct media_entity at the graph.
+ *			on the media pads to be walked and the links through
+ *			which they were reached.
+ * @stack.pad:		pointer to &struct media_pad at the graph.
  * @stack.link:		pointer to &struct list_head.
  * @ent_enum:		Visited entities
  * @top:		The top of the stack
  */
 struct media_graph {
 	struct {
-		struct media_entity *entity;
+		struct media_pad *pad;
 		struct list_head *link;
 	} stack[MEDIA_ENTITY_ENUM_MAX_DEPTH];
 
-- 
2.25.1

[PATCH v10 09/38] media: entity: Walk the graph based on pads

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

Instead of iterating over graph entities during the walk, iterate the pads
through which the entity was first reached. This is required in order to
make the entity pipeline pad-based rather than entity based.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 Documentation/driver-api/media/mc-core.rst    |  7 ++-
 drivers/media/mc/mc-entity.c                  | 51 +++++++++++--------
 drivers/media/platform/exynos4-is/media-dev.c | 20 ++++----
 drivers/media/platform/omap3isp/ispvideo.c    | 17 ++++---
 drivers/media/platform/vsp1/vsp1_video.c      | 12 ++---
 drivers/media/platform/xilinx/xilinx-dma.c    | 12 ++---
 drivers/media/v4l2-core/v4l2-mc.c             | 24 ++++-----
 drivers/staging/media/omap4iss/iss_video.c    | 36 ++++++-------
 include/media/media-entity.h                  |  7 +--
 9 files changed, 101 insertions(+), 85 deletions(-)

diff --git a/Documentation/driver-api/media/mc-core.rst b/Documentation/driver-api/media/mc-core.rst
index ba0aee982124..8a13640bed56 100644
--- a/Documentation/driver-api/media/mc-core.rst
+++ b/Documentation/driver-api/media/mc-core.rst
@@ -169,8 +169,11 @@ Drivers initiate a graph traversal by calling
 The graph structure, provided by the caller, is initialized to start graph
 traversal at the given pad in an entity.
 
-Drivers can then retrieve the next entity by calling
-:c:func:`media_graph_walk_next()`
+Drivers can then retrieve the next pad by calling
+:c:func:`media_graph_walk_next()`. Only the pad through which the entity
+is first reached is returned. If the caller is interested in knowing which
+further pads would be connected, the :c:func:`media_entity_has_route()`
+function can be used for that.
 
 When the graph traversal is complete the function will return ``NULL``.
 
diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index e3a932ab9abd..06147f38ce2e 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -341,9 +341,9 @@ static void media_graph_walk_iter(struct media_graph *graph)
 	lockdep_assert_held(&next->graph_obj.mdev->graph_mutex);
 }
 
-struct media_entity *media_graph_walk_next(struct media_graph *graph)
+struct media_pad *media_graph_walk_next(struct media_graph *graph)
 {
-	struct media_entity *entity;
+	struct media_pad *pad;
 
 	if (stack_top(graph) == NULL)
 		return NULL;
@@ -356,11 +356,11 @@ struct media_entity *media_graph_walk_next(struct media_graph *graph)
 	while (link_top(graph) != &stack_top(graph)->entity->links)
 		media_graph_walk_iter(graph);
 
-	entity = stack_pop(graph)->entity;
-	dev_dbg(entity->graph_obj.mdev->dev,
-		"walk: returning entity '%s'\n", entity->name);
+	pad = stack_pop(graph);
+	dev_dbg(pad->graph_obj.mdev->dev,
+		"walk: returning pad '%s':%u\n", pad->entity->name, pad->index);
 
-	return entity;
+	return pad;
 }
 EXPORT_SYMBOL_GPL(media_graph_walk_next);
 
@@ -408,7 +408,8 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 {
 	struct media_device *mdev = entity->graph_obj.mdev;
 	struct media_graph *graph = &pipe->graph;
-	struct media_entity *entity_err = entity;
+	struct media_pad *pad = entity->pads;
+	struct media_pad *pad_err = pad;
 	struct media_link *link;
 	int ret;
 
@@ -418,9 +419,11 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 			goto error_graph_walk_start;
 	}
 
-	media_graph_walk_start(&pipe->graph, entity->pads);
+	media_graph_walk_start(&pipe->graph, pad);
+
+	while ((pad = media_graph_walk_next(graph))) {
+		struct media_entity *entity = pad->entity;
 
-	while ((entity = media_graph_walk_next(graph))) {
 		DECLARE_BITMAP(active, MEDIA_ENTITY_MAX_PADS);
 		DECLARE_BITMAP(has_no_links, MEDIA_ENTITY_MAX_PADS);
 
@@ -429,7 +432,7 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 		if (entity->pipe && entity->pipe != pipe) {
 			pr_err("Pipe active for %s. Can't start for %s\n",
 				entity->name,
-				entity_err->name);
+				pad_err->entity->name);
 			ret = -EBUSY;
 			goto error;
 		}
@@ -447,26 +450,27 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 		bitmap_fill(has_no_links, entity->num_pads);
 
 		list_for_each_entry(link, &entity->links, list) {
-			struct media_pad *pad = link->sink->entity == entity
-						? link->sink : link->source;
+			struct media_pad *other_pad =
+				link->sink->entity == entity ?
+				link->sink : link->source;
 
 			/* Mark that a pad is connected by a link. */
-			bitmap_clear(has_no_links, pad->index, 1);
+			bitmap_clear(has_no_links, other_pad->index, 1);
 
 			/*
 			 * Pads that either do not need to connect or
 			 * are connected through an enabled link are
 			 * fine.
 			 */
-			if (!(pad->flags & MEDIA_PAD_FL_MUST_CONNECT) ||
+			if (!(other_pad->flags & MEDIA_PAD_FL_MUST_CONNECT) ||
 			    link->flags & MEDIA_LNK_FL_ENABLED)
-				bitmap_set(active, pad->index, 1);
+				bitmap_set(active, other_pad->index, 1);
 
 			/*
 			 * Link validation will only take place for
 			 * sink ends of the link that are enabled.
 			 */
-			if (link->sink != pad ||
+			if (link->sink != other_pad ||
 			    !(link->flags & MEDIA_LNK_FL_ENABLED))
 				continue;
 
@@ -502,9 +506,11 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 	 * Link validation on graph failed. We revert what we did and
 	 * return the error.
 	 */
-	media_graph_walk_start(graph, entity_err->pads);
+	media_graph_walk_start(graph, pad_err);
+
+	while ((pad_err = media_graph_walk_next(graph))) {
+		struct media_entity *entity_err = pad_err->entity;
 
-	while ((entity_err = media_graph_walk_next(graph))) {
 		/* Sanity check for negative stream_count */
 		if (!WARN_ON_ONCE(entity_err->stream_count <= 0)) {
 			entity_err->stream_count--;
@@ -516,7 +522,7 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 		 * We haven't increased stream_count further than this
 		 * so we quit here.
 		 */
-		if (entity_err == entity)
+		if (pad_err == pad)
 			break;
 	}
 
@@ -543,8 +549,9 @@ EXPORT_SYMBOL_GPL(media_pipeline_start);
 
 void __media_pipeline_stop(struct media_entity *entity)
 {
-	struct media_graph *graph = &entity->pipe->graph;
 	struct media_pipeline *pipe = entity->pipe;
+	struct media_graph *graph = &pipe->graph;
+	struct media_pad *pad;
 
 	/*
 	 * If the following check fails, the driver has performed an
@@ -555,7 +562,9 @@ void __media_pipeline_stop(struct media_entity *entity)
 
 	media_graph_walk_start(graph, entity->pads);
 
-	while ((entity = media_graph_walk_next(graph))) {
+	while ((pad = media_graph_walk_next(graph))) {
+		struct media_entity *entity = pad->entity;
+
 		/* Sanity check for negative stream_count */
 		if (!WARN_ON_ONCE(entity->stream_count <= 0)) {
 			entity->stream_count--;
diff --git a/drivers/media/platform/exynos4-is/media-dev.c b/drivers/media/platform/exynos4-is/media-dev.c
index 4ec7f7a6b0f6..10429466a394 100644
--- a/drivers/media/platform/exynos4-is/media-dev.c
+++ b/drivers/media/platform/exynos4-is/media-dev.c
@@ -1164,7 +1164,7 @@ static int __fimc_md_modify_pipeline(struct media_entity *entity, bool enable)
 static int __fimc_md_modify_pipelines(struct media_entity *entity, bool enable,
 				      struct media_graph *graph)
 {
-	struct media_entity *entity_err = entity;
+	struct media_pad *pad, *pad_err = entity->pads;
 	int ret;
 
 	/*
@@ -1173,13 +1173,13 @@ static int __fimc_md_modify_pipelines(struct media_entity *entity, bool enable,
 	 * through active links. This is needed as we cannot power on/off the
 	 * subdevs in random order.
 	 */
-	media_graph_walk_start(graph, entity->pads);
+	media_graph_walk_start(graph, pad_err);
 
-	while ((entity = media_graph_walk_next(graph))) {
-		if (!is_media_entity_v4l2_video_device(entity))
+	while ((pad = media_graph_walk_next(graph))) {
+		if (!is_media_entity_v4l2_video_device(pad->entity))
 			continue;
 
-		ret  = __fimc_md_modify_pipeline(entity, enable);
+		ret  = __fimc_md_modify_pipeline(pad->entity, enable);
 
 		if (ret < 0)
 			goto err;
@@ -1188,15 +1188,15 @@ static int __fimc_md_modify_pipelines(struct media_entity *entity, bool enable,
 	return 0;
 
 err:
-	media_graph_walk_start(graph, entity_err->pads);
+	media_graph_walk_start(graph, pad_err);
 
-	while ((entity_err = media_graph_walk_next(graph))) {
-		if (!is_media_entity_v4l2_video_device(entity_err))
+	while ((pad_err = media_graph_walk_next(graph))) {
+		if (!is_media_entity_v4l2_video_device(pad_err->entity))
 			continue;
 
-		__fimc_md_modify_pipeline(entity_err, !enable);
+		__fimc_md_modify_pipeline(pad_err->entity, !enable);
 
-		if (entity_err == entity)
+		if (pad_err == pad)
 			break;
 	}
 
diff --git a/drivers/media/platform/omap3isp/ispvideo.c b/drivers/media/platform/omap3isp/ispvideo.c
index 3c1485d59404..5c1cbb1a9003 100644
--- a/drivers/media/platform/omap3isp/ispvideo.c
+++ b/drivers/media/platform/omap3isp/ispvideo.c
@@ -222,8 +222,8 @@ static int isp_video_get_graph_data(struct isp_video *video,
 				    struct isp_pipeline *pipe)
 {
 	struct media_graph graph;
-	struct media_entity *entity = &video->video.entity;
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_pad *pad = video->video.entity.pads;
+	struct media_device *mdev = video->video.entity.graph_obj.mdev;
 	struct isp_video *far_end = NULL;
 	int ret;
 
@@ -234,23 +234,24 @@ static int isp_video_get_graph_data(struct isp_video *video,
 		return ret;
 	}
 
-	media_graph_walk_start(&graph, entity->pads);
+	media_graph_walk_start(&graph, pad);
 
-	while ((entity = media_graph_walk_next(&graph))) {
+	while ((pad = media_graph_walk_next(&graph))) {
 		struct isp_video *__video;
 
-		media_entity_enum_set(&pipe->ent_enum, entity);
+		media_entity_enum_set(&pipe->ent_enum, pad->entity);
 
 		if (far_end != NULL)
 			continue;
 
-		if (entity == &video->video.entity)
+		if (pad == video->video.entity.pads)
 			continue;
 
-		if (!is_media_entity_v4l2_video_device(entity))
+		if (!is_media_entity_v4l2_video_device(pad->entity))
 			continue;
 
-		__video = to_isp_video(media_entity_to_video_device(entity));
+		__video = to_isp_video(media_entity_to_video_device(
+					       pad->entity));
 		if (__video->type != video->type)
 			far_end = __video;
 	}
diff --git a/drivers/media/platform/vsp1/vsp1_video.c b/drivers/media/platform/vsp1/vsp1_video.c
index 61e4fbaba7b7..f2c36f2fdf53 100644
--- a/drivers/media/platform/vsp1/vsp1_video.c
+++ b/drivers/media/platform/vsp1/vsp1_video.c
@@ -559,8 +559,8 @@ static int vsp1_video_pipeline_build(struct vsp1_pipeline *pipe,
 				     struct vsp1_video *video)
 {
 	struct media_graph graph;
-	struct media_entity *entity = &video->video.entity;
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_pad *pad = video->video.entity.pads;
+	struct media_device *mdev = video->video.entity.graph_obj.mdev;
 	unsigned int i;
 	int ret;
 
@@ -569,17 +569,17 @@ static int vsp1_video_pipeline_build(struct vsp1_pipeline *pipe,
 	if (ret)
 		return ret;
 
-	media_graph_walk_start(&graph, entity->pads);
+	media_graph_walk_start(&graph, pad);
 
-	while ((entity = media_graph_walk_next(&graph))) {
+	while ((pad = media_graph_walk_next(&graph))) {
 		struct v4l2_subdev *subdev;
 		struct vsp1_rwpf *rwpf;
 		struct vsp1_entity *e;
 
-		if (!is_media_entity_v4l2_subdev(entity))
+		if (!is_media_entity_v4l2_subdev(pad->entity))
 			continue;
 
-		subdev = media_entity_to_v4l2_subdev(entity);
+		subdev = media_entity_to_v4l2_subdev(pad->entity);
 		e = to_vsp1_entity(subdev);
 		list_add_tail(&e->list_pipe, &pipe->entities);
 		e->pipe = pipe;
diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
index af0d55ab6c15..d33f99c6ffa4 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.c
+++ b/drivers/media/platform/xilinx/xilinx-dma.c
@@ -174,8 +174,8 @@ static int xvip_pipeline_validate(struct xvip_pipeline *pipe,
 				  struct xvip_dma *start)
 {
 	struct media_graph graph;
-	struct media_entity *entity = &start->video.entity;
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_pad *pad = start->video.entity.pads;
+	struct media_device *mdev = pad->entity->graph_obj.mdev;
 	unsigned int num_inputs = 0;
 	unsigned int num_outputs = 0;
 	int ret;
@@ -189,15 +189,15 @@ static int xvip_pipeline_validate(struct xvip_pipeline *pipe,
 		return ret;
 	}
 
-	media_graph_walk_start(&graph, entity->pads);
+	media_graph_walk_start(&graph, pad);
 
-	while ((entity = media_graph_walk_next(&graph))) {
+	while ((pad = media_graph_walk_next(&graph))) {
 		struct xvip_dma *dma;
 
-		if (entity->function != MEDIA_ENT_F_IO_V4L)
+		if (pad->entity->function != MEDIA_ENT_F_IO_V4L)
 			continue;
 
-		dma = to_xvip_dma(media_entity_to_video_device(entity));
+		dma = to_xvip_dma(media_entity_to_video_device(pad->entity));
 
 		if (dma->pad.flags & MEDIA_PAD_FL_SINK) {
 			pipe->output = dma;
diff --git a/drivers/media/v4l2-core/v4l2-mc.c b/drivers/media/v4l2-core/v4l2-mc.c
index d215fe31b9a2..cbeb580c6754 100644
--- a/drivers/media/v4l2-core/v4l2-mc.c
+++ b/drivers/media/v4l2-core/v4l2-mc.c
@@ -434,13 +434,14 @@ EXPORT_SYMBOL_GPL(v4l2_create_fwnode_links);
 static int pipeline_pm_use_count(struct media_entity *entity,
 	struct media_graph *graph)
 {
+	struct media_pad *pad;
 	int use = 0;
 
 	media_graph_walk_start(graph, entity->pads);
 
-	while ((entity = media_graph_walk_next(graph))) {
-		if (is_media_entity_v4l2_video_device(entity))
-			use += entity->use_count;
+	while ((pad = media_graph_walk_next(graph))) {
+		if (is_media_entity_v4l2_video_device(pad->entity))
+			use += pad->entity->use_count;
 	}
 
 	return use;
@@ -493,7 +494,7 @@ static int pipeline_pm_power_one(struct media_entity *entity, int change)
 static int pipeline_pm_power(struct media_entity *entity, int change,
 	struct media_graph *graph)
 {
-	struct media_entity *first = entity;
+	struct media_pad *tmp_pad, *pad;
 	int ret = 0;
 
 	if (!change)
@@ -501,19 +502,18 @@ static int pipeline_pm_power(struct media_entity *entity, int change,
 
 	media_graph_walk_start(graph, entity->pads);
 
-	while (!ret && (entity = media_graph_walk_next(graph)))
-		if (is_media_entity_v4l2_subdev(entity))
-			ret = pipeline_pm_power_one(entity, change);
+	while (!ret && (pad = media_graph_walk_next(graph)))
+		if (is_media_entity_v4l2_subdev(pad->entity))
+			ret = pipeline_pm_power_one(pad->entity, change);
 
 	if (!ret)
 		return ret;
 
-	media_graph_walk_start(graph, first->pads);
+	media_graph_walk_start(graph, entity->pads);
 
-	while ((first = media_graph_walk_next(graph))
-	       && first != entity)
-		if (is_media_entity_v4l2_subdev(first))
-			pipeline_pm_power_one(first, -change);
+	while ((tmp_pad = media_graph_walk_next(graph)) && tmp_pad != pad)
+		if (is_media_entity_v4l2_subdev(tmp_pad->entity))
+			pipeline_pm_power_one(tmp_pad->entity, -change);
 
 	return ret;
 }
diff --git a/drivers/staging/media/omap4iss/iss_video.c b/drivers/staging/media/omap4iss/iss_video.c
index 760cd0ab1feb..8c25ad73a81e 100644
--- a/drivers/staging/media/omap4iss/iss_video.c
+++ b/drivers/staging/media/omap4iss/iss_video.c
@@ -206,8 +206,8 @@ static struct iss_video *
 iss_video_far_end(struct iss_video *video)
 {
 	struct media_graph graph;
-	struct media_entity *entity = &video->video.entity;
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_pad *pad = video->video.entity.pads;
+	struct media_device *mdev = video->video.entity.graph_obj.mdev;
 	struct iss_video *far_end = NULL;
 
 	mutex_lock(&mdev->graph_mutex);
@@ -217,16 +217,17 @@ iss_video_far_end(struct iss_video *video)
 		return NULL;
 	}
 
-	media_graph_walk_start(&graph, entity->pads);
+	media_graph_walk_start(&graph, pad);
 
-	while ((entity = media_graph_walk_next(&graph))) {
-		if (entity == &video->video.entity)
+	while ((pad = media_graph_walk_next(&graph))) {
+		if (pad->entity == &video->video.entity)
 			continue;
 
-		if (!is_media_entity_v4l2_video_device(entity))
+		if (!is_media_entity_v4l2_video_device(pad->entity))
 			continue;
 
-		far_end = to_iss_video(media_entity_to_video_device(entity));
+		far_end = to_iss_video(media_entity_to_video_device(
+						pad->entity));
 		if (far_end->type != video->type)
 			break;
 
@@ -853,8 +854,8 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	struct iss_video_fh *vfh = to_iss_video_fh(fh);
 	struct iss_video *video = video_drvdata(file);
 	struct media_graph graph;
-	struct media_entity *entity = &video->video.entity;
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_pad *pad = video->video.entity.pads;
+	struct media_device *mdev = video->video.entity.graph_obj.mdev;
 	enum iss_pipeline_state state;
 	struct iss_pipeline *pipe;
 	struct iss_video *far_end;
@@ -870,31 +871,32 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	 * Start streaming on the pipeline. No link touching an entity in the
 	 * pipeline can be activated or deactivated once streaming is started.
 	 */
-	pipe = entity->pipe
-	     ? to_iss_pipeline(entity) : &video->pipe;
+	pipe = pad->entity->pipe
+	     ? to_iss_pipeline(pad->entity) : &video->pipe;
 	pipe->external = NULL;
 	pipe->external_rate = 0;
 	pipe->external_bpp = 0;
 
-	ret = media_entity_enum_init(&pipe->ent_enum, entity->graph_obj.mdev);
+	ret = media_entity_enum_init(&pipe->ent_enum,
+				     pad->entity->graph_obj.mdev);
 	if (ret)
 		goto err_graph_walk_init;
 
-	ret = media_graph_walk_init(&graph, entity->graph_obj.mdev);
+	ret = media_graph_walk_init(&graph, pad->entity->graph_obj.mdev);
 	if (ret)
 		goto err_graph_walk_init;
 
 	if (video->iss->pdata->set_constraints)
 		video->iss->pdata->set_constraints(video->iss, true);
 
-	ret = media_pipeline_start(entity, &pipe->pipe);
+	ret = media_pipeline_start(pad->entity, &pipe->pipe);
 	if (ret < 0)
 		goto err_media_pipeline_start;
 
 	mutex_lock(&mdev->graph_mutex);
-	media_graph_walk_start(&graph, entity->pads);
-	while ((entity = media_graph_walk_next(&graph)))
-		media_entity_enum_set(&pipe->ent_enum, entity);
+	media_graph_walk_start(&graph, pad);
+	while ((pad = media_graph_walk_next(&graph)))
+		media_entity_enum_set(&pipe->ent_enum, pad->entity);
 	mutex_unlock(&mdev->graph_mutex);
 
 	/*
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 5b55d6179e13..926fd201eae3 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -926,10 +926,11 @@ void media_graph_walk_start(struct media_graph *graph, struct media_pad *pad);
  * The graph structure must have been previously initialized with a call to
  * media_graph_walk_start().
  *
- * Return: returns the next entity in the graph or %NULL if the whole graph
- * have been traversed.
+ * Return: returns the next entity in the graph, identified by the pad through
+ * which it has been reached. If the whole graph has been traversed, return
+ * %NULL.
  */
-struct media_entity *media_graph_walk_next(struct media_graph *graph);
+struct media_pad *media_graph_walk_next(struct media_graph *graph);
 
 /**
  * media_pipeline_start - Mark a pipeline as streaming
-- 
2.25.1

[PATCH v10 10/38] media: mc: Start walk from a specific pad in use count calculation

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

With the addition of the recent has_route() media entity op, the pads of a
media entity are no longer all interconnected. This has to be taken into
account in power management.

Prepare for the addition of a helper function supporting S_ROUTING.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-mc.c | 27 +++++++++++++--------------
 1 file changed, 13 insertions(+), 14 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-mc.c b/drivers/media/v4l2-core/v4l2-mc.c
index cbeb580c6754..35d18ed89fa5 100644
--- a/drivers/media/v4l2-core/v4l2-mc.c
+++ b/drivers/media/v4l2-core/v4l2-mc.c
@@ -427,17 +427,16 @@ EXPORT_SYMBOL_GPL(v4l2_create_fwnode_links);
 
 /*
  * pipeline_pm_use_count - Count the number of users of a pipeline
- * @entity: The entity
+ * @pad: Any pad along the pipeline
  *
  * Return the total number of users of all video device nodes in the pipeline.
  */
-static int pipeline_pm_use_count(struct media_entity *entity,
-	struct media_graph *graph)
+static int pipeline_pm_use_count(struct media_pad *pad,
+				 struct media_graph *graph)
 {
-	struct media_pad *pad;
 	int use = 0;
 
-	media_graph_walk_start(graph, entity->pads);
+	media_graph_walk_start(graph, pad);
 
 	while ((pad = media_graph_walk_next(graph))) {
 		if (is_media_entity_v4l2_video_device(pad->entity))
@@ -483,7 +482,7 @@ static int pipeline_pm_power_one(struct media_entity *entity, int change)
 
 /*
  * pipeline_pm_power - Apply power change to all entities in a pipeline
- * @entity: The entity
+ * @pad: Any pad along the pipeline
  * @change: Use count change
  *
  * Walk the pipeline to update the use count and the power state of all non-node
@@ -491,16 +490,16 @@ static int pipeline_pm_power_one(struct media_entity *entity, int change)
  *
  * Return 0 on success or a negative error code on failure.
  */
-static int pipeline_pm_power(struct media_entity *entity, int change,
-	struct media_graph *graph)
+static int pipeline_pm_power(struct media_pad *pad, int change,
+			     struct media_graph *graph)
 {
-	struct media_pad *tmp_pad, *pad;
+	struct media_pad *tmp_pad, *first = pad;
 	int ret = 0;
 
 	if (!change)
 		return 0;
 
-	media_graph_walk_start(graph, entity->pads);
+	media_graph_walk_start(graph, pad);
 
 	while (!ret && (pad = media_graph_walk_next(graph)))
 		if (is_media_entity_v4l2_subdev(pad->entity))
@@ -509,7 +508,7 @@ static int pipeline_pm_power(struct media_entity *entity, int change,
 	if (!ret)
 		return ret;
 
-	media_graph_walk_start(graph, entity->pads);
+	media_graph_walk_start(graph, first);
 
 	while ((tmp_pad = media_graph_walk_next(graph)) && tmp_pad != pad)
 		if (is_media_entity_v4l2_subdev(tmp_pad->entity))
@@ -531,7 +530,7 @@ static int v4l2_pipeline_pm_use(struct media_entity *entity, unsigned int use)
 	WARN_ON(entity->use_count < 0);
 
 	/* Apply power change to connected non-nodes. */
-	ret = pipeline_pm_power(entity, change, &mdev->pm_count_walk);
+	ret = pipeline_pm_power(entity->pads, change, &mdev->pm_count_walk);
 	if (ret < 0)
 		entity->use_count -= change;
 
@@ -557,8 +556,8 @@ int v4l2_pipeline_link_notify(struct media_link *link, u32 flags,
 			      unsigned int notification)
 {
 	struct media_graph *graph = &link->graph_obj.mdev->pm_count_walk;
-	struct media_entity *source = link->source->entity;
-	struct media_entity *sink = link->sink->entity;
+	struct media_pad *source = link->source;
+	struct media_pad *sink = link->sink;
 	int source_use;
 	int sink_use;
 	int ret = 0;
-- 
2.25.1

[PATCH v10 11/38] media: entity: Add iterator helper for entity pads

[Tomi Valkeinen]

From: Jacopo Mondi <jacopo+renesas@jmondi.org>

Add an iterator helper to easily cycle through all pads in an entity and
use it in media-entity and media-device code where appropriate.

Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-device.c | 13 ++++++-------
 drivers/media/mc/mc-entity.c | 11 ++++++-----
 include/media/media-entity.h | 12 ++++++++++++
 3 files changed, 24 insertions(+), 12 deletions(-)

diff --git a/drivers/media/mc/mc-device.c b/drivers/media/mc/mc-device.c
index cf5e459b1d96..cb569beab99e 100644
--- a/drivers/media/mc/mc-device.c
+++ b/drivers/media/mc/mc-device.c
@@ -581,7 +581,7 @@ static void __media_device_unregister_entity(struct media_entity *entity)
 	struct media_device *mdev = entity->graph_obj.mdev;
 	struct media_link *link, *tmp;
 	struct media_interface *intf;
-	unsigned int i;
+	struct media_pad *iter;
 
 	ida_free(&mdev->entity_internal_idx, entity->internal_idx);
 
@@ -597,8 +597,8 @@ static void __media_device_unregister_entity(struct media_entity *entity)
 	__media_entity_remove_links(entity);
 
 	/* Remove all pads that belong to this entity */
-	for (i = 0; i < entity->num_pads; i++)
-		media_gobj_destroy(&entity->pads[i].graph_obj);
+	media_entity_for_each_pad(entity, iter)
+		media_gobj_destroy(&iter->graph_obj);
 
 	/* Remove the entity */
 	media_gobj_destroy(&entity->graph_obj);
@@ -617,7 +617,7 @@ int __must_check media_device_register_entity(struct media_device *mdev,
 					      struct media_entity *entity)
 {
 	struct media_entity_notify *notify, *next;
-	unsigned int i;
+	struct media_pad *iter;
 	int ret;
 
 	if (entity->function == MEDIA_ENT_F_V4L2_SUBDEV_UNKNOWN ||
@@ -646,9 +646,8 @@ int __must_check media_device_register_entity(struct media_device *mdev,
 	media_gobj_create(mdev, MEDIA_GRAPH_ENTITY, &entity->graph_obj);
 
 	/* Initialize objects at the pads */
-	for (i = 0; i < entity->num_pads; i++)
-		media_gobj_create(mdev, MEDIA_GRAPH_PAD,
-			       &entity->pads[i].graph_obj);
+	media_entity_for_each_pad(entity, iter)
+		media_gobj_create(mdev, MEDIA_GRAPH_PAD, &iter->graph_obj);
 
 	/* invoke entity_notify callbacks */
 	list_for_each_entry_safe(notify, next, &mdev->entity_notify, list)
diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 06147f38ce2e..2b438c481812 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -198,7 +198,8 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 			   struct media_pad *pads)
 {
 	struct media_device *mdev = entity->graph_obj.mdev;
-	unsigned int i;
+	struct media_pad *iter;
+	unsigned int i = 0;
 
 	if (num_pads >= MEDIA_ENTITY_MAX_PADS)
 		return -E2BIG;
@@ -209,12 +210,12 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 	if (mdev)
 		mutex_lock(&mdev->graph_mutex);
 
-	for (i = 0; i < num_pads; i++) {
-		pads[i].entity = entity;
-		pads[i].index = i;
+	media_entity_for_each_pad(entity, iter) {
+		iter->entity = entity;
+		iter->index = i++;
 		if (mdev)
 			media_gobj_create(mdev, MEDIA_GRAPH_PAD,
-					&entity->pads[i].graph_obj);
+					&iter->graph_obj);
 	}
 
 	if (mdev)
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 926fd201eae3..5f6eed24e63f 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -1107,3 +1107,15 @@ void media_remove_intf_links(struct media_interface *intf);
 	 (entity)->ops->operation((entity) , ##args) : -ENOIOCTLCMD)
 
 #endif
+
+/**
+ * media_entity_for_each_pad - Iterate on all pads in an entity
+ * @entity: The entity the pads belong to
+ * @iter: The iterator pad
+ *
+ * Iterate on all pads in a media entity.
+ */
+#define media_entity_for_each_pad(entity, iter)			\
+	for (iter = (entity)->pads;				\
+	     iter < &(entity)->pads[(entity)->num_pads];	\
+	     ++iter)
-- 
2.25.1

[PATCH v10 12/38] media: entity: Move the pipeline from entity to pads

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

This moves the pipe and stream_count fields from struct media_entity to
struct media_pad. Effectively streams become pad-specific rather than
being entity specific, allowing several independent streams to traverse a
single entity and an entity to be part of several streams.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>

- Update documentation to use 'pads'
- Use the media pad iterator in media_entity.c
- Update rcar-dma.c to use the new per-pad stream count
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>

- Fix cleanup in the error path of __media_pipeline_start()
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c                  | 68 +++++++++++--------
 drivers/media/platform/exynos4-is/fimc-isp.c  |  2 +-
 drivers/media/platform/exynos4-is/fimc-lite.c |  2 +-
 drivers/media/platform/omap3isp/isp.c         |  2 +-
 drivers/media/platform/omap3isp/ispvideo.c    |  2 +-
 drivers/media/platform/omap3isp/ispvideo.h    |  2 +-
 drivers/media/platform/rcar-vin/rcar-core.c   | 16 +++--
 drivers/media/platform/rcar-vin/rcar-dma.c    |  2 +-
 drivers/media/platform/xilinx/xilinx-dma.c    |  2 +-
 drivers/media/platform/xilinx/xilinx-dma.h    |  2 +-
 drivers/staging/media/imx/imx-media-utils.c   |  2 +-
 drivers/staging/media/omap4iss/iss.c          |  2 +-
 drivers/staging/media/omap4iss/iss_video.c    |  2 +-
 drivers/staging/media/omap4iss/iss_video.h    |  2 +-
 include/media/media-entity.h                  | 21 +++---
 15 files changed, 74 insertions(+), 55 deletions(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 2b438c481812..8ad4cb845f4a 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -424,24 +424,30 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 
 	while ((pad = media_graph_walk_next(graph))) {
 		struct media_entity *entity = pad->entity;
+		bool skip_validation = pad->pipe != NULL;
+		struct media_pad *iter;
 
 		DECLARE_BITMAP(active, MEDIA_ENTITY_MAX_PADS);
 		DECLARE_BITMAP(has_no_links, MEDIA_ENTITY_MAX_PADS);
 
-		entity->stream_count++;
+		ret = 0;
 
-		if (entity->pipe && entity->pipe != pipe) {
-			pr_err("Pipe active for %s. Can't start for %s\n",
-				entity->name,
-				pad_err->entity->name);
-			ret = -EBUSY;
-			goto error;
+		media_entity_for_each_pad(entity, iter) {
+			if (iter->pipe && iter->pipe != pipe) {
+				pr_err("Pipe active for %s. Can't start for %s\n",
+				       entity->name, iter->entity->name);
+				ret = -EBUSY;
+			} else {
+				iter->pipe = pipe;
+			}
+			iter->stream_count++;
 		}
 
-		entity->pipe = pipe;
+		if (ret)
+			goto error;
 
-		/* Already streaming --- no need to check. */
-		if (entity->stream_count > 1)
+		/* Already part of the pipeline, skip validation. */
+		if (skip_validation)
 			continue;
 
 		if (!entity->ops || !entity->ops->link_validate)
@@ -510,20 +516,23 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 	media_graph_walk_start(graph, pad_err);
 
 	while ((pad_err = media_graph_walk_next(graph))) {
-		struct media_entity *entity_err = pad_err->entity;
-
-		/* Sanity check for negative stream_count */
-		if (!WARN_ON_ONCE(entity_err->stream_count <= 0)) {
-			entity_err->stream_count--;
-			if (entity_err->stream_count == 0)
-				entity_err->pipe = NULL;
+		struct media_entity *entity = pad_err->entity;
+		struct media_pad *iter;
+
+		media_entity_for_each_pad(entity, iter) {
+			/* Sanity check for negative stream_count */
+			if (!WARN_ON_ONCE(iter->stream_count <= 0)) {
+				--iter->stream_count;
+				if (iter->stream_count == 0)
+					iter->pipe = NULL;
+			}
 		}
 
 		/*
 		 * We haven't increased stream_count further than this
 		 * so we quit here.
 		 */
-		if (pad_err == pad)
+		if (pad_err->entity == pad->entity)
 			break;
 	}
 
@@ -550,7 +559,7 @@ EXPORT_SYMBOL_GPL(media_pipeline_start);
 
 void __media_pipeline_stop(struct media_entity *entity)
 {
-	struct media_pipeline *pipe = entity->pipe;
+	struct media_pipeline *pipe = entity->pads->pipe;
 	struct media_graph *graph = &pipe->graph;
 	struct media_pad *pad;
 
@@ -565,12 +574,15 @@ void __media_pipeline_stop(struct media_entity *entity)
 
 	while ((pad = media_graph_walk_next(graph))) {
 		struct media_entity *entity = pad->entity;
-
-		/* Sanity check for negative stream_count */
-		if (!WARN_ON_ONCE(entity->stream_count <= 0)) {
-			entity->stream_count--;
-			if (entity->stream_count == 0)
-				entity->pipe = NULL;
+		struct media_pad *iter;
+
+		media_entity_for_each_pad(entity, iter) {
+			/* Sanity check for negative stream_count */
+			if (!WARN_ON_ONCE(iter->stream_count <= 0)) {
+				iter->stream_count--;
+				if (iter->stream_count == 0)
+					iter->pipe = NULL;
+			}
 		}
 	}
 
@@ -840,7 +852,7 @@ int __media_entity_setup_link(struct media_link *link, u32 flags)
 {
 	const u32 mask = MEDIA_LNK_FL_ENABLED;
 	struct media_device *mdev;
-	struct media_entity *source, *sink;
+	struct media_pad *source, *sink;
 	int ret = -EBUSY;
 
 	if (link == NULL)
@@ -856,8 +868,8 @@ int __media_entity_setup_link(struct media_link *link, u32 flags)
 	if (link->flags == flags)
 		return 0;
 
-	source = link->source->entity;
-	sink = link->sink->entity;
+	source = link->source;
+	sink = link->sink;
 
 	if (!(link->flags & MEDIA_LNK_FL_DYNAMIC) &&
 	    (source->stream_count || sink->stream_count))
diff --git a/drivers/media/platform/exynos4-is/fimc-isp.c b/drivers/media/platform/exynos4-is/fimc-isp.c
index 855235bea46d..80274e29ccc5 100644
--- a/drivers/media/platform/exynos4-is/fimc-isp.c
+++ b/drivers/media/platform/exynos4-is/fimc-isp.c
@@ -226,7 +226,7 @@ static int fimc_isp_subdev_set_fmt(struct v4l2_subdev *sd,
 			}
 		}
 	} else {
-		if (sd->entity.stream_count == 0) {
+		if (sd->entity.pads->stream_count == 0) {
 			if (fmt->pad == FIMC_ISP_SD_PAD_SINK) {
 				struct v4l2_subdev_format format = *fmt;
 
diff --git a/drivers/media/platform/exynos4-is/fimc-lite.c b/drivers/media/platform/exynos4-is/fimc-lite.c
index aaa3af0493ce..67bfb1ad2ba2 100644
--- a/drivers/media/platform/exynos4-is/fimc-lite.c
+++ b/drivers/media/platform/exynos4-is/fimc-lite.c
@@ -1073,7 +1073,7 @@ static int fimc_lite_subdev_set_fmt(struct v4l2_subdev *sd,
 	mutex_lock(&fimc->lock);
 
 	if ((atomic_read(&fimc->out_path) == FIMC_IO_ISP &&
-	    sd->entity.stream_count > 0) ||
+	    sd->entity.pads->stream_count > 0) ||
 	    (atomic_read(&fimc->out_path) == FIMC_IO_DMA &&
 	    vb2_is_busy(&fimc->vb_queue))) {
 		mutex_unlock(&fimc->lock);
diff --git a/drivers/media/platform/omap3isp/isp.c b/drivers/media/platform/omap3isp/isp.c
index 20f59c59ff8a..28aab16d7662 100644
--- a/drivers/media/platform/omap3isp/isp.c
+++ b/drivers/media/platform/omap3isp/isp.c
@@ -936,7 +936,7 @@ static int isp_pipeline_is_last(struct media_entity *me)
 	struct isp_pipeline *pipe;
 	struct media_pad *pad;
 
-	if (!me->pipe)
+	if (!me->pads->pipe)
 		return 0;
 	pipe = to_isp_pipeline(me);
 	if (pipe->stream_state == ISP_PIPELINE_STREAM_STOPPED)
diff --git a/drivers/media/platform/omap3isp/ispvideo.c b/drivers/media/platform/omap3isp/ispvideo.c
index 5c1cbb1a9003..a8438040c4aa 100644
--- a/drivers/media/platform/omap3isp/ispvideo.c
+++ b/drivers/media/platform/omap3isp/ispvideo.c
@@ -1094,7 +1094,7 @@ isp_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	/* Start streaming on the pipeline. No link touching an entity in the
 	 * pipeline can be activated or deactivated once streaming is started.
 	 */
-	pipe = video->video.entity.pipe
+	pipe = video->video.entity.pads->pipe
 	     ? to_isp_pipeline(&video->video.entity) : &video->pipe;
 
 	ret = media_entity_enum_init(&pipe->ent_enum, &video->isp->media_dev);
diff --git a/drivers/media/platform/omap3isp/ispvideo.h b/drivers/media/platform/omap3isp/ispvideo.h
index a0908670c0cf..4c9c5b719ec5 100644
--- a/drivers/media/platform/omap3isp/ispvideo.h
+++ b/drivers/media/platform/omap3isp/ispvideo.h
@@ -100,7 +100,7 @@ struct isp_pipeline {
 };
 
 #define to_isp_pipeline(__e) \
-	container_of((__e)->pipe, struct isp_pipeline, pipe)
+	container_of((__e)->pads->pipe, struct isp_pipeline, pipe)
 
 static inline int isp_pipeline_ready(struct isp_pipeline *pipe)
 {
diff --git a/drivers/media/platform/rcar-vin/rcar-core.c b/drivers/media/platform/rcar-vin/rcar-core.c
index 33957cc9118c..e59453d6b7c3 100644
--- a/drivers/media/platform/rcar-vin/rcar-core.c
+++ b/drivers/media/platform/rcar-vin/rcar-core.c
@@ -132,13 +132,17 @@ static int rvin_group_link_notify(struct media_link *link, u32 flags,
 		return 0;
 
 	/*
-	 * Don't allow link changes if any entity in the graph is
-	 * streaming, modifying the CHSEL register fields can disrupt
-	 * running streams.
+	 * Don't allow link changes if any stream in the graph is active as
+	 * modifying the CHSEL register fields can disrupt running streams.
 	 */
-	media_device_for_each_entity(entity, &group->mdev)
-		if (entity->stream_count)
-			return -EBUSY;
+	media_device_for_each_entity(entity, &group->mdev) {
+		struct media_pad *iter;
+
+		media_entity_for_each_pad(entity, iter) {
+			if (iter->stream_count)
+				return -EBUSY;
+		}
+	}
 
 	mutex_lock(&group->lock);
 
diff --git a/drivers/media/platform/rcar-vin/rcar-dma.c b/drivers/media/platform/rcar-vin/rcar-dma.c
index f5f722ab1d4e..80b7ae47d165 100644
--- a/drivers/media/platform/rcar-vin/rcar-dma.c
+++ b/drivers/media/platform/rcar-vin/rcar-dma.c
@@ -1231,7 +1231,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
 	 */
 	mdev = vin->vdev.entity.graph_obj.mdev;
 	mutex_lock(&mdev->graph_mutex);
-	pipe = sd->entity.pipe ? sd->entity.pipe : &vin->vdev.pipe;
+	pipe = sd->entity.pads->pipe ? sd->entity.pads->pipe : &vin->vdev.pipe;
 	ret = __media_pipeline_start(&vin->vdev.entity, pipe);
 	mutex_unlock(&mdev->graph_mutex);
 	if (ret)
diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
index d33f99c6ffa4..03ee19d00041 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.c
+++ b/drivers/media/platform/xilinx/xilinx-dma.c
@@ -402,7 +402,7 @@ static int xvip_dma_start_streaming(struct vb2_queue *vq, unsigned int count)
 	 * Use the pipeline object embedded in the first DMA object that starts
 	 * streaming.
 	 */
-	pipe = dma->video.entity.pipe
+	pipe = dma->video.entity.pads->pipe
 	     ? to_xvip_pipeline(&dma->video.entity) : &dma->pipe;
 
 	ret = media_pipeline_start(&dma->video.entity, &pipe->pipe);
diff --git a/drivers/media/platform/xilinx/xilinx-dma.h b/drivers/media/platform/xilinx/xilinx-dma.h
index 2378bdae57ae..69ced71a5696 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.h
+++ b/drivers/media/platform/xilinx/xilinx-dma.h
@@ -47,7 +47,7 @@ struct xvip_pipeline {
 
 static inline struct xvip_pipeline *to_xvip_pipeline(struct media_entity *e)
 {
-	return container_of(e->pipe, struct xvip_pipeline, pipe);
+	return container_of(e->pads->pipe, struct xvip_pipeline, pipe);
 }
 
 /**
diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c
index 6f90acf9c725..535da4dda3c6 100644
--- a/drivers/staging/media/imx/imx-media-utils.c
+++ b/drivers/staging/media/imx/imx-media-utils.c
@@ -913,7 +913,7 @@ int imx_media_pipeline_set_stream(struct imx_media_dev *imxmd,
 			__media_pipeline_stop(entity);
 	} else {
 		v4l2_subdev_call(sd, video, s_stream, 0);
-		if (entity->pipe)
+		if (entity->pads->pipe)
 			__media_pipeline_stop(entity);
 	}
 
diff --git a/drivers/staging/media/omap4iss/iss.c b/drivers/staging/media/omap4iss/iss.c
index 68588e9dab0b..4c6f25aa8b57 100644
--- a/drivers/staging/media/omap4iss/iss.c
+++ b/drivers/staging/media/omap4iss/iss.c
@@ -548,7 +548,7 @@ static int iss_pipeline_is_last(struct media_entity *me)
 	struct iss_pipeline *pipe;
 	struct media_pad *pad;
 
-	if (!me->pipe)
+	if (!me->pads->pipe)
 		return 0;
 	pipe = to_iss_pipeline(me);
 	if (pipe->stream_state == ISS_PIPELINE_STREAM_STOPPED)
diff --git a/drivers/staging/media/omap4iss/iss_video.c b/drivers/staging/media/omap4iss/iss_video.c
index 8c25ad73a81e..b74f7891711d 100644
--- a/drivers/staging/media/omap4iss/iss_video.c
+++ b/drivers/staging/media/omap4iss/iss_video.c
@@ -871,7 +871,7 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	 * Start streaming on the pipeline. No link touching an entity in the
 	 * pipeline can be activated or deactivated once streaming is started.
 	 */
-	pipe = pad->entity->pipe
+	pipe = pad->pipe
 	     ? to_iss_pipeline(pad->entity) : &video->pipe;
 	pipe->external = NULL;
 	pipe->external_rate = 0;
diff --git a/drivers/staging/media/omap4iss/iss_video.h b/drivers/staging/media/omap4iss/iss_video.h
index 526281bf0051..9b8ec27bf87d 100644
--- a/drivers/staging/media/omap4iss/iss_video.h
+++ b/drivers/staging/media/omap4iss/iss_video.h
@@ -91,7 +91,7 @@ struct iss_pipeline {
 };
 
 #define to_iss_pipeline(__e) \
-	container_of((__e)->pipe, struct iss_pipeline, pipe)
+	container_of((__e)->pads->pipe, struct iss_pipeline, pipe)
 
 static inline int iss_pipeline_ready(struct iss_pipeline *pipe)
 {
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 5f6eed24e63f..c9d97c902d05 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -180,15 +180,24 @@ enum media_pad_signal_type {
  *
  * @graph_obj:	Embedded structure containing the media object common data
  * @entity:	Entity this pad belongs to
+ * @pipe:	Pipeline this pad belongs to
+ * @stream_count: Stream count for the pad
  * @index:	Pad index in the entity pads array, numbered from 0 to n
  * @sig_type:	Type of the signal inside a media pad
  * @flags:	Pad flags, as defined in
  *		:ref:`include/uapi/linux/media.h <media_header>`
  *		(seek for ``MEDIA_PAD_FL_*``)
+ * .. note::
+ *
+ *    @stream_count reference count must never be negative, but is a signed
+ *    integer on purpose: a simple ``WARN_ON(<0)`` check can be used to
+ *    detect reference count bugs that would make it negative.
  */
 struct media_pad {
 	struct media_gobj graph_obj;	/* must be first field in struct */
 	struct media_entity *entity;
+	struct media_pipeline *pipe;
+	int stream_count;
 	u16 index;
 	enum media_pad_signal_type sig_type;
 	unsigned long flags;
@@ -267,9 +276,7 @@ enum media_entity_type {
  * @pads:	Pads array with the size defined by @num_pads.
  * @links:	List of data links.
  * @ops:	Entity operations.
- * @stream_count: Stream count for the entity.
  * @use_count:	Use count for the entity.
- * @pipe:	Pipeline this entity belongs to.
  * @info:	Union with devnode information.  Kept just for backward
  *		compatibility.
  * @info.dev:	Contains device major and minor info.
@@ -282,10 +289,9 @@ enum media_entity_type {
  *
  * .. note::
  *
- *    @stream_count and @use_count reference counts must never be
- *    negative, but are signed integers on purpose: a simple ``WARN_ON(<0)``
- *    check can be used to detect reference count bugs that would make them
- *    negative.
+ *    @use_count reference count must never be negative, but is a signed
+ *    integer on purpose: a simple ``WARN_ON(<0)`` check can be used to
+ *    detect reference count bugs that would make it negative.
  */
 struct media_entity {
 	struct media_gobj graph_obj;	/* must be first field in struct */
@@ -304,11 +310,8 @@ struct media_entity {
 
 	const struct media_entity_operations *ops;
 
-	int stream_count;
 	int use_count;
 
-	struct media_pipeline *pipe;
-
 	union {
 		struct {
 			u32 major;
-- 
2.25.1

[PATCH v10 13/38] media: entity: Use pad as the starting point for a pipeline

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

The pipeline has been moved from the entity to the pads; reflect this in
the media pipeline function API.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 Documentation/driver-api/media/mc-core.rst    |  9 +++--
 drivers/media/mc/mc-entity.c                  | 24 ++++++-------
 drivers/media/pci/intel/ipu3/ipu3-cio2-main.c |  6 ++--
 .../media/platform/exynos4-is/fimc-capture.c  |  8 ++---
 .../platform/exynos4-is/fimc-isp-video.c      |  8 ++---
 drivers/media/platform/exynos4-is/fimc-lite.c |  8 ++---
 drivers/media/platform/omap3isp/ispvideo.c    |  6 ++--
 .../media/platform/qcom/camss/camss-video.c   |  6 ++--
 drivers/media/platform/rcar-vin/rcar-dma.c    |  6 ++--
 .../platform/rockchip/rkisp1/rkisp1-capture.c |  6 ++--
 .../media/platform/s3c-camif/camif-capture.c  |  6 ++--
 drivers/media/platform/stm32/stm32-dcmi.c     |  6 ++--
 .../platform/sunxi/sun4i-csi/sun4i_dma.c      |  6 ++--
 .../platform/sunxi/sun6i-csi/sun6i_video.c    |  6 ++--
 drivers/media/platform/ti-vpe/cal-video.c     |  6 ++--
 drivers/media/platform/vsp1/vsp1_video.c      |  6 ++--
 drivers/media/platform/xilinx/xilinx-dma.c    |  6 ++--
 .../media/test-drivers/vimc/vimc-capture.c    |  6 ++--
 drivers/media/usb/au0828/au0828-core.c        |  8 ++---
 drivers/staging/media/imx/imx-media-utils.c   |  6 ++--
 drivers/staging/media/ipu3/ipu3-v4l2.c        |  6 ++--
 drivers/staging/media/omap4iss/iss_video.c    |  6 ++--
 drivers/staging/media/tegra-video/tegra210.c  |  6 ++--
 include/media/media-entity.h                  | 34 +++++++++----------
 24 files changed, 99 insertions(+), 102 deletions(-)

diff --git a/Documentation/driver-api/media/mc-core.rst b/Documentation/driver-api/media/mc-core.rst
index 8a13640bed56..136047a22744 100644
--- a/Documentation/driver-api/media/mc-core.rst
+++ b/Documentation/driver-api/media/mc-core.rst
@@ -213,15 +213,14 @@ When starting streaming, drivers must notify all entities in the pipeline to
 prevent link states from being modified during streaming by calling
 :c:func:`media_pipeline_start()`.
 
-The function will mark all entities connected to the given entity through
-enabled links, either directly or indirectly, as streaming.
+The function will mark all the pads connected to the given pad through
+enabled routes and links, either directly or indirectly, as streaming.
 
 The struct media_pipeline instance pointed to by
-the pipe argument will be stored in every entity in the pipeline.
+the pipe argument will be stored in every pad in the pipeline.
 Drivers should embed the struct media_pipeline
 in higher-level pipeline structures and can then access the
-pipeline through the struct media_entity
-pipe field.
+pipeline through the struct media_pad pipe field.
 
 Calls to :c:func:`media_pipeline_start()` can be nested.
 The pipeline pointer must be identical for all nested calls to the function.
diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 8ad4cb845f4a..b44ab423b49b 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -404,12 +404,11 @@ EXPORT_SYMBOL_GPL(media_entity_get_fwnode_pad);
  * Pipeline management
  */
 
-__must_check int __media_pipeline_start(struct media_entity *entity,
+__must_check int __media_pipeline_start(struct media_pad *pad,
 					struct media_pipeline *pipe)
 {
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_device *mdev = pad->graph_obj.mdev;
 	struct media_graph *graph = &pipe->graph;
-	struct media_pad *pad = entity->pads;
 	struct media_pad *pad_err = pad;
 	struct media_link *link;
 	int ret;
@@ -544,24 +543,23 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 }
 EXPORT_SYMBOL_GPL(__media_pipeline_start);
 
-__must_check int media_pipeline_start(struct media_entity *entity,
+__must_check int media_pipeline_start(struct media_pad *pad,
 				      struct media_pipeline *pipe)
 {
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_device *mdev = pad->graph_obj.mdev;
 	int ret;
 
 	mutex_lock(&mdev->graph_mutex);
-	ret = __media_pipeline_start(entity, pipe);
+	ret = __media_pipeline_start(pad, pipe);
 	mutex_unlock(&mdev->graph_mutex);
 	return ret;
 }
 EXPORT_SYMBOL_GPL(media_pipeline_start);
 
-void __media_pipeline_stop(struct media_entity *entity)
+void __media_pipeline_stop(struct media_pad *pad)
 {
-	struct media_pipeline *pipe = entity->pads->pipe;
+	struct media_pipeline *pipe = pad->pipe;
 	struct media_graph *graph = &pipe->graph;
-	struct media_pad *pad;
 
 	/*
 	 * If the following check fails, the driver has performed an
@@ -570,7 +568,7 @@ void __media_pipeline_stop(struct media_entity *entity)
 	if (WARN_ON(!pipe))
 		return;
 
-	media_graph_walk_start(graph, entity->pads);
+	media_graph_walk_start(graph, pad);
 
 	while ((pad = media_graph_walk_next(graph))) {
 		struct media_entity *entity = pad->entity;
@@ -592,12 +590,12 @@ void __media_pipeline_stop(struct media_entity *entity)
 }
 EXPORT_SYMBOL_GPL(__media_pipeline_stop);
 
-void media_pipeline_stop(struct media_entity *entity)
+void media_pipeline_stop(struct media_pad *pad)
 {
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_device *mdev = pad->graph_obj.mdev;
 
 	mutex_lock(&mdev->graph_mutex);
-	__media_pipeline_stop(entity);
+	__media_pipeline_stop(pad);
 	mutex_unlock(&mdev->graph_mutex);
 }
 EXPORT_SYMBOL_GPL(media_pipeline_stop);
diff --git a/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c b/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c
index 47db0ee0fcbf..74db79b2990f 100644
--- a/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c
+++ b/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c
@@ -981,7 +981,7 @@ static int cio2_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
 		return r;
 	}
 
-	r = media_pipeline_start(&q->vdev.entity, &q->pipe);
+	r = media_pipeline_start(q->vdev.entity.pads, &q->pipe);
 	if (r)
 		goto fail_pipeline;
 
@@ -1001,7 +1001,7 @@ static int cio2_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
 fail_csi2_subdev:
 	cio2_hw_exit(cio2, q);
 fail_hw:
-	media_pipeline_stop(&q->vdev.entity);
+	media_pipeline_stop(q->vdev.entity.pads);
 fail_pipeline:
 	dev_dbg(&cio2->pci_dev->dev, "failed to start streaming (%d)\n", r);
 	cio2_vb2_return_all_buffers(q, VB2_BUF_STATE_QUEUED);
@@ -1022,7 +1022,7 @@ static void cio2_vb2_stop_streaming(struct vb2_queue *vq)
 	cio2_hw_exit(cio2, q);
 	synchronize_irq(cio2->pci_dev->irq);
 	cio2_vb2_return_all_buffers(q, VB2_BUF_STATE_ERROR);
-	media_pipeline_stop(&q->vdev.entity);
+	media_pipeline_stop(q->vdev.entity.pads);
 	pm_runtime_put(&cio2->pci_dev->dev);
 	cio2->streaming = false;
 }
diff --git a/drivers/media/platform/exynos4-is/fimc-capture.c b/drivers/media/platform/exynos4-is/fimc-capture.c
index 7ff4024003f4..eaac0ac4e406 100644
--- a/drivers/media/platform/exynos4-is/fimc-capture.c
+++ b/drivers/media/platform/exynos4-is/fimc-capture.c
@@ -524,7 +524,7 @@ static int fimc_capture_release(struct file *file)
 	mutex_lock(&fimc->lock);
 
 	if (close && vc->streaming) {
-		media_pipeline_stop(&vc->ve.vdev.entity);
+		media_pipeline_stop(vc->ve.vdev.entity.pads);
 		vc->streaming = false;
 	}
 
@@ -1184,7 +1184,7 @@ static int fimc_cap_streamon(struct file *file, void *priv,
 	if (fimc_capture_active(fimc))
 		return -EBUSY;
 
-	ret = media_pipeline_start(entity, &vc->ve.pipe->mp);
+	ret = media_pipeline_start(entity->pads, &vc->ve.pipe->mp);
 	if (ret < 0)
 		return ret;
 
@@ -1218,7 +1218,7 @@ static int fimc_cap_streamon(struct file *file, void *priv,
 	}
 
 err_p_stop:
-	media_pipeline_stop(entity);
+	media_pipeline_stop(entity->pads);
 	return ret;
 }
 
@@ -1234,7 +1234,7 @@ static int fimc_cap_streamoff(struct file *file, void *priv,
 		return ret;
 
 	if (vc->streaming) {
-		media_pipeline_stop(&vc->ve.vdev.entity);
+		media_pipeline_stop(vc->ve.vdev.entity.pads);
 		vc->streaming = false;
 	}
 
diff --git a/drivers/media/platform/exynos4-is/fimc-isp-video.c b/drivers/media/platform/exynos4-is/fimc-isp-video.c
index 83688a7982f7..e2862b3dcdfc 100644
--- a/drivers/media/platform/exynos4-is/fimc-isp-video.c
+++ b/drivers/media/platform/exynos4-is/fimc-isp-video.c
@@ -312,7 +312,7 @@ static int isp_video_release(struct file *file)
 	is_singular_file = v4l2_fh_is_singular_file(file);
 
 	if (is_singular_file && ivc->streaming) {
-		media_pipeline_stop(entity);
+		media_pipeline_stop(entity->pads);
 		ivc->streaming = 0;
 	}
 
@@ -493,7 +493,7 @@ static int isp_video_streamon(struct file *file, void *priv,
 	struct media_entity *me = &ve->vdev.entity;
 	int ret;
 
-	ret = media_pipeline_start(me, &ve->pipe->mp);
+	ret = media_pipeline_start(me->pads, &ve->pipe->mp);
 	if (ret < 0)
 		return ret;
 
@@ -508,7 +508,7 @@ static int isp_video_streamon(struct file *file, void *priv,
 	isp->video_capture.streaming = 1;
 	return 0;
 p_stop:
-	media_pipeline_stop(me);
+	media_pipeline_stop(me->pads);
 	return ret;
 }
 
@@ -523,7 +523,7 @@ static int isp_video_streamoff(struct file *file, void *priv,
 	if (ret < 0)
 		return ret;
 
-	media_pipeline_stop(&video->ve.vdev.entity);
+	media_pipeline_stop(video->ve.vdev.entity.pads);
 	video->streaming = 0;
 	return 0;
 }
diff --git a/drivers/media/platform/exynos4-is/fimc-lite.c b/drivers/media/platform/exynos4-is/fimc-lite.c
index 67bfb1ad2ba2..a979600ff6e6 100644
--- a/drivers/media/platform/exynos4-is/fimc-lite.c
+++ b/drivers/media/platform/exynos4-is/fimc-lite.c
@@ -516,7 +516,7 @@ static int fimc_lite_release(struct file *file)
 	if (v4l2_fh_is_singular_file(file) &&
 	    atomic_read(&fimc->out_path) == FIMC_IO_DMA) {
 		if (fimc->streaming) {
-			media_pipeline_stop(entity);
+			media_pipeline_stop(entity->pads);
 			fimc->streaming = false;
 		}
 		fimc_lite_stop_capture(fimc, false);
@@ -822,7 +822,7 @@ static int fimc_lite_streamon(struct file *file, void *priv,
 	if (fimc_lite_active(fimc))
 		return -EBUSY;
 
-	ret = media_pipeline_start(entity, &fimc->ve.pipe->mp);
+	ret = media_pipeline_start(entity->pads, &fimc->ve.pipe->mp);
 	if (ret < 0)
 		return ret;
 
@@ -839,7 +839,7 @@ static int fimc_lite_streamon(struct file *file, void *priv,
 	}
 
 err_p_stop:
-	media_pipeline_stop(entity);
+	media_pipeline_stop(entity->pads);
 	return 0;
 }
 
@@ -853,7 +853,7 @@ static int fimc_lite_streamoff(struct file *file, void *priv,
 	if (ret < 0)
 		return ret;
 
-	media_pipeline_stop(&fimc->ve.vdev.entity);
+	media_pipeline_stop(fimc->ve.vdev.entity.pads);
 	fimc->streaming = false;
 	return 0;
 }
diff --git a/drivers/media/platform/omap3isp/ispvideo.c b/drivers/media/platform/omap3isp/ispvideo.c
index a8438040c4aa..87334477f223 100644
--- a/drivers/media/platform/omap3isp/ispvideo.c
+++ b/drivers/media/platform/omap3isp/ispvideo.c
@@ -1105,7 +1105,7 @@ isp_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	pipe->l3_ick = clk_get_rate(video->isp->clock[ISP_CLK_L3_ICK]);
 	pipe->max_rate = pipe->l3_ick;
 
-	ret = media_pipeline_start(&video->video.entity, &pipe->pipe);
+	ret = media_pipeline_start(video->video.entity.pads, &pipe->pipe);
 	if (ret < 0)
 		goto err_pipeline_start;
 
@@ -1162,7 +1162,7 @@ isp_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	return 0;
 
 err_check_format:
-	media_pipeline_stop(&video->video.entity);
+	media_pipeline_stop(video->video.entity.pads);
 err_pipeline_start:
 	/* TODO: Implement PM QoS */
 	/* The DMA queue must be emptied here, otherwise CCDC interrupts that
@@ -1229,7 +1229,7 @@ isp_video_streamoff(struct file *file, void *fh, enum v4l2_buf_type type)
 	video->error = false;
 
 	/* TODO: Implement PM QoS */
-	media_pipeline_stop(&video->video.entity);
+	media_pipeline_stop(video->video.entity.pads);
 
 	media_entity_enum_cleanup(&pipe->ent_enum);
 
diff --git a/drivers/media/platform/qcom/camss/camss-video.c b/drivers/media/platform/qcom/camss/camss-video.c
index f282275af626..5cd494f17589 100644
--- a/drivers/media/platform/qcom/camss/camss-video.c
+++ b/drivers/media/platform/qcom/camss/camss-video.c
@@ -491,7 +491,7 @@ static int video_start_streaming(struct vb2_queue *q, unsigned int count)
 	struct v4l2_subdev *subdev;
 	int ret;
 
-	ret = media_pipeline_start(&vdev->entity, &video->pipe);
+	ret = media_pipeline_start(vdev->entity.pads, &video->pipe);
 	if (ret < 0)
 		return ret;
 
@@ -520,7 +520,7 @@ static int video_start_streaming(struct vb2_queue *q, unsigned int count)
 	return 0;
 
 error:
-	media_pipeline_stop(&vdev->entity);
+	media_pipeline_stop(vdev->entity.pads);
 
 	video->ops->flush_buffers(video, VB2_BUF_STATE_QUEUED);
 
@@ -551,7 +551,7 @@ static void video_stop_streaming(struct vb2_queue *q)
 		v4l2_subdev_call(subdev, video, s_stream, 0);
 	}
 
-	media_pipeline_stop(&vdev->entity);
+	media_pipeline_stop(vdev->entity.pads);
 
 	video->ops->flush_buffers(video, VB2_BUF_STATE_ERROR);
 }
diff --git a/drivers/media/platform/rcar-vin/rcar-dma.c b/drivers/media/platform/rcar-vin/rcar-dma.c
index 80b7ae47d165..83b2f923cf98 100644
--- a/drivers/media/platform/rcar-vin/rcar-dma.c
+++ b/drivers/media/platform/rcar-vin/rcar-dma.c
@@ -1215,7 +1215,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
 	sd = media_entity_to_v4l2_subdev(pad->entity);
 
 	if (!on) {
-		media_pipeline_stop(&vin->vdev.entity);
+		media_pipeline_stop(vin->vdev.entity.pads);
 		return v4l2_subdev_call(sd, video, s_stream, 0);
 	}
 
@@ -1232,7 +1232,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
 	mdev = vin->vdev.entity.graph_obj.mdev;
 	mutex_lock(&mdev->graph_mutex);
 	pipe = sd->entity.pads->pipe ? sd->entity.pads->pipe : &vin->vdev.pipe;
-	ret = __media_pipeline_start(&vin->vdev.entity, pipe);
+	ret = __media_pipeline_start(vin->vdev.entity.pads, pipe);
 	mutex_unlock(&mdev->graph_mutex);
 	if (ret)
 		return ret;
@@ -1241,7 +1241,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
 	if (ret == -ENOIOCTLCMD)
 		ret = 0;
 	if (ret)
-		media_pipeline_stop(&vin->vdev.entity);
+		media_pipeline_stop(vin->vdev.entity.pads);
 
 	return ret;
 }
diff --git a/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c b/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
index 41988eb0ec0a..2efbe181b6f1 100644
--- a/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
+++ b/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
@@ -988,7 +988,7 @@ static void rkisp1_vb2_stop_streaming(struct vb2_queue *queue)
 
 	rkisp1_dummy_buf_destroy(cap);
 
-	media_pipeline_stop(&node->vdev.entity);
+	media_pipeline_stop(node->vdev.entity.pads);
 
 	mutex_unlock(&cap->rkisp1->stream_lock);
 }
@@ -1002,7 +1002,7 @@ rkisp1_vb2_start_streaming(struct vb2_queue *queue, unsigned int count)
 
 	mutex_lock(&cap->rkisp1->stream_lock);
 
-	ret = media_pipeline_start(entity, &cap->rkisp1->pipe);
+	ret = media_pipeline_start(entity->pads, &cap->rkisp1->pipe);
 	if (ret) {
 		dev_err(cap->rkisp1->dev, "start pipeline failed %d\n", ret);
 		goto err_ret_buffers;
@@ -1038,7 +1038,7 @@ rkisp1_vb2_start_streaming(struct vb2_queue *queue, unsigned int count)
 err_destroy_dummy:
 	rkisp1_dummy_buf_destroy(cap);
 err_pipeline_stop:
-	media_pipeline_stop(entity);
+	media_pipeline_stop(entity->pads);
 err_ret_buffers:
 	rkisp1_return_all_buffers(cap, VB2_BUF_STATE_QUEUED);
 	mutex_unlock(&cap->rkisp1->stream_lock);
diff --git a/drivers/media/platform/s3c-camif/camif-capture.c b/drivers/media/platform/s3c-camif/camif-capture.c
index 140854ab4dd8..0189b8a33032 100644
--- a/drivers/media/platform/s3c-camif/camif-capture.c
+++ b/drivers/media/platform/s3c-camif/camif-capture.c
@@ -848,13 +848,13 @@ static int s3c_camif_streamon(struct file *file, void *priv,
 	if (s3c_vp_active(vp))
 		return 0;
 
-	ret = media_pipeline_start(sensor, camif->m_pipeline);
+	ret = media_pipeline_start(sensor->pads, camif->m_pipeline);
 	if (ret < 0)
 		return ret;
 
 	ret = camif_pipeline_validate(camif);
 	if (ret < 0) {
-		media_pipeline_stop(sensor);
+		media_pipeline_stop(sensor->pads);
 		return ret;
 	}
 
@@ -878,7 +878,7 @@ static int s3c_camif_streamoff(struct file *file, void *priv,
 
 	ret = vb2_streamoff(&vp->vb_queue, type);
 	if (ret == 0)
-		media_pipeline_stop(&camif->sensor.sd->entity);
+		media_pipeline_stop(camif->sensor.sd->entity.pads);
 	return ret;
 }
 
diff --git a/drivers/media/platform/stm32/stm32-dcmi.c b/drivers/media/platform/stm32/stm32-dcmi.c
index d914ccef9831..457bc838bb73 100644
--- a/drivers/media/platform/stm32/stm32-dcmi.c
+++ b/drivers/media/platform/stm32/stm32-dcmi.c
@@ -730,7 +730,7 @@ static int dcmi_start_streaming(struct vb2_queue *vq, unsigned int count)
 		goto err_unlocked;
 	}
 
-	ret = media_pipeline_start(&dcmi->vdev->entity, &dcmi->pipeline);
+	ret = media_pipeline_start(dcmi->vdev->entity.pads, &dcmi->pipeline);
 	if (ret < 0) {
 		dev_err(dcmi->dev, "%s: Failed to start streaming, media pipeline start error (%d)\n",
 			__func__, ret);
@@ -844,7 +844,7 @@ static int dcmi_start_streaming(struct vb2_queue *vq, unsigned int count)
 	dcmi_pipeline_stop(dcmi);
 
 err_media_pipeline_stop:
-	media_pipeline_stop(&dcmi->vdev->entity);
+	media_pipeline_stop(dcmi->vdev->entity.pads);
 
 err_pm_put:
 	pm_runtime_put(dcmi->dev);
@@ -871,7 +871,7 @@ static void dcmi_stop_streaming(struct vb2_queue *vq)
 
 	dcmi_pipeline_stop(dcmi);
 
-	media_pipeline_stop(&dcmi->vdev->entity);
+	media_pipeline_stop(dcmi->vdev->entity.pads);
 
 	spin_lock_irq(&dcmi->irqlock);
 
diff --git a/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c b/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
index 2c39cd7f2862..be0defdf74f1 100644
--- a/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
+++ b/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
@@ -266,7 +266,7 @@ static int sun4i_csi_start_streaming(struct vb2_queue *vq, unsigned int count)
 		goto err_clear_dma_queue;
 	}
 
-	ret = media_pipeline_start(&csi->vdev.entity, &csi->vdev.pipe);
+	ret = media_pipeline_start(csi->vdev.entity.pads, &csi->vdev.pipe);
 	if (ret < 0)
 		goto err_free_scratch_buffer;
 
@@ -330,7 +330,7 @@ static int sun4i_csi_start_streaming(struct vb2_queue *vq, unsigned int count)
 	sun4i_csi_capture_stop(csi);
 
 err_disable_pipeline:
-	media_pipeline_stop(&csi->vdev.entity);
+	media_pipeline_stop(csi->vdev.entity.pads);
 
 err_free_scratch_buffer:
 	dma_free_coherent(csi->dev, csi->scratch.size, csi->scratch.vaddr,
@@ -359,7 +359,7 @@ static void sun4i_csi_stop_streaming(struct vb2_queue *vq)
 	return_all_buffers(csi, VB2_BUF_STATE_ERROR);
 	spin_unlock_irqrestore(&csi->qlock, flags);
 
-	media_pipeline_stop(&csi->vdev.entity);
+	media_pipeline_stop(csi->vdev.entity.pads);
 
 	dma_free_coherent(csi->dev, csi->scratch.size, csi->scratch.vaddr,
 			  csi->scratch.paddr);
diff --git a/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c b/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c
index 07b2161392d2..6baa597a4dad 100644
--- a/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c
+++ b/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c
@@ -141,7 +141,7 @@ static int sun6i_video_start_streaming(struct vb2_queue *vq, unsigned int count)
 
 	video->sequence = 0;
 
-	ret = media_pipeline_start(&video->vdev.entity, &video->vdev.pipe);
+	ret = media_pipeline_start(video->vdev.entity.pads, &video->vdev.pipe);
 	if (ret < 0)
 		goto clear_dma_queue;
 
@@ -207,7 +207,7 @@ static int sun6i_video_start_streaming(struct vb2_queue *vq, unsigned int count)
 stop_csi_stream:
 	sun6i_csi_set_stream(video->csi, false);
 stop_media_pipeline:
-	media_pipeline_stop(&video->vdev.entity);
+	media_pipeline_stop(video->vdev.entity.pads);
 clear_dma_queue:
 	spin_lock_irqsave(&video->dma_queue_lock, flags);
 	list_for_each_entry(buf, &video->dma_queue, list)
@@ -231,7 +231,7 @@ static void sun6i_video_stop_streaming(struct vb2_queue *vq)
 
 	sun6i_csi_set_stream(video->csi, false);
 
-	media_pipeline_stop(&video->vdev.entity);
+	media_pipeline_stop(video->vdev.entity.pads);
 
 	/* Release all active buffers */
 	spin_lock_irqsave(&video->dma_queue_lock, flags);
diff --git a/drivers/media/platform/ti-vpe/cal-video.c b/drivers/media/platform/ti-vpe/cal-video.c
index 7799da1cc261..861a20ee6885 100644
--- a/drivers/media/platform/ti-vpe/cal-video.c
+++ b/drivers/media/platform/ti-vpe/cal-video.c
@@ -711,7 +711,7 @@ static int cal_start_streaming(struct vb2_queue *vq, unsigned int count)
 	dma_addr_t addr;
 	int ret;
 
-	ret = media_pipeline_start(&ctx->vdev.entity, &ctx->phy->pipe);
+	ret = media_pipeline_start(ctx->vdev.entity.pads, &ctx->phy->pipe);
 	if (ret < 0) {
 		ctx_err(ctx, "Failed to start media pipeline: %d\n", ret);
 		goto error_release_buffers;
@@ -764,7 +764,7 @@ static int cal_start_streaming(struct vb2_queue *vq, unsigned int count)
 	cal_ctx_unprepare(ctx);
 
 error_pipeline:
-	media_pipeline_stop(&ctx->vdev.entity);
+	media_pipeline_stop(ctx->vdev.entity.pads);
 error_release_buffers:
 	cal_release_buffers(ctx, VB2_BUF_STATE_QUEUED);
 
@@ -785,7 +785,7 @@ static void cal_stop_streaming(struct vb2_queue *vq)
 
 	cal_release_buffers(ctx, VB2_BUF_STATE_ERROR);
 
-	media_pipeline_stop(&ctx->vdev.entity);
+	media_pipeline_stop(ctx->vdev.entity.pads);
 }
 
 static const struct vb2_ops cal_video_qops = {
diff --git a/drivers/media/platform/vsp1/vsp1_video.c b/drivers/media/platform/vsp1/vsp1_video.c
index f2c36f2fdf53..978f820b0f34 100644
--- a/drivers/media/platform/vsp1/vsp1_video.c
+++ b/drivers/media/platform/vsp1/vsp1_video.c
@@ -927,7 +927,7 @@ static void vsp1_video_stop_streaming(struct vb2_queue *vq)
 	}
 	mutex_unlock(&pipe->lock);
 
-	media_pipeline_stop(&video->video.entity);
+	media_pipeline_stop(video->video.entity.pads);
 	vsp1_video_release_buffers(video);
 	vsp1_video_pipeline_put(pipe);
 }
@@ -1048,7 +1048,7 @@ vsp1_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 		return PTR_ERR(pipe);
 	}
 
-	ret = __media_pipeline_start(&video->video.entity, &pipe->pipe);
+	ret = __media_pipeline_start(video->video.entity.pads, &pipe->pipe);
 	if (ret < 0) {
 		mutex_unlock(&mdev->graph_mutex);
 		goto err_pipe;
@@ -1072,7 +1072,7 @@ vsp1_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	return 0;
 
 err_stop:
-	media_pipeline_stop(&video->video.entity);
+	media_pipeline_stop(video->video.entity.pads);
 err_pipe:
 	vsp1_video_pipeline_put(pipe);
 	return ret;
diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
index 03ee19d00041..f04b3d190562 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.c
+++ b/drivers/media/platform/xilinx/xilinx-dma.c
@@ -405,7 +405,7 @@ static int xvip_dma_start_streaming(struct vb2_queue *vq, unsigned int count)
 	pipe = dma->video.entity.pads->pipe
 	     ? to_xvip_pipeline(&dma->video.entity) : &dma->pipe;
 
-	ret = media_pipeline_start(&dma->video.entity, &pipe->pipe);
+	ret = media_pipeline_start(dma->video.entity.pads, &pipe->pipe);
 	if (ret < 0)
 		goto error;
 
@@ -431,7 +431,7 @@ static int xvip_dma_start_streaming(struct vb2_queue *vq, unsigned int count)
 	return 0;
 
 error_stop:
-	media_pipeline_stop(&dma->video.entity);
+	media_pipeline_stop(dma->video.entity.pads);
 
 error:
 	/* Give back all queued buffers to videobuf2. */
@@ -459,7 +459,7 @@ static void xvip_dma_stop_streaming(struct vb2_queue *vq)
 
 	/* Cleanup the pipeline and mark it as being stopped. */
 	xvip_pipeline_cleanup(pipe);
-	media_pipeline_stop(&dma->video.entity);
+	media_pipeline_stop(dma->video.entity.pads);
 
 	/* Give back all queued buffers to videobuf2. */
 	spin_lock_irq(&dma->queued_lock);
diff --git a/drivers/media/test-drivers/vimc/vimc-capture.c b/drivers/media/test-drivers/vimc/vimc-capture.c
index 5e9fd902cd37..10724b0a868c 100644
--- a/drivers/media/test-drivers/vimc/vimc-capture.c
+++ b/drivers/media/test-drivers/vimc/vimc-capture.c
@@ -246,7 +246,7 @@ static int vimc_cap_start_streaming(struct vb2_queue *vq, unsigned int count)
 	vcap->sequence = 0;
 
 	/* Start the media pipeline */
-	ret = media_pipeline_start(entity, &vcap->stream.pipe);
+	ret = media_pipeline_start(entity->pads, &vcap->stream.pipe);
 	if (ret) {
 		vimc_cap_return_all_buffers(vcap, VB2_BUF_STATE_QUEUED);
 		return ret;
@@ -254,7 +254,7 @@ static int vimc_cap_start_streaming(struct vb2_queue *vq, unsigned int count)
 
 	ret = vimc_streamer_s_stream(&vcap->stream, &vcap->ved, 1);
 	if (ret) {
-		media_pipeline_stop(entity);
+		media_pipeline_stop(entity->pads);
 		vimc_cap_return_all_buffers(vcap, VB2_BUF_STATE_QUEUED);
 		return ret;
 	}
@@ -273,7 +273,7 @@ static void vimc_cap_stop_streaming(struct vb2_queue *vq)
 	vimc_streamer_s_stream(&vcap->stream, &vcap->ved, 0);
 
 	/* Stop the media pipeline */
-	media_pipeline_stop(&vcap->vdev.entity);
+	media_pipeline_stop(vcap->vdev.entity.pads);
 
 	/* Release all active buffers */
 	vimc_cap_return_all_buffers(vcap, VB2_BUF_STATE_ERROR);
diff --git a/drivers/media/usb/au0828/au0828-core.c b/drivers/media/usb/au0828/au0828-core.c
index caefac07af92..877e85a451cb 100644
--- a/drivers/media/usb/au0828/au0828-core.c
+++ b/drivers/media/usb/au0828/au0828-core.c
@@ -410,7 +410,7 @@ static int au0828_enable_source(struct media_entity *entity,
 		goto end;
 	}
 
-	ret = __media_pipeline_start(entity, pipe);
+	ret = __media_pipeline_start(entity->pads, pipe);
 	if (ret) {
 		pr_err("Start Pipeline: %s->%s Error %d\n",
 			source->name, entity->name, ret);
@@ -501,12 +501,12 @@ static void au0828_disable_source(struct media_entity *entity)
 				return;
 
 			/* stop pipeline */
-			__media_pipeline_stop(dev->active_link_owner);
+			__media_pipeline_stop(dev->active_link_owner->pads);
 			pr_debug("Pipeline stop for %s\n",
 				dev->active_link_owner->name);
 
 			ret = __media_pipeline_start(
-					dev->active_link_user,
+					dev->active_link_user->pads,
 					dev->active_link_user_pipe);
 			if (ret) {
 				pr_err("Start Pipeline: %s->%s %d\n",
@@ -532,7 +532,7 @@ static void au0828_disable_source(struct media_entity *entity)
 			return;
 
 		/* stop pipeline */
-		__media_pipeline_stop(dev->active_link_owner);
+		__media_pipeline_stop(dev->active_link_owner->pads);
 		pr_debug("Pipeline stop for %s\n",
 			dev->active_link_owner->name);
 
diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c
index 535da4dda3c6..74218af45551 100644
--- a/drivers/staging/media/imx/imx-media-utils.c
+++ b/drivers/staging/media/imx/imx-media-utils.c
@@ -905,16 +905,16 @@ int imx_media_pipeline_set_stream(struct imx_media_dev *imxmd,
 	mutex_lock(&imxmd->md.graph_mutex);
 
 	if (on) {
-		ret = __media_pipeline_start(entity, &imxmd->pipe);
+		ret = __media_pipeline_start(entity->pads, &imxmd->pipe);
 		if (ret)
 			goto out;
 		ret = v4l2_subdev_call(sd, video, s_stream, 1);
 		if (ret)
-			__media_pipeline_stop(entity);
+			__media_pipeline_stop(entity->pads);
 	} else {
 		v4l2_subdev_call(sd, video, s_stream, 0);
 		if (entity->pads->pipe)
-			__media_pipeline_stop(entity);
+			__media_pipeline_stop(entity->pads);
 	}
 
 out:
diff --git a/drivers/staging/media/ipu3/ipu3-v4l2.c b/drivers/staging/media/ipu3/ipu3-v4l2.c
index 38a240764509..db5867da3f11 100644
--- a/drivers/staging/media/ipu3/ipu3-v4l2.c
+++ b/drivers/staging/media/ipu3/ipu3-v4l2.c
@@ -485,7 +485,7 @@ static int imgu_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
 
 	pipe = node->pipe;
 	imgu_pipe = &imgu->imgu_pipe[pipe];
-	r = media_pipeline_start(&node->vdev.entity, &imgu_pipe->pipeline);
+	r = media_pipeline_start(node->vdev.entity.pads, &imgu_pipe->pipeline);
 	if (r < 0)
 		goto fail_return_bufs;
 
@@ -510,7 +510,7 @@ static int imgu_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
 	return 0;
 
 fail_stop_pipeline:
-	media_pipeline_stop(&node->vdev.entity);
+	media_pipeline_stop(node->vdev.entity.pads);
 fail_return_bufs:
 	imgu_return_all_buffers(imgu, node, VB2_BUF_STATE_QUEUED);
 
@@ -550,7 +550,7 @@ static void imgu_vb2_stop_streaming(struct vb2_queue *vq)
 	imgu_return_all_buffers(imgu, node, VB2_BUF_STATE_ERROR);
 	mutex_unlock(&imgu->streaming_lock);
 
-	media_pipeline_stop(&node->vdev.entity);
+	media_pipeline_stop(node->vdev.entity.pads);
 }
 
 /******************** v4l2_ioctl_ops ********************/
diff --git a/drivers/staging/media/omap4iss/iss_video.c b/drivers/staging/media/omap4iss/iss_video.c
index b74f7891711d..20fac40581c6 100644
--- a/drivers/staging/media/omap4iss/iss_video.c
+++ b/drivers/staging/media/omap4iss/iss_video.c
@@ -889,7 +889,7 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	if (video->iss->pdata->set_constraints)
 		video->iss->pdata->set_constraints(video->iss, true);
 
-	ret = media_pipeline_start(pad->entity, &pipe->pipe);
+	ret = media_pipeline_start(pad, &pipe->pipe);
 	if (ret < 0)
 		goto err_media_pipeline_start;
 
@@ -980,7 +980,7 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 err_omap4iss_set_stream:
 	vb2_streamoff(&vfh->queue, type);
 err_iss_video_check_format:
-	media_pipeline_stop(&video->video.entity);
+	media_pipeline_stop(video->video.entity.pads);
 err_media_pipeline_start:
 	if (video->iss->pdata->set_constraints)
 		video->iss->pdata->set_constraints(video->iss, false);
@@ -1034,7 +1034,7 @@ iss_video_streamoff(struct file *file, void *fh, enum v4l2_buf_type type)
 
 	if (video->iss->pdata->set_constraints)
 		video->iss->pdata->set_constraints(video->iss, false);
-	media_pipeline_stop(&video->video.entity);
+	media_pipeline_stop(video->video.entity.pads);
 
 done:
 	mutex_unlock(&video->stream_lock);
diff --git a/drivers/staging/media/tegra-video/tegra210.c b/drivers/staging/media/tegra-video/tegra210.c
index f10a041e3e6c..d2d7dd0e8624 100644
--- a/drivers/staging/media/tegra-video/tegra210.c
+++ b/drivers/staging/media/tegra-video/tegra210.c
@@ -547,7 +547,7 @@ static int tegra210_vi_start_streaming(struct vb2_queue *vq, u32 count)
 		       VI_INCR_SYNCPT_NO_STALL);
 
 	/* start the pipeline */
-	ret = media_pipeline_start(&chan->video.entity, pipe);
+	ret = media_pipeline_start(chan->video.entity.pads, pipe);
 	if (ret < 0)
 		goto error_pipeline_start;
 
@@ -595,7 +595,7 @@ static int tegra210_vi_start_streaming(struct vb2_queue *vq, u32 count)
 error_kthread_start:
 	tegra_channel_set_stream(chan, false);
 error_set_stream:
-	media_pipeline_stop(&chan->video.entity);
+	media_pipeline_stop(chan->video.entity.pads);
 error_pipeline_start:
 	tegra_channel_release_buffers(chan, VB2_BUF_STATE_QUEUED);
 	return ret;
@@ -617,7 +617,7 @@ static void tegra210_vi_stop_streaming(struct vb2_queue *vq)
 
 	tegra_channel_release_buffers(chan, VB2_BUF_STATE_ERROR);
 	tegra_channel_set_stream(chan, false);
-	media_pipeline_stop(&chan->video.entity);
+	media_pipeline_stop(chan->video.entity.pads);
 }
 
 /*
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index c9d97c902d05..516d73a2941e 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -937,53 +937,53 @@ struct media_pad *media_graph_walk_next(struct media_graph *graph);
 
 /**
  * media_pipeline_start - Mark a pipeline as streaming
- * @entity: Starting entity
- * @pipe: Media pipeline to be assigned to all entities in the pipeline.
+ * @pad: Starting pad
+ * @pipe: Media pipeline to be assigned to all pads in the pipeline.
  *
- * Mark all entities connected to a given entity through enabled links, either
- * directly or indirectly, as streaming. The given pipeline object is assigned
- * to every entity in the pipeline and stored in the media_entity pipe field.
+ * Mark all pads connected to a given pad through enabled routes or links,
+ * either directly or indirectly, as streaming. The given pipeline object is
+ * assigned to every pad in the pipeline and stored in the media_pad pipe
+ * field.
  *
  * Calls to this function can be nested, in which case the same number of
  * media_pipeline_stop() calls will be required to stop streaming. The
  * pipeline pointer must be identical for all nested calls to
  * media_pipeline_start().
  */
-__must_check int media_pipeline_start(struct media_entity *entity,
+__must_check int media_pipeline_start(struct media_pad *pad,
 				      struct media_pipeline *pipe);
 /**
  * __media_pipeline_start - Mark a pipeline as streaming
  *
- * @entity: Starting entity
- * @pipe: Media pipeline to be assigned to all entities in the pipeline.
+ * @pad: Starting pad
+ * @pipe: Media pipeline to be assigned to all pads in the pipeline.
  *
  * ..note:: This is the non-locking version of media_pipeline_start()
  */
-__must_check int __media_pipeline_start(struct media_entity *entity,
+__must_check int __media_pipeline_start(struct media_pad *pad,
 					struct media_pipeline *pipe);
 
 /**
  * media_pipeline_stop - Mark a pipeline as not streaming
- * @entity: Starting entity
+ * @pad: Starting pad
  *
- * Mark all entities connected to a given entity through enabled links, either
- * directly or indirectly, as not streaming. The media_entity pipe field is
- * reset to %NULL.
+ * Mark all pads connected to a given pad through enabled routes or links,
+ * either directly or indirectly, as not streaming.
  *
  * If multiple calls to media_pipeline_start() have been made, the same
  * number of calls to this function are required to mark the pipeline as not
- * streaming.
+ * streaming and reset the media_pad pipe field to %NULL.
  */
-void media_pipeline_stop(struct media_entity *entity);
+void media_pipeline_stop(struct media_pad *pad);
 
 /**
  * __media_pipeline_stop - Mark a pipeline as not streaming
  *
- * @entity: Starting entity
+ * @pad: Starting pad
  *
  * .. note:: This is the non-locking version of media_pipeline_stop()
  */
-void __media_pipeline_stop(struct media_entity *entity);
+void __media_pipeline_stop(struct media_pad *pad);
 
 /**
  * media_devnode_create() - creates and initializes a device node interface
-- 
2.25.1

[PATCH v10 14/38] media: entity: Add has_route entity operation

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

The optional operation can be used by entities to report whether two
pads are internally connected.

While at there, fix a Sphinx compiler warning in a comment block a few
lines above.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Michal Simek <michal.simek@xilinx.com>
Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 include/media/media-entity.h | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 516d73a2941e..ad4020b2df65 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -187,6 +187,7 @@ enum media_pad_signal_type {
  * @flags:	Pad flags, as defined in
  *		:ref:`include/uapi/linux/media.h <media_header>`
  *		(seek for ``MEDIA_PAD_FL_*``)
+ *
  * .. note::
  *
  *    @stream_count reference count must never be negative, but is a signed
@@ -214,6 +215,10 @@ struct media_pad {
  * @link_validate:	Return whether a link is valid from the entity point of
  *			view. The media_pipeline_start() function
  *			validates all links by calling this operation. Optional.
+ * @has_route:		Return whether a route exists inside the entity between
+ *			two given pads. Pads are passed to the operation ordered
+ *			by index. Optional: If the operation isn't implemented
+ *			all pads will be considered as connected.
  *
  * .. note::
  *
@@ -227,6 +232,8 @@ struct media_entity_operations {
 			  const struct media_pad *local,
 			  const struct media_pad *remote, u32 flags);
 	int (*link_validate)(struct media_link *link);
+	bool (*has_route)(struct media_entity *entity, unsigned int pad0,
+			  unsigned int pad1);
 };
 
 /**
-- 
2.25.1

[PATCH v10 15/38] media: entity: Add media_entity_has_route() function

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

This is a wrapper around the media entity has_route operation.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Michal Simek <michal.simek@xilinx.com>
Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 19 +++++++++++++++++++
 include/media/media-entity.h | 17 +++++++++++++++++
 2 files changed, 36 insertions(+)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index b44ab423b49b..a83f004efd37 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -229,6 +229,25 @@ EXPORT_SYMBOL_GPL(media_entity_pads_init);
  * Graph traversal
  */
 
+bool media_entity_has_route(struct media_entity *entity, unsigned int pad0,
+			    unsigned int pad1)
+{
+	if (pad0 >= entity->num_pads || pad1 >= entity->num_pads)
+		return false;
+
+	if (pad0 == pad1)
+		return true;
+
+	if (!entity->ops || !entity->ops->has_route)
+		return true;
+
+	if (entity->pads[pad1].index < entity->pads[pad0].index)
+		swap(pad0, pad1);
+
+	return entity->ops->has_route(entity, pad0, pad1);
+}
+EXPORT_SYMBOL_GPL(media_entity_has_route);
+
 static struct media_pad *
 media_pad_other(struct media_pad *pad, struct media_link *link)
 {
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index ad4020b2df65..b3069eef7fdb 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -904,6 +904,23 @@ int media_entity_get_fwnode_pad(struct media_entity *entity,
 __must_check int media_graph_walk_init(
 	struct media_graph *graph, struct media_device *mdev);
 
+/**
+ * media_entity_has_route - Check if two entity pads are connected internally
+ *
+ * @entity: The entity
+ * @pad0: The first pad index
+ * @pad1: The second pad index
+ *
+ * This function can be used to check whether two pads of an entity are
+ * connected internally in the entity.
+ *
+ * The caller must hold entity->graph_obj.mdev->mutex.
+ *
+ * Return: true if the pads are connected internally and false otherwise.
+ */
+bool media_entity_has_route(struct media_entity *entity, unsigned int pad0,
+			    unsigned int pad1);
+
 /**
  * media_graph_walk_cleanup - Release resources used by graph walk.
  *
-- 
2.25.1

[PATCH v10 16/38] media: entity: Use routing information during graph traversal

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Take internal routing information as reported by the entity has_route
operation into account during graph traversal to avoid following
unrelated links.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Michal Simek <michal.simek@xilinx.com>
Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 46 ++++++++++++++++++++++--------------
 1 file changed, 28 insertions(+), 18 deletions(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index a83f004efd37..58cdc9c6b342 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -248,15 +248,6 @@ bool media_entity_has_route(struct media_entity *entity, unsigned int pad0,
 }
 EXPORT_SYMBOL_GPL(media_entity_has_route);
 
-static struct media_pad *
-media_pad_other(struct media_pad *pad, struct media_link *link)
-{
-	if (link->source == pad)
-		return link->sink;
-	else
-		return link->source;
-}
-
 /* push an entity to traversal stack */
 static void stack_push(struct media_graph *graph, struct media_pad *pad)
 {
@@ -327,7 +318,8 @@ static void media_graph_walk_iter(struct media_graph *graph)
 {
 	struct media_pad *pad = stack_top(graph);
 	struct media_link *link;
-	struct media_pad *next;
+	struct media_pad *remote;
+	struct media_pad *local;
 
 	link = list_entry(link_top(graph), typeof(*link), list);
 
@@ -341,24 +333,42 @@ static void media_graph_walk_iter(struct media_graph *graph)
 		return;
 	}
 
-	/* Get the entity at the other end of the link. */
-	next = media_pad_other(pad, link);
+	/*
+	 * Get the local pad, the remote pad and the entity at the other
+	 * end of the link.
+	 */
+	if (link->source->entity == pad->entity) {
+		remote = link->sink;
+		local = link->source;
+	} else {
+		remote = link->source;
+		local = link->sink;
+	}
+
+	/*
+	 * Are the local pad and the pad we came from connected
+	 * internally in the entity ?
+	 */
+	if (!media_entity_has_route(pad->entity, pad->index, local->index)) {
+		link_top(graph) = link_top(graph)->next;
+		return;
+	}
 
 	/* Has the entity already been visited? */
-	if (media_entity_enum_test_and_set(&graph->ent_enum, next->entity)) {
+	if (media_entity_enum_test_and_set(&graph->ent_enum, remote->entity)) {
 		link_top(graph) = link_top(graph)->next;
 		dev_dbg(pad->graph_obj.mdev->dev,
 			"walk: skipping entity '%s' (already seen)\n",
-			next->entity->name);
+			remote->entity->name);
 		return;
 	}
 
 	/* Push the new entity to stack and start over. */
 	link_top(graph) = link_top(graph)->next;
-	stack_push(graph, next);
-	dev_dbg(next->graph_obj.mdev->dev, "walk: pushing '%s':%u on stack\n",
-		next->entity->name, next->index);
-	lockdep_assert_held(&next->graph_obj.mdev->graph_mutex);
+	stack_push(graph, remote);
+	dev_dbg(remote->graph_obj.mdev->dev, "walk: pushing '%s':%u on stack\n",
+		remote->entity->name, remote->index);
+	lockdep_assert_held(&remote->graph_obj.mdev->graph_mutex);
 }
 
 struct media_pad *media_graph_walk_next(struct media_graph *graph)
-- 
2.25.1

[PATCH v10 17/38] media: entity: Skip link validation for pads to which there is no route

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

Links are validated along the pipeline which is about to start streaming.
Not all the pads in entities that are traversed along that pipeline are
part of the pipeline, however. Skip the link validation for such pads.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 58cdc9c6b342..e963850b32df 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -489,6 +489,11 @@ __must_check int __media_pipeline_start(struct media_pad *pad,
 				link->sink->entity == entity ?
 				link->sink : link->source;
 
+			/* Ignore pads to which there is no route. */
+			if (!media_entity_has_route(entity, pad->index,
+						    other_pad->index))
+				continue;
+
 			/* Mark that a pad is connected by a link. */
 			bitmap_clear(has_no_links, other_pad->index, 1);
 
-- 
2.25.1

[PATCH v10 18/38] media: entity: Add an iterator helper for connected pads

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

Add a helper macro for iterating over pads that are connected through
enabled routes. This can be used to find all the connected pads within an
entity, for instance starting from the pad which has been obtained during
the graph walk.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>

- Make __media_entity_next_routed_pad() return NULL and adjust the
  iterator to handle that
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 14 ++++++++++++++
 include/media/media-entity.h | 27 +++++++++++++++++++++++++++
 2 files changed, 41 insertions(+)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index e963850b32df..072f017b399a 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -248,6 +248,20 @@ bool media_entity_has_route(struct media_entity *entity, unsigned int pad0,
 }
 EXPORT_SYMBOL_GPL(media_entity_has_route);
 
+struct media_pad *__media_entity_next_routed_pad(struct media_pad *root,
+						 struct media_pad *iter)
+{
+	struct media_entity *entity = root->entity;
+
+	for (; iter < &entity->pads[entity->num_pads]; iter++) {
+		if (media_entity_has_route(entity, root->index, iter->index))
+			return iter;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(__media_entity_next_routed_pad);
+
 /* push an entity to traversal stack */
 static void stack_push(struct media_graph *graph, struct media_pad *pad)
 {
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index b3069eef7fdb..f90cc6417d30 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -921,6 +921,33 @@ __must_check int media_graph_walk_init(
 bool media_entity_has_route(struct media_entity *entity, unsigned int pad0,
 			    unsigned int pad1);
 
+/**
+ * __media_entity_next_routed_pad - Get next pad connected to @root
+ *
+ * @root: The root pad to which the iterated pads have a route
+ * @iter: The iterator pad
+ *
+ * Get next pad which has a route to @root.
+ */
+struct media_pad *__media_entity_next_routed_pad(struct media_pad *root,
+						 struct media_pad *iter);
+
+/**
+ * media_entity_for_each_routed_pad - Iterate over entity pads connected by
+ *				      routes
+ *
+ * @root: The root pad to which the iterated pads have a route
+ * @iter: The iterator pad
+ *
+ * Iterate over all pads of an entity which have an internal route to @root pad.
+ * The iteration will include the @root pad itself.
+ */
+#define media_entity_for_each_routed_pad(root, iter)                           \
+	for (iter = __media_entity_next_routed_pad(root,                       \
+						   (root)->entity->pads);      \
+	     iter != NULL;                                                     \
+	     iter = __media_entity_next_routed_pad(root, iter + 1))
+
 /**
  * media_graph_walk_cleanup - Release resources used by graph walk.
  *
-- 
2.25.1

[PATCH v10 19/38] media: entity: Add only connected pads to the pipeline

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

A single entity may contain multiple pipelines. Only add pads that were
connected to the pad through which the entity was reached to the pipeline.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 072f017b399a..4eb4b94c09e2 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -474,7 +474,7 @@ __must_check int __media_pipeline_start(struct media_pad *pad,
 
 		ret = 0;
 
-		media_entity_for_each_pad(entity, iter) {
+		media_entity_for_each_routed_pad(pad, iter) {
 			if (iter->pipe && iter->pipe != pipe) {
 				pr_err("Pipe active for %s. Can't start for %s\n",
 				       entity->name, iter->entity->name);
@@ -563,10 +563,9 @@ __must_check int __media_pipeline_start(struct media_pad *pad,
 	media_graph_walk_start(graph, pad_err);
 
 	while ((pad_err = media_graph_walk_next(graph))) {
-		struct media_entity *entity = pad_err->entity;
 		struct media_pad *iter;
 
-		media_entity_for_each_pad(entity, iter) {
+		media_entity_for_each_routed_pad(pad_err, iter) {
 			/* Sanity check for negative stream_count */
 			if (!WARN_ON_ONCE(iter->stream_count <= 0)) {
 				--iter->stream_count;
@@ -619,10 +618,9 @@ void __media_pipeline_stop(struct media_pad *pad)
 	media_graph_walk_start(graph, pad);
 
 	while ((pad = media_graph_walk_next(graph))) {
-		struct media_entity *entity = pad->entity;
 		struct media_pad *iter;
 
-		media_entity_for_each_pad(entity, iter) {
+		media_entity_for_each_routed_pad(pad, iter) {
 			/* Sanity check for negative stream_count */
 			if (!WARN_ON_ONCE(iter->stream_count <= 0)) {
 				iter->stream_count--;
-- 
2.25.1

[PATCH v10 20/38] media: entity: Add debug information in graph walk route check

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

Add debug printout in graph walk route check.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 4eb4b94c09e2..663773f6bb2f 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -365,6 +365,9 @@ static void media_graph_walk_iter(struct media_graph *graph)
 	 */
 	if (!media_entity_has_route(pad->entity, pad->index, local->index)) {
 		link_top(graph) = link_top(graph)->next;
+		dev_dbg(pad->graph_obj.mdev->dev,
+			"walk: skipping \"%s\":%u -> %u (no route)\n",
+			pad->entity->name, pad->index, local->index);
 		return;
 	}
 
-- 
2.25.1

[PATCH v10 21/38] media: Add bus type to frame descriptors

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

Add the media bus type to the frame descriptor. CSI-2 specific
information will be added in next patch to the frame descriptor.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>

- Make the bus type a named enum
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
---
 include/media/v4l2-subdev.h | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 873bbe0686e3..03e58fc35092 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -344,12 +344,32 @@ struct v4l2_mbus_frame_desc_entry {
 
 #define V4L2_FRAME_DESC_ENTRY_MAX	4
 
+/**
+ * enum v4l2_mbus_frame_desc_type - media bus frame description type
+ *
+ * @V4L2_MBUS_FRAME_DESC_TYPE_UNDEFINED:
+ *	Undefined frame desc type. Drivers should not use this, it is
+ *	for backwards compatibility.
+ * @V4L2_MBUS_FRAME_DESC_TYPE_PARALLEL:
+ *	Parallel media bus.
+ * @V4L2_MBUS_FRAME_DESC_TYPE_CSI2:
+ *	CSI-2 media bus. Frame desc parameters must be set in
+ *	&struct v4l2_mbus_frame_desc_entry->csi2.
+ */
+enum v4l2_mbus_frame_desc_type {
+	V4L2_MBUS_FRAME_DESC_TYPE_UNDEFINED = 0,
+	V4L2_MBUS_FRAME_DESC_TYPE_PARALLEL,
+	V4L2_MBUS_FRAME_DESC_TYPE_CSI2,
+};
+
 /**
  * struct v4l2_mbus_frame_desc - media bus data frame description
+ * @type: type of the bus (enum v4l2_mbus_frame_desc_type)
  * @entry: frame descriptors array
  * @num_entries: number of entries in @entry array
  */
 struct v4l2_mbus_frame_desc {
+	enum v4l2_mbus_frame_desc_type type;
 	struct v4l2_mbus_frame_desc_entry entry[V4L2_FRAME_DESC_ENTRY_MAX];
 	unsigned short num_entries;
 };
-- 
2.25.1

[PATCH v10 22/38] media: Add CSI-2 bus configuration to frame descriptors

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

Add CSI-2 bus specific configuration to the frame descriptors. This allows
obtaining the virtual channel and data type information for each stream
the transmitter is sending.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
---
 include/media/v4l2-subdev.h | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 03e58fc35092..30aa622c8cb9 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -312,6 +312,17 @@ struct v4l2_subdev_audio_ops {
 	int (*s_stream)(struct v4l2_subdev *sd, int enable);
 };
 
+/**
+ * struct v4l2_mbus_frame_desc_entry_csi2
+ *
+ * @vc: CSI-2 virtual channel
+ * @dt: CSI-2 data type ID
+ */
+struct v4l2_mbus_frame_desc_entry_csi2 {
+	u8 vc;
+	u8 dt;
+};
+
 /**
  * enum v4l2_mbus_frame_desc_flags - media bus frame description flags
  *
@@ -335,11 +346,16 @@ enum v4l2_mbus_frame_desc_flags {
  *		%FRAME_DESC_FL_BLOB is not set.
  * @length:	number of octets per frame, valid if @flags
  *		%V4L2_MBUS_FRAME_DESC_FL_LEN_MAX is set.
+ * @bus:	Bus-specific frame descriptor parameters
+ * @bus.csi2:	CSI-2-specific bus configuration
  */
 struct v4l2_mbus_frame_desc_entry {
 	enum v4l2_mbus_frame_desc_flags flags;
 	u32 pixelcode;
 	u32 length;
+	union {
+		struct v4l2_mbus_frame_desc_entry_csi2 csi2;
+	} bus;
 };
 
 #define V4L2_FRAME_DESC_ENTRY_MAX	4
-- 
2.25.1

[PATCH v10 23/38] media: Add stream to frame descriptor

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

The stream field identifies the stream this frame descriptor applies to in
routing configuration across a multiplexed link.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 include/media/v4l2-subdev.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 30aa622c8cb9..2921885eb390 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -342,6 +342,7 @@ enum v4l2_mbus_frame_desc_flags {
  * struct v4l2_mbus_frame_desc_entry - media bus frame description structure
  *
  * @flags:	bitmask flags, as defined by &enum v4l2_mbus_frame_desc_flags.
+ * @stream:	stream in routing configuration
  * @pixelcode:	media bus pixel code, valid if @flags
  *		%FRAME_DESC_FL_BLOB is not set.
  * @length:	number of octets per frame, valid if @flags
@@ -351,6 +352,7 @@ enum v4l2_mbus_frame_desc_flags {
  */
 struct v4l2_mbus_frame_desc_entry {
 	enum v4l2_mbus_frame_desc_flags flags;
+	u32 stream;
 	u32 pixelcode;
 	u32 length;
 	union {
-- 
2.25.1

[PATCH v10 24/38] media: subdev: increase V4L2_FRAME_DESC_ENTRY_MAX to 8

[Tomi Valkeinen]

V4L2_FRAME_DESC_ENTRY_MAX is currently set to 4. In theory it's possible
to have an arbitrary amount of streams in a single pad, so preferably
there should be no hardcoded maximum number.

However, I believe a reasonable max is 8, which would cover a CSI-2 pad
with 4 streams of pixel data and 4 streams of metadata.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
---
 include/media/v4l2-subdev.h | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 2921885eb390..a82fc74f4646 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -360,7 +360,11 @@ struct v4l2_mbus_frame_desc_entry {
 	} bus;
 };
 
-#define V4L2_FRAME_DESC_ENTRY_MAX	4
+ /*
+  * FIXME: If this number is too small, it should be dropped altogether and the
+  * API switched to a dynamic number of frame descriptor entries.
+  */
+#define V4L2_FRAME_DESC_ENTRY_MAX	8
 
 /**
  * enum v4l2_mbus_frame_desc_type - media bus frame description type
-- 
2.25.1

[PATCH v10 25/38] media: add V4L2_SUBDEV_FL_MULTIPLEXED

[Tomi Valkeinen]

Add subdev flag V4L2_SUBDEV_FL_MULTIPLEXED. It is used to indicate that
the subdev supports the new API with multiplexed streams (routing,
stream configs).

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 include/media/v4l2-subdev.h | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index a82fc74f4646..45861bcdccf5 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -892,6 +892,17 @@ struct v4l2_subdev_internal_ops {
  * should set this flag.
  */
 #define V4L2_SUBDEV_FL_HAS_EVENTS		(1U << 3)
+/*
+ * Set this flag if this subdev supports multiplexed streams. This means
+ * that the driver supports routing and handles the stream parameter in its
+ * v4l2_subdev_pad_ops handlers. More specifically, this means:
+ *
+ * - Centrally managed active state is enabled
+ * - Legacy pad config is _not_ supported (state->pads)
+ * - Routing ioctls are available
+ * - Multiple streams per pad are supported
+ */
+#define V4L2_SUBDEV_FL_MULTIPLEXED		(1U << 4)
 
 struct regulator_bulk_data;
 
-- 
2.25.1

[PATCH v10 26/38] media: add V4L2_SUBDEV_CAP_MPLEXED

[Tomi Valkeinen]

Add a subdev capability flag to expose to userspace if a subdev supports
multiplexed streams.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 4 +++-
 include/uapi/linux/v4l2-subdev.h      | 3 +++
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 2053fe1cd67d..721148e35624 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -419,7 +419,9 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 
 		memset(cap->reserved, 0, sizeof(cap->reserved));
 		cap->version = LINUX_VERSION_CODE;
-		cap->capabilities = ro_subdev ? V4L2_SUBDEV_CAP_RO_SUBDEV : 0;
+		cap->capabilities =
+			(ro_subdev ? V4L2_SUBDEV_CAP_RO_SUBDEV : 0) |
+			((sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED) ? V4L2_SUBDEV_CAP_MPLEXED : 0);
 
 		return 0;
 	}
diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
index 658106f5b5dc..d91ab6f22fa7 100644
--- a/include/uapi/linux/v4l2-subdev.h
+++ b/include/uapi/linux/v4l2-subdev.h
@@ -188,6 +188,9 @@ struct v4l2_subdev_capability {
 /* The v4l2 sub-device video device node is registered in read-only mode. */
 #define V4L2_SUBDEV_CAP_RO_SUBDEV		0x00000001
 
+/* The v4l2 sub-device supports multiplexed streams. */
+#define V4L2_SUBDEV_CAP_MPLEXED			0x00000002
+
 /* Backwards compatibility define --- to be removed */
 #define v4l2_subdev_edid v4l2_edid
 
-- 
2.25.1

[PATCH v10 27/38] media: Documentation: Add GS_ROUTING documentation

[Tomi Valkeinen]

From: Jacopo Mondi <jacopo+renesas@jmondi.org>

Add documentation for VIDIOC_SUBDEV_G/S_ROUTING ioctl and add
description of multiplexed media pads and internal routing to the
V4L2-subdev documentation section.

Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
---
 .../userspace-api/media/v4l/dev-subdev.rst    |   2 +
 .../userspace-api/media/v4l/user-func.rst     |   1 +
 .../media/v4l/vidioc-subdev-g-routing.rst     | 150 ++++++++++++++++++
 3 files changed, 153 insertions(+)
 create mode 100644 Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst

diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index fd1de0a73a9f..a67c2749089a 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -29,6 +29,8 @@ will feature a character device node on which ioctls can be called to
 
 -  negotiate image formats on individual pads
 
+-  inspect and modify internal data routing between pads of the same entity
+
 Sub-device character device nodes, conventionally named
 ``/dev/v4l-subdev*``, use major number 81.
 
diff --git a/Documentation/userspace-api/media/v4l/user-func.rst b/Documentation/userspace-api/media/v4l/user-func.rst
index 53e604bd7d60..228c1521f190 100644
--- a/Documentation/userspace-api/media/v4l/user-func.rst
+++ b/Documentation/userspace-api/media/v4l/user-func.rst
@@ -70,6 +70,7 @@ Function Reference
     vidioc-subdev-g-crop
     vidioc-subdev-g-fmt
     vidioc-subdev-g-frame-interval
+    vidioc-subdev-g-routing
     vidioc-subdev-g-selection
     vidioc-subdev-querycap
     vidioc-subscribe-event
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
new file mode 100644
index 000000000000..a0d9c79e162f
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
@@ -0,0 +1,150 @@
+.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
+.. c:namespace:: V4L
+
+.. _VIDIOC_SUBDEV_G_ROUTING:
+
+******************************************************
+ioctl VIDIOC_SUBDEV_G_ROUTING, VIDIOC_SUBDEV_S_ROUTING
+******************************************************
+
+Name
+====
+
+VIDIOC_SUBDEV_G_ROUTING - VIDIOC_SUBDEV_S_ROUTING - Get or set routing between streams of media pads in a media entity.
+
+
+Synopsis
+========
+
+.. c:function:: int ioctl( int fd, VIDIOC_SUBDEV_G_ROUTING, struct v4l2_subdev_routing *argp )
+    :name: VIDIOC_SUBDEV_G_ROUTING
+
+.. c:function:: int ioctl( int fd, VIDIOC_SUBDEV_S_ROUTING, struct v4l2_subdev_routing *argp )
+    :name: VIDIOC_SUBDEV_S_ROUTING
+
+
+Arguments
+=========
+
+``fd``
+    File descriptor returned by :ref:`open() <func-open>`.
+
+``argp``
+    Pointer to struct :c:type:`v4l2_subdev_routing`.
+
+
+Description
+===========
+
+These ioctls are used to get and set the routing in a media entity.
+The routing configuration determines the flows of data inside an entity.
+
+Drivers report their current routing tables using the
+``VIDIOC_SUBDEV_G_ROUTING`` ioctl and application may enable or disable routes
+with the ``VIDIOC_SUBDEV_S_ROUTING`` ioctl, by adding or removing routes and
+setting or clearing flags of the  ``flags`` field of a
+struct :c:type:`v4l2_subdev_route`.
+
+All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called. This
+means that the userspace mut reconfigure all streams after calling the ioctl
+with e.g. ``VIDIOC_SUBDEV_S_FMT``.
+
+A special case for routing are routes marked with
+``V4L2_SUBDEV_ROUTE_FL_SOURCE`` flag. These routes are used to describe
+source endpoints on sensors and the sink fields are unused.
+
+When inspecting routes through ``VIDIOC_SUBDEV_G_ROUTING`` and the application
+provided ``num_routes`` is not big enough to contain all the available routes
+the subdevice exposes, drivers return the ENOSPC error code and adjust the
+value of the ``num_routes`` field. Application should then reserve enough memory
+for all the route entries and call ``VIDIOC_SUBDEV_G_ROUTING`` again.
+
+.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
+
+.. c:type:: v4l2_subdev_routing
+
+.. flat-table:: struct v4l2_subdev_routing
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u32
+      - ``which``
+      - Format to modified, from enum
+        :ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
+    * - struct :c:type:`v4l2_subdev_route`
+      - ``routes[]``
+      - Array of struct :c:type:`v4l2_subdev_route` entries
+    * - __u32
+      - ``num_routes``
+      - Number of entries of the routes array
+    * - __u32
+      - ``reserved``\ [5]
+      - Reserved for future extensions. Applications and drivers must set
+	the array to zero.
+
+.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
+
+.. c:type:: v4l2_subdev_route
+
+.. flat-table:: struct v4l2_subdev_route
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u32
+      - ``sink_pad``
+      - Sink pad number.
+    * - __u32
+      - ``sink_stream``
+      - Sink pad stream number.
+    * - __u32
+      - ``source_pad``
+      - Source pad number.
+    * - __u32
+      - ``source_stream``
+      - Source pad stream number.
+    * - __u32
+      - ``flags``
+      - Route enable/disable flags
+	:ref:`v4l2_subdev_routing_flags <v4l2-subdev-routing-flags>`.
+    * - __u32
+      - ``reserved``\ [5]
+      - Reserved for future extensions. Applications and drivers must set
+	the array to zero.
+
+.. tabularcolumns:: |p{6.6cm}|p{2.2cm}|p{8.7cm}|
+
+.. _v4l2-subdev-routing-flags:
+
+.. flat-table:: enum v4l2_subdev_routing_flags
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       3 1 4
+
+    * - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - 0
+      - The route is enabled. Set by applications.
+    * - V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
+      - 1
+      - The route is immutable. Set by the driver.
+    * - V4L2_SUBDEV_ROUTE_FL_SOURCE
+      - 2
+      - The route is a source route, and the ``sink_pad`` and ``sink_stream``
+        fields are unused. Set by the driver.
+
+Return Value
+============
+
+On success 0 is returned, on error -1 and the ``errno`` variable is set
+appropriately. The generic error codes are described at the
+:ref:`Generic Error Codes <gen-errors>` chapter.
+
+ENOSPC
+   The number of provided route entries is less than the available ones.
+
+EINVAL
+   The sink or source pad identifiers reference a non-existing pad, or reference
+   pads of different types (ie. the sink_pad identifiers refers to a source pad)
+   or the sink or source stream identifiers reference a non-existing stream on
+   the sink or source pad.
-- 
2.25.1

[PATCH v10 28/38] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Add support for subdev internal routing. A route is defined as a single
stream from a sink pad to a source pad.

The userspace can configure the routing via two new ioctls,
VIDIOC_SUBDEV_G_ROUTING and VIDIOC_SUBDEV_S_ROUTING, and subdevs can
implement the functionality with v4l2_subdev_pad_ops.set_routing().

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Michal Simek <michal.simek@xilinx.com>

- Add sink and source streams for multiplexed links
- Copy the argument back in case of an error. This is needed to let the
  caller know the number of routes.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>

- Expand and refine documentation.
- Make the 'routes' pointer a __u64 __user pointer so that a compat32
  version of the ioctl is not required.
- Add struct v4l2_subdev_krouting to be used for subdevice operations.

Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>

- Fix typecasing warnings
- Check sink & source pad types
- Add 'which' field
- Add V4L2_SUBDEV_ROUTE_FL_SOURCE
- Routing to subdev state
- Dropped get_routing subdev op

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-ioctl.c  | 25 ++++++++-
 drivers/media/v4l2-core/v4l2-subdev.c | 75 +++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 22 ++++++++
 include/uapi/linux/v4l2-subdev.h      | 57 ++++++++++++++++++++
 4 files changed, 178 insertions(+), 1 deletion(-)

diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index 05d5db3d85e5..8e9315ffcb99 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -16,6 +16,7 @@
 #include <linux/kernel.h>
 #include <linux/version.h>
 
+#include <linux/v4l2-subdev.h>
 #include <linux/videodev2.h>
 
 #include <media/v4l2-common.h>
@@ -3065,6 +3066,21 @@ static int check_array_args(unsigned int cmd, void *parg, size_t *array_size,
 		ret = 1;
 		break;
 	}
+
+	case VIDIOC_SUBDEV_G_ROUTING:
+	case VIDIOC_SUBDEV_S_ROUTING: {
+		struct v4l2_subdev_routing *routing = parg;
+
+		if (routing->num_routes > 256)
+			return -EINVAL;
+
+		*user_ptr = u64_to_user_ptr(routing->routes);
+		*kernel_ptr = (void **)&routing->routes;
+		*array_size = sizeof(struct v4l2_subdev_route)
+			    * routing->num_routes;
+		ret = 1;
+		break;
+	}
 	}
 
 	return ret;
@@ -3328,8 +3344,15 @@ video_usercopy(struct file *file, unsigned int orig_cmd, unsigned long arg,
 	/*
 	 * Some ioctls can return an error, but still have valid
 	 * results that must be returned.
+	 *
+	 * FIXME: subdev IOCTLS are partially handled here and partially in
+	 * v4l2-subdev.c and the 'always_copy' flag can only be set for IOCTLS
+	 * defined here as part of the 'v4l2_ioctls' array. As
+	 * VIDIOC_SUBDEV_G_ROUTING needs to return results to applications even
+	 * in case of failure, but it is not defined here as part of the
+	 * 'v4l2_ioctls' array, insert an ad-hoc check to address that.
 	 */
-	if (err < 0 && !always_copy)
+	if (err < 0 && !always_copy && cmd != VIDIOC_SUBDEV_G_ROUTING)
 		goto out;
 
 out_array_args:
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 721148e35624..06e234e0aa98 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -394,6 +394,12 @@ subdev_ioctl_get_state(struct v4l2_subdev *sd, struct v4l2_subdev_fh *subdev_fh,
 		which = ((struct v4l2_subdev_selection *)arg)->which;
 		break;
 	}
+
+	case VIDIOC_SUBDEV_G_ROUTING:
+	case VIDIOC_SUBDEV_S_ROUTING: {
+		which = ((struct v4l2_subdev_routing *)arg)->which;
+		break;
+	}
 	}
 
 	return which == V4L2_SUBDEV_FORMAT_TRY ?
@@ -712,6 +718,74 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 	case VIDIOC_SUBDEV_QUERYSTD:
 		return v4l2_subdev_call(sd, video, querystd, arg);
 
+	case VIDIOC_SUBDEV_G_ROUTING: {
+		struct v4l2_subdev_routing *routing = arg;
+		struct v4l2_subdev_krouting *krouting;
+
+		if (!(sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED))
+			return -ENOIOCTLCMD;
+
+		memset(routing->reserved, 0, sizeof(routing->reserved));
+
+		krouting = &state->routing;
+
+		if (routing->num_routes < krouting->num_routes) {
+			routing->num_routes = krouting->num_routes;
+			return -ENOSPC;
+		}
+
+		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
+		       krouting->routes,
+		       krouting->num_routes * sizeof(*krouting->routes));
+		routing->num_routes = krouting->num_routes;
+
+		return 0;
+	}
+
+	case VIDIOC_SUBDEV_S_ROUTING: {
+		struct v4l2_subdev_routing *routing = arg;
+		struct v4l2_subdev_route *routes =
+			(struct v4l2_subdev_route *)(uintptr_t)routing->routes;
+		struct v4l2_subdev_krouting krouting = {};
+		unsigned int i;
+
+		if (!(sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED))
+			return -ENOIOCTLCMD;
+
+		if (routing->which != V4L2_SUBDEV_FORMAT_TRY && ro_subdev)
+			return -EPERM;
+
+		memset(routing->reserved, 0, sizeof(routing->reserved));
+
+		for (i = 0; i < routing->num_routes; ++i) {
+			const struct v4l2_subdev_route *route = &routes[i];
+			const struct media_pad *pads = sd->entity.pads;
+
+			/* Do not check sink pad for source routes */
+			if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE)) {
+				if (route->sink_pad >= sd->entity.num_pads)
+					return -EINVAL;
+
+				if (!(pads[route->sink_pad].flags &
+				      MEDIA_PAD_FL_SINK))
+					return -EINVAL;
+			}
+
+			if (route->source_pad >= sd->entity.num_pads)
+				return -EINVAL;
+
+			if (!(pads[route->source_pad].flags &
+			      MEDIA_PAD_FL_SOURCE))
+				return -EINVAL;
+		}
+
+		krouting.num_routes = routing->num_routes;
+		krouting.routes = routes;
+
+		return v4l2_subdev_call(sd, pad, set_routing, state,
+					routing->which, &krouting);
+	}
+
 	default:
 		return v4l2_subdev_call(sd, core, ioctl, cmd, arg);
 	}
@@ -972,6 +1046,7 @@ void __v4l2_subdev_state_free(struct v4l2_subdev_state *state)
 
 	mutex_destroy(&state->lock);
 
+	kfree(state->routing.routes);
 	kvfree(state->pads);
 	kfree(state);
 }
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 45861bcdccf5..946455cc1f01 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -697,11 +697,25 @@ struct v4l2_subdev_pad_config {
 	struct v4l2_rect try_compose;
 };
 
+/**
+ * struct v4l2_subdev_krouting - subdev routing table
+ *
+ * @num_routes: number of routes
+ * @routes: &struct v4l2_subdev_route
+ *
+ * This structure contains the routing table for a subdev.
+ */
+struct v4l2_subdev_krouting {
+	unsigned int num_routes;
+	struct v4l2_subdev_route *routes;
+};
+
 /**
  * struct v4l2_subdev_state - Used for storing subdev state information.
  *
  * @lock: mutex for the state
  * @pads: &struct v4l2_subdev_pad_config array
+ * @routing: routing table for the subdev
  *
  * This structure only needs to be passed to the pad op if the 'which' field
  * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
@@ -711,6 +725,7 @@ struct v4l2_subdev_state {
 	/* lock for the struct v4l2_subdev_state fields */
 	struct mutex lock;
 	struct v4l2_subdev_pad_config *pads;
+	struct v4l2_subdev_krouting routing;
 };
 
 /**
@@ -774,6 +789,9 @@ struct v4l2_subdev_state {
  *		     applied to the hardware. The operation shall fail if the
  *		     pad index it has been called on is not valid or in case of
  *		     unrecoverable failures.
+ *
+ * @set_routing: enable or disable data connection routes described in the
+ *		 subdevice routing table.
  */
 struct v4l2_subdev_pad_ops {
 	int (*init_cfg)(struct v4l2_subdev *sd,
@@ -818,6 +836,10 @@ struct v4l2_subdev_pad_ops {
 			       struct v4l2_mbus_config *config);
 	int (*set_mbus_config)(struct v4l2_subdev *sd, unsigned int pad,
 			       struct v4l2_mbus_config *config);
+	int (*set_routing)(struct v4l2_subdev *sd,
+			   struct v4l2_subdev_state *state,
+			   enum v4l2_subdev_format_whence which,
+			   struct v4l2_subdev_krouting *route);
 };
 
 /**
diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
index d91ab6f22fa7..1ec3141bf860 100644
--- a/include/uapi/linux/v4l2-subdev.h
+++ b/include/uapi/linux/v4l2-subdev.h
@@ -191,6 +191,61 @@ struct v4l2_subdev_capability {
 /* The v4l2 sub-device supports multiplexed streams. */
 #define V4L2_SUBDEV_CAP_MPLEXED			0x00000002
 
+/*
+ * Is the route active? An active route will start when streaming is enabled
+ * on a video node.
+ */
+#define V4L2_SUBDEV_ROUTE_FL_ACTIVE		_BITUL(0)
+
+/*
+ * Is the route immutable, i.e. can it be activated and inactivated?
+ * Set by the driver.
+ */
+#define V4L2_SUBDEV_ROUTE_FL_IMMUTABLE		_BITUL(1)
+
+/*
+ * Is the route a source endpoint? A source endpoint route refers to a stream
+ * generated internally by the subdevice (usually a sensor), and thus there
+ * is no sink-side endpoint for the route. The sink_pad and sink_stream
+ * fields are unused.
+ * Set by the driver.
+ */
+#define V4L2_SUBDEV_ROUTE_FL_SOURCE		_BITUL(2)
+
+/**
+ * struct v4l2_subdev_route - A route inside a subdev
+ *
+ * @sink_pad: the sink pad index
+ * @sink_stream: the sink stream identifier
+ * @source_pad: the source pad index
+ * @source_stream: the source stream identifier
+ * @flags: route flags V4L2_SUBDEV_ROUTE_FL_*
+ * @reserved: drivers and applications must zero this array
+ */
+struct v4l2_subdev_route {
+	__u32 sink_pad;
+	__u32 sink_stream;
+	__u32 source_pad;
+	__u32 source_stream;
+	__u32 flags;
+	__u32 reserved[5];
+};
+
+/**
+ * struct v4l2_subdev_routing - Subdev routing information
+ *
+ * @which: configuration type (from enum v4l2_subdev_format_whence)
+ * @num_routes: the total number of routes in the routes array
+ * @routes: pointer to the routes array
+ * @reserved: drivers and applications must zero this array
+ */
+struct v4l2_subdev_routing {
+	__u32 which;
+	__u32 num_routes;
+	__u64 routes;
+	__u32 reserved[6];
+};
+
 /* Backwards compatibility define --- to be removed */
 #define v4l2_subdev_edid v4l2_edid
 
@@ -206,6 +261,8 @@ struct v4l2_subdev_capability {
 #define VIDIOC_SUBDEV_S_CROP			_IOWR('V', 60, struct v4l2_subdev_crop)
 #define VIDIOC_SUBDEV_G_SELECTION		_IOWR('V', 61, struct v4l2_subdev_selection)
 #define VIDIOC_SUBDEV_S_SELECTION		_IOWR('V', 62, struct v4l2_subdev_selection)
+#define VIDIOC_SUBDEV_G_ROUTING			_IOWR('V', 38, struct v4l2_subdev_routing)
+#define VIDIOC_SUBDEV_S_ROUTING			_IOWR('V', 39, struct v4l2_subdev_routing)
 /* The following ioctls are identical to the ioctls in videodev2.h */
 #define VIDIOC_SUBDEV_G_STD			_IOR('V', 23, v4l2_std_id)
 #define VIDIOC_SUBDEV_S_STD			_IOW('V', 24, v4l2_std_id)
-- 
2.25.1

[PATCH v10 29/38] media: subdev: add v4l2_subdev_has_route()

[Tomi Valkeinen]

Add a v4l2_subdev_has_route() helper function which can be used for
media_entity_operations.has_route op.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 31 +++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 16 ++++++++++++++
 2 files changed, 47 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 06e234e0aa98..67f3c14d7398 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1000,6 +1000,37 @@ int v4l2_subdev_link_validate(struct media_link *link)
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
 
+bool v4l2_subdev_has_route(struct media_entity *entity, unsigned int pad0,
+			   unsigned int pad1)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct v4l2_subdev_krouting *routing;
+	unsigned int i;
+	struct v4l2_subdev_state *state;
+
+	state = v4l2_subdev_lock_active_state(sd);
+
+	routing = &state->routing;
+
+	for (i = 0; i < routing->num_routes; ++i) {
+		struct v4l2_subdev_route *route = &routing->routes[i];
+
+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
+			continue;
+
+		if ((route->sink_pad == pad0 && route->source_pad == pad1) ||
+		    (route->source_pad == pad0 && route->sink_pad == pad1)) {
+			v4l2_subdev_unlock_state(state);
+			return true;
+		}
+	}
+
+	v4l2_subdev_unlock_state(state);
+
+	return false;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_has_route);
+
 struct v4l2_subdev_state *
 __v4l2_subdev_state_alloc(struct v4l2_subdev *sd, const char *lock_name,
 			  struct lock_class_key *lock_key)
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 946455cc1f01..350ee4d9600f 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1230,6 +1230,22 @@ int v4l2_subdev_link_validate_default(struct v4l2_subdev *sd,
  */
 int v4l2_subdev_link_validate(struct media_link *link);
 
+/**
+ * v4l2_subdev_has_route - MC has_route implementation for subdevs
+ *
+ * @entity: pointer to &struct media_entity
+ * @pad0: pad number for the first pad
+ * @pad1: pad number for the second pad
+ *
+ * This function looks at the routing in subdev's active state and returns if
+ * there is a route connecting pad0 and pad1.
+ *
+ * This function can be used as implementation for
+ * media_entity_operations.has_route.
+ */
+bool v4l2_subdev_has_route(struct media_entity *entity, unsigned int pad0,
+			   unsigned int pad1);
+
 /**
  * __v4l2_subdev_state_alloc - allocate v4l2_subdev_state
  *
-- 
2.25.1

[PATCH v10 30/38] media: subdev: add v4l2_subdev_set_routing helper()

[Tomi Valkeinen]

Add a helper function to set the subdev routing. The helper can be used
from subdev driver's set_routing op to store the routing table.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 27 +++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 16 ++++++++++++++++
 2 files changed, 43 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 67f3c14d7398..633e1fccd377 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1153,3 +1153,30 @@ void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state)
 	mutex_unlock(&state->lock);
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_unlock_state);
+
+int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_state *state,
+			    struct v4l2_subdev_krouting *routing)
+{
+	struct v4l2_subdev_krouting *dst = &state->routing;
+	const struct v4l2_subdev_krouting *src = routing;
+
+	lockdep_assert_held(&state->lock);
+
+	kfree(dst->routes);
+	dst->routes = NULL;
+	dst->num_routes = 0;
+
+	if (src->num_routes > 0) {
+		dst->routes = kmemdup(src->routes,
+				      src->num_routes * sizeof(*src->routes),
+				      GFP_KERNEL);
+		if (!dst->routes)
+			return -ENOMEM;
+
+		dst->num_routes = src->num_routes;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing);
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 350ee4d9600f..8525f39c2c67 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1439,4 +1439,20 @@ v4l2_subdev_lock_and_return_state(struct v4l2_subdev *sd,
 	return state;
 }
 
+/**
+ * v4l2_subdev_set_routing() - Set given routing to subdev state
+ * @sd: The subdevice
+ * @state: The subdevice state
+ * @routing: Routing that will be copied to subdev state
+ *
+ * This will release old routing table (if any) from the state, allocate
+ * enough space for the given routing, and copy the routing.
+ *
+ * This can be used from the subdev driver's set_routing op, after validating
+ * the routing.
+ */
+int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_state *state,
+			    struct v4l2_subdev_krouting *routing);
+
 #endif
-- 
2.25.1

[PATCH v10 31/38] media: Documentation: add multiplexed streams documentation

[Tomi Valkeinen]

Add documentation related to multiplexed streams.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 .../driver-api/media/v4l2-subdev.rst          |   8 +
 .../userspace-api/media/v4l/dev-subdev.rst    | 165 ++++++++++++++++++
 2 files changed, 173 insertions(+)

diff --git a/Documentation/driver-api/media/v4l2-subdev.rst b/Documentation/driver-api/media/v4l2-subdev.rst
index db012729f8e3..1d986e814b4e 100644
--- a/Documentation/driver-api/media/v4l2-subdev.rst
+++ b/Documentation/driver-api/media/v4l2-subdev.rst
@@ -546,6 +546,14 @@ subdev drivers. In these cases the called subdev ops must also handle the NULL
 case. This can be easily managed by the use of
 v4l2_subdev_validate_and_lock_state() helper.
 
+Streams, multiplexed media pads and internal routing
+----------------------------------------------------
+
+A subdevice driver can implement support for multiplexed streams by setting
+the V4L2_SUBDEV_FL_MULTIPLEXED subdev flag and implementing support for
+centrally managed subdev active state, routing and stream based
+configuration.
+
 V4L2 sub-device functions and data structures
 ---------------------------------------------
 
diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index a67c2749089a..fd042afeddd6 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -503,3 +503,168 @@ source pads.
     :maxdepth: 1
 
     subdev-formats
+
+Streams, multiplexed media pads and internal routing
+----------------------------------------------------
+
+Commonly V4L2 subdevices support only separate video streams, that is, each
+link in the media graph and each pad in a subdevice pass through a single
+video stream. Thus each pad contains a format configuration for that single
+stream. In some cases a subdev can do stream processing and split a stream
+into two or compose two streams into one, but the inputs and outputs for the
+subdev are still a single stream per pad.
+
+Some hardware, e.g. MIPI CSI-2, support multiplexed streams, that is, a single
+bus carries multiple streams. Thus a camera could output two streams, a pixel
+stream and a metadata stream, and a bridge subdev could route the streams
+from multiple input pads into a single output pad.
+
+Subdevice drivers that support multiplexed streams are compatible with
+non-multiplexed subdev drivers, but, of course, require a routing configuration
+where the link between those two types of drivers contains only a single
+stream.
+
+Understanding streams
+^^^^^^^^^^^^^^^^^^^^^
+
+A stream is a stream of content (e.g. pixel data or metadata) flowing through
+the media pipeline from a source (e.g. a sensor) towards the final sink(e.g. a
+receiver and demultiplexer in a SoC). Each media link carries all the streams
+from one end of the link to the other, and subdevices have routing tables which
+describe how the incoming streams from sink pads are routed to the source
+pads.
+
+A stream ID (often just "stream") is a media link-local identifier for a
+stream. In other words, a configuration for a particular stream ID must exist
+on both sides of a media link, but another stream ID can be used for the same
+stream at the other side of the subdevice.
+
+A stream at a specific point in the media pipeline is identified with the
+subdev and a (pad, stream) pair. For subdevices that do not support
+multiplexed streams the 'stream' is always 0.
+
+Configuring streams
+^^^^^^^^^^^^^^^^^^^
+
+The configuration of the streams is done individually for each subdevice and
+the validity of the streams between subdevices is validated when the pipeline
+is started.
+
+There are three steps in configuring the streams:
+
+1) Set up links. Connect the pads between subdevices using the :ref:`Media
+Controller API <media_controller>`
+
+2) Routing. The routing table for the subdevice must be set with
+:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that
+setting the routing table will reset all the stream configurations.
+
+3) Configure streams. Each route endpoint must be configured
+with :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>`.
+
+Multiplexed streams setup example
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A simple example of a multiplexed stream setup might be as follows:
+
+- Two identical sensors (Sensor A and Sensor B). Each sensor has a single
+  source pad (pad 0), and outputs two streams, pixel data and metadata.
+
+- Multiplexer bridge (Bridge). The bridge has two sink pads, connected to the
+  sensors (pads 0, 1), and one source pad (pad 2), which outputs all 4
+  streams.
+
+- Receiver in the SoC (Receiver). The receiver has a single sink pad (pad 0),
+  connected to the bridge, and four source pads (pads 1-4), going to the DMA
+  engine. The receiver demultiplexes the incoming streams to the four source
+  pads.
+
+- Four DMA Engines in the SoC (DMA Engine). Each DMA engine is connected to a
+  single source pad in the receiver.
+
+The sensors, the bridge and the receiver are modeled as V4L2 subdevices,
+exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are
+modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
+
+To configure this pipeline, the userspace must take the following steps:
+
+1) Set up media links between entities: connect the sensors to the bridge,
+bridge to the receiver, and the receiver to the DMA engines. This step does
+not differ from normal non-multiplexed media controller setup.
+
+2) Configure routing.
+
+.. flat-table:: Sensor routing table (identical on both sensors)
+    :header-rows:  1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0 (unused)
+      - 0/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE | V4L2_SUBDEV_ROUTE_FL_SOURCE
+      - Pixel data stream. Source route, i.e. the sink fields are unused.
+    * - 0/0 (unused)
+      - 0/1
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE | V4L2_SUBDEV_ROUTE_FL_SOURCE
+      - Metadata stream. Source route, i.e. the sink fields are unused.
+
+.. flat-table:: Bridge routing table
+    :header-rows:  1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0
+      - 2/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from Sensor A
+    * - 0/1
+      - 2/1
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from Sensor A
+    * - 1/0
+      - 2/2
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from Sensor B
+    * - 1/1
+      - 2/3
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from Sensor B
+
+.. flat-table:: Receiver routing table
+    :header-rows:  1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0
+      - 1/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from Sensor A
+    * - 0/1
+      - 2/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from Sensor A
+    * - 0/2
+      - 3/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from Sensor B
+    * - 0/3
+      - 4/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from Sensor B
+
+3) Configure streams
+
+After configuring the routing table, the next step is configuring the streams.
+This step is similar to configuring the pads in a non-multiplexed streams
+setup, with the difference that we need to configure each (pad, stream) pair
+(i.e. route endpoint) instead of just a pad.
+
+A common way to accomplish this is to start from the sensors and propagate the
+configurations along the stream towards the receiver, using VIDIOC_SUBDEV_S_FMT
+to configure each stream endpoint in each subdev.
-- 
2.25.1

[PATCH v10 32/38] media: subdev: add stream based configuration

[Tomi Valkeinen]

Add support to manage configurations (format, crop, compose) per stream,
instead of per pad. This is accomplished with data structures that hold
an array of all subdev's stream configurations.

The number of streams can vary at runtime based on routing. Every time
the routing is changed, the stream configurations need to be
re-initialized.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 .../v4l/vidioc-subdev-enum-frame-interval.rst |   5 +-
 .../v4l/vidioc-subdev-enum-frame-size.rst     |   5 +-
 .../v4l/vidioc-subdev-enum-mbus-code.rst      |   5 +-
 .../media/v4l/vidioc-subdev-g-crop.rst        |   5 +-
 .../media/v4l/vidioc-subdev-g-fmt.rst         |   5 +-
 .../v4l/vidioc-subdev-g-frame-interval.rst    |   5 +-
 .../media/v4l/vidioc-subdev-g-selection.rst   |   5 +-
 drivers/media/v4l2-core/v4l2-subdev.c         | 140 +++++++++++++++++-
 include/media/v4l2-subdev.h                   |  48 ++++++
 include/uapi/linux/v4l2-subdev.h              |  28 +++-
 10 files changed, 229 insertions(+), 22 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
index 3703943b412f..8def4c05d3da 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
@@ -92,7 +92,10 @@ multiple pads of the same sub-device is not defined.
       - Frame intervals to be enumerated, from enum
 	:ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
index c25a9896df0e..3ef361c0dca7 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
@@ -97,7 +97,10 @@ information about try formats.
       - Frame sizes to be enumerated, from enum
 	:ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
index 417f1a19bcc4..248f6f9ee7c5 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
@@ -73,7 +73,10 @@ information about the try formats.
       - ``flags``
       - See :ref:`v4l2-subdev-mbus-code-flags`
     * - __u32
-      - ``reserved``\ [7]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [6]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
index bd15c0a5a66b..1d267f7e7991 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
@@ -96,7 +96,10 @@ modified format should be as close as possible to the original request.
       - ``rect``
       - Crop rectangle boundaries, in pixels.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
index 7acdbb939d89..ed253a1e44b7 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
@@ -102,7 +102,10 @@ should be as close as possible to the original request.
       - Definition of an image format, see :c:type:`v4l2_mbus_framefmt` for
 	details.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
index d7fe7543c506..842f962d2aea 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
@@ -90,7 +90,10 @@ the same sub-device is not defined.
       - ``interval``
       - Period, in seconds, between consecutive video frames.
     * - __u32
-      - ``reserved``\ [9]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [8]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
index f9172a42f036..6b629c19168c 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
@@ -94,7 +94,10 @@ Selection targets and flags are documented in
       - ``r``
       - Selection rectangle, in pixels.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 633e1fccd377..9b944520eb97 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -149,14 +149,45 @@ static inline int check_pad(struct v4l2_subdev *sd, u32 pad)
 	return 0;
 }
 
-static int check_state_pads(u32 which, struct v4l2_subdev_state *state)
+static int check_state_pads(struct v4l2_subdev *sd, u32 which,
+			    struct v4l2_subdev_state *state)
 {
+	if (sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED)
+		return 0;
+
 	if (which == V4L2_SUBDEV_FORMAT_TRY && (!state || !state->pads))
 		return -EINVAL;
 
 	return 0;
 }
 
+static int check_state_pad_stream(struct v4l2_subdev *sd,
+				  struct v4l2_subdev_state *state, u32 pad,
+				  u32 stream)
+{
+	struct v4l2_mbus_framefmt *fmt;
+
+	if (!(sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED))
+		return 0;
+
+	/*
+	 * We need to take the state lock to access the format, but as we then
+	 * have to unlock, nothing prevents someone changing the state before
+	 * this call thread enters the driver's op and the driver has the
+	 * chance to lock the state.
+	 */
+	v4l2_subdev_lock_state(state);
+
+	fmt = v4l2_state_get_stream_format(state, pad, stream);
+
+	v4l2_subdev_unlock_state(state);
+
+	if (!fmt)
+		return -EINVAL;
+
+	return 0;
+}
+
 static inline int check_format(struct v4l2_subdev *sd,
 			       struct v4l2_subdev_state *state,
 			       struct v4l2_subdev_format *format)
@@ -165,7 +196,8 @@ static inline int check_format(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	return check_which(format->which) ? : check_pad(sd, format->pad) ? :
-	       check_state_pads(format->which, state);
+	       check_state_pads(sd, format->which, state) ? :
+	       check_state_pad_stream(sd, state, format->pad, format->stream);
 }
 
 static int call_get_fmt(struct v4l2_subdev *sd,
@@ -192,7 +224,8 @@ static int call_enum_mbus_code(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	return check_which(code->which) ? : check_pad(sd, code->pad) ? :
-	       check_state_pads(code->which, state) ? :
+	       check_state_pads(sd, code->which, state) ? :
+	       check_state_pad_stream(sd, state, code->pad, code->stream) ? :
 	       sd->ops->pad->enum_mbus_code(sd, state, code);
 }
 
@@ -204,7 +237,8 @@ static int call_enum_frame_size(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	return check_which(fse->which) ? : check_pad(sd, fse->pad) ? :
-	       check_state_pads(fse->which, state) ? :
+	       check_state_pads(sd, fse->which, state) ? :
+	       check_state_pad_stream(sd, state, fse->pad, fse->stream) ? :
 	       sd->ops->pad->enum_frame_size(sd, state, fse);
 }
 
@@ -239,7 +273,8 @@ static int call_enum_frame_interval(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	return check_which(fie->which) ? : check_pad(sd, fie->pad) ? :
-	       check_state_pads(fie->which, state) ? :
+	       check_state_pads(sd, fie->which, state) ? :
+	       check_state_pad_stream(sd, state, fie->pad, fie->stream) ? :
 	       sd->ops->pad->enum_frame_interval(sd, state, fie);
 }
 
@@ -251,7 +286,8 @@ static inline int check_selection(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	return check_which(sel->which) ? : check_pad(sd, sel->pad) ? :
-	       check_state_pads(sel->which, state);
+	       check_state_pads(sd, sel->which, state) ? :
+	       check_state_pad_stream(sd, state, sel->pad, sel->stream);
 }
 
 static int call_get_selection(struct v4l2_subdev *sd,
@@ -869,6 +905,71 @@ const struct v4l2_file_operations v4l2_subdev_fops = {
 	.poll = subdev_poll,
 };
 
+static int
+v4l2_init_stream_configs(struct v4l2_subdev_stream_configs *stream_configs,
+			 const struct v4l2_subdev_krouting *routing)
+{
+	u32 num_configs = 0;
+	unsigned int i;
+	u32 format_idx = 0;
+
+	kvfree(stream_configs->configs);
+	stream_configs->configs = NULL;
+	stream_configs->num_configs = 0;
+
+	/* Count number of formats needed */
+	for (i = 0; i < routing->num_routes; ++i) {
+		struct v4l2_subdev_route *route = &routing->routes[i];
+
+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
+			continue;
+
+		/*
+		 * Each route needs a format on both ends of the route, except
+		 * for source streams which only need one format.
+		 */
+		num_configs +=
+			(route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE) ? 1 : 2;
+	}
+
+	if (num_configs) {
+		stream_configs->configs =
+			kvcalloc(num_configs, sizeof(*stream_configs->configs),
+				 GFP_KERNEL);
+
+		if (!stream_configs->configs)
+			return -ENOMEM;
+
+		stream_configs->num_configs = num_configs;
+	}
+
+	/*
+	 * Fill in the 'pad' and stream' value for each item in the array from
+	 * the routing table
+	 */
+	for (i = 0; i < routing->num_routes; ++i) {
+		struct v4l2_subdev_route *route = &routing->routes[i];
+		u32 idx;
+
+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
+			continue;
+
+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE)) {
+			idx = format_idx++;
+
+			stream_configs->configs[idx].pad = route->sink_pad;
+			stream_configs->configs[idx].stream = route->sink_stream;
+		}
+
+		idx = format_idx++;
+
+		stream_configs->configs[idx].pad = route->source_pad;
+		stream_configs->configs[idx].stream = route->source_stream;
+	}
+
+	return 0;
+}
+
 #ifdef CONFIG_MEDIA_CONTROLLER
 
 int v4l2_subdev_get_fwnode_pad_1_to_1(struct media_entity *entity,
@@ -1044,7 +1145,8 @@ __v4l2_subdev_state_alloc(struct v4l2_subdev *sd, const char *lock_name,
 
 	__mutex_init(&state->lock, lock_name, lock_key);
 
-	if (sd->entity.num_pads) {
+	/* Drivers that support streams do not need the legacy pad config */
+	if (!(sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED) && sd->entity.num_pads) {
 		state->pads = kvmalloc_array(sd->entity.num_pads,
 					     sizeof(*state->pads),
 					     GFP_KERNEL | __GFP_ZERO);
@@ -1078,6 +1180,7 @@ void __v4l2_subdev_state_free(struct v4l2_subdev_state *state)
 	mutex_destroy(&state->lock);
 
 	kfree(state->routing.routes);
+	kvfree(state->stream_configs.configs);
 	kvfree(state->pads);
 	kfree(state);
 }
@@ -1177,6 +1280,27 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
 		dst->num_routes = src->num_routes;
 	}
 
-	return 0;
+	return v4l2_init_stream_configs(&state->stream_configs, dst);
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing);
+
+struct v4l2_mbus_framefmt *
+v4l2_state_get_stream_format(struct v4l2_subdev_state *state, unsigned int pad,
+			     u32 stream)
+{
+	struct v4l2_subdev_stream_configs *stream_configs;
+	unsigned int i;
+
+	lockdep_assert_held(&state->lock);
+
+	stream_configs = &state->stream_configs;
+
+	for (i = 0; i < stream_configs->num_configs; ++i) {
+		if (stream_configs->configs[i].pad == pad &&
+		    stream_configs->configs[i].stream == stream)
+			return &stream_configs->configs[i].fmt;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(v4l2_state_get_stream_format);
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 8525f39c2c67..7d7ef36e2597 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -697,6 +697,37 @@ struct v4l2_subdev_pad_config {
 	struct v4l2_rect try_compose;
 };
 
+/**
+ * struct v4l2_subdev_stream_config - Used for storing stream configuration.
+ *
+ * @pad: pad number
+ * @stream: stream number
+ * @fmt: &struct v4l2_mbus_framefmt
+ * @crop: &struct v4l2_rect to be used for crop
+ * @compose: &struct v4l2_rect to be used for compose
+ *
+ * This structure stores configuration for a stream.
+ */
+struct v4l2_subdev_stream_config {
+	u32 pad;
+	u32 stream;
+
+	struct v4l2_mbus_framefmt fmt;
+	struct v4l2_rect crop;
+	struct v4l2_rect compose;
+};
+
+/**
+ * struct v4l2_subdev_stream_configs - A collection of stream configs.
+ *
+ * @num_configs: number of entries in @config.
+ * @configs: an array of &struct v4l2_subdev_stream_configs.
+ */
+struct v4l2_subdev_stream_configs {
+	u32 num_configs;
+	struct v4l2_subdev_stream_config *configs;
+};
+
 /**
  * struct v4l2_subdev_krouting - subdev routing table
  *
@@ -716,6 +747,7 @@ struct v4l2_subdev_krouting {
  * @lock: mutex for the state
  * @pads: &struct v4l2_subdev_pad_config array
  * @routing: routing table for the subdev
+ * @stream_configs: stream configurations (only for V4L2_SUBDEV_FL_MULTIPLEXED)
  *
  * This structure only needs to be passed to the pad op if the 'which' field
  * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
@@ -726,6 +758,7 @@ struct v4l2_subdev_state {
 	struct mutex lock;
 	struct v4l2_subdev_pad_config *pads;
 	struct v4l2_subdev_krouting routing;
+	struct v4l2_subdev_stream_configs stream_configs;
 };
 
 /**
@@ -1455,4 +1488,19 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
 			    struct v4l2_subdev_state *state,
 			    struct v4l2_subdev_krouting *routing);
 
+/**
+ * v4l2_state_get_stream_format() - Get pointer to a stream format
+ * @state: subdevice state
+ * @pad: pad id
+ * @stream: stream id
+ *
+ * This returns a pointer to &struct v4l2_mbus_framefmt for the given pad +
+ * stream in the subdev state.
+ *
+ * If the state does not contain the given pad + stream, NULL is returned.
+ */
+struct v4l2_mbus_framefmt *
+v4l2_state_get_stream_format(struct v4l2_subdev_state *state, unsigned int pad,
+			     u32 stream);
+
 #endif
diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
index 1ec3141bf860..480891dba193 100644
--- a/include/uapi/linux/v4l2-subdev.h
+++ b/include/uapi/linux/v4l2-subdev.h
@@ -44,13 +44,15 @@ enum v4l2_subdev_format_whence {
  * @which: format type (from enum v4l2_subdev_format_whence)
  * @pad: pad number, as reported by the media API
  * @format: media bus format (format code and frame size)
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_format {
 	__u32 which;
 	__u32 pad;
 	struct v4l2_mbus_framefmt format;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 /**
@@ -58,13 +60,15 @@ struct v4l2_subdev_format {
  * @which: format type (from enum v4l2_subdev_format_whence)
  * @pad: pad number, as reported by the media API
  * @rect: pad crop rectangle boundaries
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_crop {
 	__u32 which;
 	__u32 pad;
 	struct v4l2_rect rect;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 #define V4L2_SUBDEV_MBUS_CODE_CSC_COLORSPACE	0x00000001
@@ -80,6 +84,7 @@ struct v4l2_subdev_crop {
  * @code: format code (MEDIA_BUS_FMT_ definitions)
  * @which: format type (from enum v4l2_subdev_format_whence)
  * @flags: flags set by the driver, (V4L2_SUBDEV_MBUS_CODE_*)
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_mbus_code_enum {
@@ -88,7 +93,8 @@ struct v4l2_subdev_mbus_code_enum {
 	__u32 code;
 	__u32 which;
 	__u32 flags;
-	__u32 reserved[7];
+	__u32 stream;
+	__u32 reserved[6];
 };
 
 /**
@@ -101,6 +107,7 @@ struct v4l2_subdev_mbus_code_enum {
  * @min_height: minimum frame height, in pixels
  * @max_height: maximum frame height, in pixels
  * @which: format type (from enum v4l2_subdev_format_whence)
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_frame_size_enum {
@@ -112,19 +119,22 @@ struct v4l2_subdev_frame_size_enum {
 	__u32 min_height;
 	__u32 max_height;
 	__u32 which;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 /**
  * struct v4l2_subdev_frame_interval - Pad-level frame rate
  * @pad: pad number, as reported by the media API
  * @interval: frame interval in seconds
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_frame_interval {
 	__u32 pad;
 	struct v4l2_fract interval;
-	__u32 reserved[9];
+	__u32 stream;
+	__u32 reserved[8];
 };
 
 /**
@@ -136,6 +146,7 @@ struct v4l2_subdev_frame_interval {
  * @height: frame height in pixels
  * @interval: frame interval in seconds
  * @which: format type (from enum v4l2_subdev_format_whence)
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_frame_interval_enum {
@@ -146,7 +157,8 @@ struct v4l2_subdev_frame_interval_enum {
 	__u32 height;
 	struct v4l2_fract interval;
 	__u32 which;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 /**
@@ -158,6 +170,7 @@ struct v4l2_subdev_frame_interval_enum {
  *	    defined in v4l2-common.h; V4L2_SEL_TGT_* .
  * @flags: constraint flags, defined in v4l2-common.h; V4L2_SEL_FLAG_*.
  * @r: coordinates of the selection window
+ * @stream: stream number, defined in subdev routing
  * @reserved: for future use, set to zero for now
  *
  * Hardware may use multiple helper windows to process a video stream.
@@ -170,7 +183,8 @@ struct v4l2_subdev_selection {
 	__u32 target;
 	__u32 flags;
 	struct v4l2_rect r;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 /**
-- 
2.25.1

[PATCH v10 33/38] media: subdev: use streams in v4l2_subdev_link_validate()

[Tomi Valkeinen]

Update v4l2_subdev_link_validate() to use routing and streams for
validation.

Instead of just looking at the format on the pad on both ends of the
link, the routing tables are used to collect all the streams going from
the source to the sink over the link, and the streams' formats on both
ends of the link are verified.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 221 +++++++++++++++++++++++---
 1 file changed, 203 insertions(+), 18 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 9b944520eb97..5245ec219ee1 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -16,6 +16,7 @@
 #include <linux/videodev2.h>
 #include <linux/export.h>
 #include <linux/version.h>
+#include <linux/sort.h>
 
 #include <media/v4l2-ctrls.h>
 #include <media/v4l2-device.h>
@@ -1052,6 +1053,7 @@ EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate_default);
 
 static int
 v4l2_subdev_link_validate_get_format(struct media_pad *pad,
+				     u32 stream,
 				     struct v4l2_subdev_format *fmt)
 {
 	if (is_media_entity_v4l2_subdev(pad->entity)) {
@@ -1063,6 +1065,7 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
 
 		fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
 		fmt->pad = pad->index;
+		fmt->stream = stream;
 		return v4l2_subdev_call(sd, pad, get_fmt, state, fmt);
 	}
 
@@ -1073,31 +1076,213 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
 	return -EINVAL;
 }
 
-int v4l2_subdev_link_validate(struct media_link *link)
+static int cmp_u32(const void *a, const void *b)
 {
-	struct v4l2_subdev *sink;
-	struct v4l2_subdev_format sink_fmt, source_fmt;
-	int rval;
+	u32 a32 = *(u32 *)a;
+	u32 b32 = *(u32 *)b;
 
-	rval = v4l2_subdev_link_validate_get_format(
-		link->source, &source_fmt);
-	if (rval < 0)
-		return 0;
+	return a32 > b32 ? 1 : (a32 < b32 ? -1 : 0);
+}
+
+static int v4l2_link_validate_get_streams(struct media_link *link,
+					  bool is_source, u32 *out_num_streams,
+					  const u32 **out_streams,
+					  bool *allocated)
+{
+	static const u32 default_streams[] = { 0 };
+	struct v4l2_subdev_krouting *routing;
+	struct v4l2_subdev *subdev;
+	u32 num_streams;
+	u32 *streams;
+	unsigned int i;
+	struct v4l2_subdev_state *state;
+
+	if (is_source)
+		subdev = media_entity_to_v4l2_subdev(link->source->entity);
+	else
+		subdev = media_entity_to_v4l2_subdev(link->sink->entity);
 
-	rval = v4l2_subdev_link_validate_get_format(
-		link->sink, &sink_fmt);
-	if (rval < 0)
+	if (!(subdev->flags & V4L2_SUBDEV_FL_MULTIPLEXED)) {
+		*out_num_streams = 1;
+		*out_streams = default_streams;
+		*allocated = false;
 		return 0;
+	}
 
-	sink = media_entity_to_v4l2_subdev(link->sink->entity);
+	state = v4l2_subdev_lock_active_state(subdev);
 
-	rval = v4l2_subdev_call(sink, pad, link_validate, link,
-				&source_fmt, &sink_fmt);
-	if (rval != -ENOIOCTLCMD)
-		return rval;
+	routing = &state->routing;
+
+	streams = kmalloc_array(routing->num_routes, sizeof(u32), GFP_KERNEL);
+
+	if (!streams) {
+		v4l2_subdev_unlock_state(state);
+		return -ENOMEM;
+	}
+
+	num_streams = 0;
 
-	return v4l2_subdev_link_validate_default(
-		sink, link, &source_fmt, &sink_fmt);
+	for (i = 0; i < routing->num_routes; ++i) {
+		struct v4l2_subdev_route *route = &routing->routes[i];
+		int j;
+		u32 route_pad;
+		u32 route_stream;
+		u32 link_pad;
+
+		if (is_source) {
+			route_pad = route->source_pad;
+			route_stream = route->source_stream;
+			link_pad = link->source->index;
+		} else {
+			route_pad = route->sink_pad;
+			route_stream = route->sink_stream;
+			link_pad = link->sink->index;
+		}
+
+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
+			continue;
+
+		if (route_pad != link_pad)
+			continue;
+
+		/* look for duplicates... */
+		for (j = 0; j < num_streams; ++j) {
+			if (streams[j] == route_stream)
+				break;
+		}
+
+		/* ...and drop the stream if we already have it */
+		if (j != num_streams)
+			continue;
+
+		streams[num_streams++] = route_stream;
+	}
+
+	v4l2_subdev_unlock_state(state);
+
+	sort(streams, num_streams, sizeof(u32), &cmp_u32, NULL);
+
+	*out_num_streams = num_streams;
+	*out_streams = streams;
+	*allocated = true;
+
+	return 0;
+}
+
+int v4l2_subdev_link_validate(struct media_link *link)
+{
+	struct v4l2_subdev *sink_subdev =
+		media_entity_to_v4l2_subdev(link->sink->entity);
+	struct device *dev = sink_subdev->entity.graph_obj.mdev->dev;
+	u32 num_source_streams;
+	const u32 *source_streams;
+	bool source_allocated;
+	u32 num_sink_streams;
+	const u32 *sink_streams;
+	bool sink_allocated;
+	unsigned int sink_idx;
+	unsigned int source_idx;
+	int ret;
+
+	dev_dbg(dev, "validating link \"%s\":%u -> \"%s\":%u\n",
+		link->source->entity->name, link->source->index,
+		link->sink->entity->name, link->sink->index);
+
+	ret = v4l2_link_validate_get_streams(link, true, &num_source_streams,
+					     &source_streams,
+					     &source_allocated);
+	if (ret)
+		return ret;
+
+	ret = v4l2_link_validate_get_streams(link, false, &num_sink_streams,
+					     &sink_streams, &sink_allocated);
+	if (ret)
+		goto free_source;
+
+	/*
+	 * It is ok to have more source streams than sink streams as extra
+	 * source streams can just be ignored (i.e. they go nowhere), but extra
+	 * sink streams is an error.
+	 */
+	if (num_source_streams < num_sink_streams) {
+		dev_err(dev,
+			"Not enough source streams: %d < %d\n",
+			num_source_streams, num_sink_streams);
+		ret = -EINVAL;
+		goto out;
+	}
+
+	/* Validate source and sink stream formats */
+
+	source_idx = 0;
+
+	for (sink_idx = 0; sink_idx < num_sink_streams; ++sink_idx) {
+		struct v4l2_subdev_format sink_fmt, source_fmt;
+		u32 stream;
+
+		stream = sink_streams[sink_idx];
+
+		for (; source_idx < num_source_streams; ++source_idx) {
+			if (source_streams[source_idx] == stream)
+				break;
+		}
+
+		if (source_idx == num_source_streams) {
+			dev_err(dev, "No source stream for sink stream %u\n",
+				stream);
+			ret = -EINVAL;
+			goto out;
+		}
+
+		dev_dbg(dev, "validating stream \"%s\":%u:%u -> \"%s\":%u:%u\n",
+			link->source->entity->name, link->source->index, stream,
+			link->sink->entity->name, link->sink->index, stream);
+
+		ret = v4l2_subdev_link_validate_get_format(link->source, stream,
+							   &source_fmt);
+		if (ret < 0) {
+			dev_dbg(dev, "Failed to get format for \"%s\":%u:%u (but that's ok)\n",
+				link->source->entity->name, link->source->index,
+				stream);
+			ret = 0;
+			continue;
+		}
+
+		ret = v4l2_subdev_link_validate_get_format(link->sink, stream,
+							   &sink_fmt);
+		if (ret < 0) {
+			dev_dbg(dev, "Failed to get format for \"%s\":%u:%u (but that's ok)\n",
+				link->sink->entity->name, link->sink->index,
+				stream);
+			ret = 0;
+			continue;
+		}
+
+		/* TODO: add stream number to link_validate() */
+		ret = v4l2_subdev_call(sink_subdev, pad, link_validate, link,
+				       &source_fmt, &sink_fmt);
+		if (!ret)
+			continue;
+
+		if (ret != -ENOIOCTLCMD)
+			goto out;
+
+		ret = v4l2_subdev_link_validate_default(sink_subdev, link,
+							&source_fmt, &sink_fmt);
+
+		if (ret)
+			goto out;
+	}
+
+out:
+	if (sink_allocated)
+		kfree(sink_streams);
+
+free_source:
+	if (source_allocated)
+		kfree(source_streams);
+
+	return ret;
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
 
-- 
2.25.1

[PATCH v10 34/38] media: subdev: add "opposite" stream helper funcs

[Tomi Valkeinen]

Add two helper functions to make dealing with streams easier:

v4l2_subdev_routing_find_opposite_end - given a routing table and a pad
+ stream, return the pad + stream on the opposite side of the subdev.

v4l2_subdev_state_get_opposite_stream_format - return a pointer to the
format on the pad + stream on the opposite side from the given pad +
stream.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 47 +++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 36 ++++++++++++++++++++
 2 files changed, 83 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 5245ec219ee1..13a092442cd4 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1489,3 +1489,50 @@ v4l2_state_get_stream_format(struct v4l2_subdev_state *state, unsigned int pad,
 	return NULL;
 }
 EXPORT_SYMBOL_GPL(v4l2_state_get_stream_format);
+
+int v4l2_subdev_routing_find_opposite_end(const struct v4l2_subdev_krouting *routing,
+					  u32 pad, u32 stream, u32 *other_pad,
+					  u32 *other_stream)
+{
+	unsigned int i;
+
+	for (i = 0; i < routing->num_routes; ++i) {
+		struct v4l2_subdev_route *route = &routing->routes[i];
+
+		if (route->source_pad == pad &&
+		    route->source_stream == stream) {
+			if (other_pad)
+				*other_pad = route->sink_pad;
+			if (other_stream)
+				*other_stream = route->sink_stream;
+			return 0;
+		}
+
+		if (route->sink_pad == pad && route->sink_stream == stream) {
+			if (other_pad)
+				*other_pad = route->source_pad;
+			if (other_stream)
+				*other_stream = route->source_stream;
+			return 0;
+		}
+	}
+
+	return -EINVAL;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_routing_find_opposite_end);
+
+struct v4l2_mbus_framefmt *
+v4l2_subdev_state_get_opposite_stream_format(struct v4l2_subdev_state *state,
+					     u32 pad, u32 stream)
+{
+	u32 other_pad, other_stream;
+	int ret;
+
+	ret = v4l2_subdev_routing_find_opposite_end(&state->routing, pad, stream,
+						    &other_pad, &other_stream);
+	if (ret)
+		return NULL;
+
+	return v4l2_state_get_stream_format(state, other_pad, other_stream);
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_opposite_stream_format);
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 7d7ef36e2597..b1761da83533 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1503,4 +1503,40 @@ struct v4l2_mbus_framefmt *
 v4l2_state_get_stream_format(struct v4l2_subdev_state *state, unsigned int pad,
 			     u32 stream);
 
+/**
+ * v4l2_subdev_routing_find_opposite_end() - Find the opposite stream
+ * @routing: routing used to find the opposite side
+ * @pad: pad id
+ * @stream: stream id
+ * @other_pad: pointer used to return the opposite pad
+ * @other_stream: pointer used to return the opposite stream
+ *
+ * This function uses the routing table to find the pad + stream which is
+ * opposite the given pad + stream.
+ *
+ * @other_pad and/or @other_stream can be NULL if the caller does not need the
+ * value.
+ *
+ * Returns 0 on success, or -EINVAL if no matching route is found.
+ */
+int v4l2_subdev_routing_find_opposite_end(const struct v4l2_subdev_krouting *routing,
+					  u32 pad, u32 stream, u32 *other_pad,
+					  u32 *other_stream);
+
+/**
+ * v4l2_subdev_state_get_opposite_stream_format() - Get pointer to opposite
+ *                                                  stream format
+ * @state: subdevice state
+ * @pad: pad id
+ * @stream: stream id
+ *
+ * This returns a pointer to &struct v4l2_mbus_framefmt for the pad + stream
+ * that is opposite the given pad + stream in the subdev state.
+ *
+ * If the state does not contain the given pad + stream, NULL is returned.
+ */
+struct v4l2_mbus_framefmt *
+v4l2_subdev_state_get_opposite_stream_format(struct v4l2_subdev_state *state,
+					     u32 pad, u32 stream);
+
 #endif
-- 
2.25.1

[PATCH v10 35/38] media: subdev: add v4l2_subdev_get_fmt() helper function

[Tomi Valkeinen]

Add v4l2_subdev_get_fmt() helper function which implements
v4l2_subdev_pad_ops.get_fmt using streams. Subdev drivers that do not
need to do anything special in their get_fmt op can use this helper
directly for v4l2_subdev_pad_ops.get_fmt.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 21 +++++++++++++++++++++
 include/media/v4l2-subdev.h           | 17 +++++++++++++++++
 2 files changed, 38 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 13a092442cd4..8469c56ae870 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1536,3 +1536,24 @@ v4l2_subdev_state_get_opposite_stream_format(struct v4l2_subdev_state *state,
 	return v4l2_state_get_stream_format(state, other_pad, other_stream);
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_opposite_stream_format);
+
+int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
+			struct v4l2_subdev_format *format)
+{
+	struct v4l2_mbus_framefmt *fmt;
+
+	v4l2_subdev_lock_state(state);
+
+	fmt = v4l2_state_get_stream_format(state, format->pad, format->stream);
+	if (!fmt) {
+		v4l2_subdev_unlock_state(state);
+		return -EINVAL;
+	}
+
+	format->format = *fmt;
+
+	v4l2_subdev_unlock_state(state);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_get_fmt);
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index b1761da83533..65c62a889f6c 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1539,4 +1539,21 @@ struct v4l2_mbus_framefmt *
 v4l2_subdev_state_get_opposite_stream_format(struct v4l2_subdev_state *state,
 					     u32 pad, u32 stream);
 
+/**
+ * v4l2_subdev_get_fmt() - Fill format based on state
+ * @sd: subdevice
+ * @state: subdevice state
+ * @format: pointer to &struct v4l2_subdev_format
+ *
+ * Fill @format based on the pad and stream given in the @format struct.
+ *
+ * This function can be used by the subdev drivers to implement
+ * v4l2_subdev_pad_ops.get_fmt if the subdev driver does not need to do
+ * anything special in their get_fmt op.
+ *
+ * Returns 0 on success, error value otherwise.
+ */
+int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
+			struct v4l2_subdev_format *format);
+
 #endif
-- 
2.25.1

[PATCH v10 36/38] media: subdev: add v4l2_subdev_set_routing_with_fmt() helper

[Tomi Valkeinen]

v4l2_subdev_set_routing_with_fmt() is the same as
v4l2_subdev_set_routing(), but additionally initializes all the streams
with the given format.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 22 ++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 16 ++++++++++++++++
 2 files changed, 38 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 8469c56ae870..83ff15dc440a 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1469,6 +1469,28 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing);
 
+int v4l2_subdev_set_routing_with_fmt(struct v4l2_subdev *sd,
+				     struct v4l2_subdev_state *state,
+				     struct v4l2_subdev_krouting *routing,
+				     const struct v4l2_mbus_framefmt *fmt)
+{
+	struct v4l2_subdev_stream_configs *stream_configs;
+	unsigned int i;
+	int ret;
+
+	ret = v4l2_subdev_set_routing(sd, state, routing);
+	if (ret)
+		return ret;
+
+	stream_configs = &state->stream_configs;
+
+	for (i = 0; i < stream_configs->num_configs; ++i)
+		stream_configs->configs[i].fmt = *fmt;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing_with_fmt);
+
 struct v4l2_mbus_framefmt *
 v4l2_state_get_stream_format(struct v4l2_subdev_state *state, unsigned int pad,
 			     u32 stream)
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 65c62a889f6c..2c70d92e497b 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1488,6 +1488,22 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
 			    struct v4l2_subdev_state *state,
 			    struct v4l2_subdev_krouting *routing);
 
+/**
+ * v4l2_subdev_set_routing_with_fmt() - Set given routing and format to subdev
+ *					state
+ * @sd: The subdevice
+ * @state: The subdevice state
+ * @routing: Routing that will be copied to subdev state
+ * @fmt: Format used to initialize all the streams
+ *
+ * This is the same as v4l2_subdev_set_routing, but additionally initializes
+ * all the streams using the given format.
+ */
+int v4l2_subdev_set_routing_with_fmt(struct v4l2_subdev *sd,
+				     struct v4l2_subdev_state *state,
+				     struct v4l2_subdev_krouting *routing,
+				     const struct v4l2_mbus_framefmt *fmt);
+
 /**
  * v4l2_state_get_stream_format() - Get pointer to a stream format
  * @state: subdevice state
-- 
2.25.1

[PATCH v10 37/38] media: subdev: add v4l2_subdev_routing_validate_1_to_1 helper

[Tomi Valkeinen]

Add a helper for verifying routing for the common case of
non-overlapping 1-to-1 streams.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 24 ++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 14 ++++++++++++++
 2 files changed, 38 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 83ff15dc440a..dc31118adc6b 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1579,3 +1579,27 @@ int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
 	return 0;
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_get_fmt);
+
+int v4l2_subdev_routing_validate_1_to_1(const struct v4l2_subdev_krouting *routing)
+{
+	unsigned int i, j;
+
+	for (i = 0; i < routing->num_routes; ++i) {
+		const struct v4l2_subdev_route *route = &routing->routes[i];
+
+		for (j = i + 1; j < routing->num_routes; ++j) {
+			const struct v4l2_subdev_route *r = &routing->routes[j];
+
+			if (route->sink_pad == r->sink_pad &&
+			    route->sink_stream == r->sink_stream)
+				return -EINVAL;
+
+			if (route->source_pad == r->source_pad &&
+			    route->source_stream == r->source_stream)
+				return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_routing_validate_1_to_1);
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 2c70d92e497b..a0c122c9f51e 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1572,4 +1572,18 @@ v4l2_subdev_state_get_opposite_stream_format(struct v4l2_subdev_state *state,
 int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
 			struct v4l2_subdev_format *format);
 
+/**
+ * v4l2_subdev_routing_validate_1_to_1() - Verify that all streams are
+ *                                         non-overlapping 1-to-1 streams
+ * @routing: routing to verify
+ *
+ * This verifies that the given routing contains only non-overlapping 1-to-1
+ * streams. In other words, no two streams have the same source or sink
+ * stream ID on a single pad. This is the most common case of routing
+ * supported by devices.
+ *
+ * Returns 0 on success, error value otherwise.
+ */
+int v4l2_subdev_routing_validate_1_to_1(const struct v4l2_subdev_krouting *routing);
+
 #endif
-- 
2.25.1

[PATCH v10 38/38] media: subdev: Add for_each_active_route() macro

[Tomi Valkeinen]

From: Jacopo Mondi <jacopo+renesas@jmondi.org>

Add a for_each_active_route() macro to replace the repeated pattern
of iterating on the active routes of a routing table.

Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 20 ++++++++++++++++++++
 include/media/v4l2-subdev.h           | 13 +++++++++++++
 2 files changed, 33 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index dc31118adc6b..dca2bea180ec 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1603,3 +1603,23 @@ int v4l2_subdev_routing_validate_1_to_1(const struct v4l2_subdev_krouting *routi
 	return 0;
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_routing_validate_1_to_1);
+
+struct v4l2_subdev_route *
+__v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting *routing,
+				struct v4l2_subdev_route *route)
+{
+	if (route)
+		++route;
+	else
+		route = &routing->routes[0];
+
+	for (; route < routing->routes + routing->num_routes; ++route) {
+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
+			continue;
+
+		return route;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(__v4l2_subdev_next_active_route);
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index a0c122c9f51e..9754913b34f8 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1586,4 +1586,17 @@ int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
  */
 int v4l2_subdev_routing_validate_1_to_1(const struct v4l2_subdev_krouting *routing);
 
+struct v4l2_subdev_route *
+__v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting *routing,
+				struct v4l2_subdev_route *route);
+
+/**
+ * for_each_active_route - iterate on all active routes of a routing table
+ * @routing: The routing table
+ * @route: The route iterator
+ */
+#define for_each_active_route(routing, route) \
+	for ((route) = NULL;                  \
+	     ((route) = __v4l2_subdev_next_active_route((routing), (route)));)
+
 #endif
-- 
2.25.1

Re: [PATCH v10 00/38] v4l: subdev internal routing and streams

[Tomi Valkeinen]

[-- Attachment #1: Type: text/plain, Size: 1321 bytes --]

On 30/11/2021 16:14, Tomi Valkeinen wrote:
> Hi,
> 
> This is v10 of the multiplexed streams series. v8 can be found from:

That's supposed to be v9, not v8.

> https://lore.kernel.org/all/20211005085750.138151-1-tomi.valkeinen@ideasonboard.com/
> 
> I have pushed my work branch to:
> 
> git://git.kernel.org/pub/scm/linux/kernel/git/tomba/linux.git multistream/work-v10
> 
> which contains the patches in this series, along with subdev drivers
> using multiplexed streams.
> 
> I have also pushed v4l-utils changes to:
> 
> https://github.com/tomba/v4l-utils.git streams-review-v1
> 
> Changes to v9:
> 
> - Add V4L2_SUBDEV_CAP_MPLEXED flag
> - Use standard kmalloc and kmemdup for routes
> - Allow NULL as pad/stream param for v4l2_state_find_opposite_end
> - Add for_each_active_route
> - Use _BITUL() in uapi header
> - Rearrange struct v4l2_subdev_routing members to align on 64 bit
> - Renames:
> 	sd->state -> sd->active_state
> 	v4l2_state_find_opposite_end -> v4l2_subdev_routing_find_opposite_end
> 	v4l2_state_get_opposite_stream_format -> v4l2_subdev_state_get_opposite_stream_format
> 	v4l2_routing_simple_verify -> v4l2_subdev_routing_validate_1_to_1
> 	v4l2_subdev_validate_and_lock_state -> v4l2_subdev_lock_and_return_state
> - Doc & comment updates

To help reviews, here's the diff to the v8.

  Tomi

[-- Attachment #2: multistream-diff.txt --]
[-- Type: text/plain, Size: 22882 bytes --]

diff --git a/Documentation/driver-api/media/v4l2-subdev.rst b/Documentation/driver-api/media/v4l2-subdev.rst
index 410e03a29f2a..1d986e814b4e 100644
--- a/Documentation/driver-api/media/v4l2-subdev.rst
+++ b/Documentation/driver-api/media/v4l2-subdev.rst
@@ -525,9 +525,9 @@ Traditionally V4L2 subdev drivers maintained internal state for the active
 configuration for the subdev. This is often implemented as an array of struct
 v4l2_mbus_framefmt, one entry for each pad.
 
-In addition to the active configuration, each subdev filehandle has contained
-an array of struct v4l2_subdev_pad_config, managed by V4L2 core, which
-contains the TRY configuration.
+In addition to the active configuration, each subdev filehandle has an array of
+struct v4l2_subdev_pad_config, managed by V4L2 core, which contains the TRY
+configuration.
 
 To simplify the subdev drivers the V4L2 subdev API now optionally supports a
 centrally managed active configuration. A subdev driver must use
@@ -535,13 +535,13 @@ v4l2_subdev_init_finalize() to initialize the active state between calls to
 media_entity_pads_init() and v4l2_*_register_subdev(), and must call
 v4l2_subdev_cleanup() to free the state.
 
-The active state must be locked before access, and can be done with
+The active state must be locked before access, and that can be done with
 v4l2_subdev_lock_state() or v4l2_subdev_lock_active_state().
 
 The V4L2 core will pass either the TRY or ACTIVE state to various subdev ops.
-Unfortunately all the subdev drivers do not comply with this yet, and may pass
+Unfortunately not all the subdev drivers comply with this yet, and may pass
 NULL for the ACTIVE case. This is only a problem for subdev drivers which use
-the cetrally managed active state and are used in media pipelines with older
+the centrally managed active state and are used in media pipelines with older
 subdev drivers. In these cases the called subdev ops must also handle the NULL
 case. This can be easily managed by the use of
 v4l2_subdev_validate_and_lock_state() helper.
diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index ea3efa97bb08..fd042afeddd6 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -508,7 +508,7 @@ Streams, multiplexed media pads and internal routing
 ----------------------------------------------------
 
 Commonly V4L2 subdevices support only separate video streams, that is, each
-link in the media grap and each pad in a subdevice passes through a single
+link in the media graph and each pad in a subdevice pass through a single
 video stream. Thus each pad contains a format configuration for that single
 stream. In some cases a subdev can do stream processing and split a stream
 into two or compose two streams into one, but the inputs and outputs for the
@@ -520,22 +520,22 @@ stream and a metadata stream, and a bridge subdev could route the streams
 from multiple input pads into a single output pad.
 
 Subdevice drivers that support multiplexed streams are compatible with
-non-multiplexed subdev drivers, but, of course, requires such a routing
-configuration where the link between those two types of drivers contain only
-a single stream.
+non-multiplexed subdev drivers, but, of course, require a routing configuration
+where the link between those two types of drivers contains only a single
+stream.
 
 Understanding streams
 ^^^^^^^^^^^^^^^^^^^^^
 
 A stream is a stream of content (e.g. pixel data or metadata) flowing through
-the media pipeline from a source (e.g. a sensor) towards the final sink
-(e.g. a receiver in a SoC). Each media link carries all the streams from
-one end of the link to the other, whereas subdevices have routing tables
-which describe how the incoming streams from sink pads are routed to the
-source pads.
+the media pipeline from a source (e.g. a sensor) towards the final sink(e.g. a
+receiver and demultiplexer in a SoC). Each media link carries all the streams
+from one end of the link to the other, and subdevices have routing tables which
+describe how the incoming streams from sink pads are routed to the source
+pads.
 
 A stream ID (often just "stream") is a media link-local identifier for a
-stream. In other words, configuration for a particular stream ID must exist
+stream. In other words, a configuration for a particular stream ID must exist
 on both sides of a media link, but another stream ID can be used for the same
 stream at the other side of the subdevice.
 
@@ -556,7 +556,8 @@ There are three steps in configuring the streams:
 Controller API <media_controller>`
 
 2) Routing. The routing table for the subdevice must be set with
-:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl.
+:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that
+setting the routing table will reset all the stream configurations.
 
 3) Configure streams. Each route endpoint must be configured
 with :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>`.
@@ -662,8 +663,8 @@ not differ from normal non-multiplexed media controller setup.
 After configuring the routing table, the next step is configuring the streams.
 This step is similar to configuring the pads in a non-multiplexed streams
 setup, with the difference that we need to configure each (pad, stream) pair
-(i.e. route endpoint), instead of just a pad.
+(i.e. route endpoint) instead of just a pad.
 
-Presuming there are no format conversions in the pipeline, the userspace needs
-to configure all the route endpoints using four formats (two pixel formats
-and two metadata formats) with VIDIOC_SUBDEV_S_FMT.
+A common way to accomplish this is to start from the sensors and propagate the
+configurations along the stream towards the receiver, using VIDIOC_SUBDEV_S_FMT
+to configure each stream endpoint in each subdev.
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
index 41f4873c49f7..a0d9c79e162f 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
@@ -41,19 +41,23 @@ The routing configuration determines the flows of data inside an entity.
 
 Drivers report their current routing tables using the
 ``VIDIOC_SUBDEV_G_ROUTING`` ioctl and application may enable or disable routes
-with the VIDIOC_SUBDEV_S_ROUTING ioctl, by adding or removing routes and setting
-or clearing the ``V4L2_SUBDEV_ROUTE_FL_ACTIVE`` flag of the  ``flags`` field of
-a struct :c:type:`v4l2_subdev_route`.
+with the ``VIDIOC_SUBDEV_S_ROUTING`` ioctl, by adding or removing routes and
+setting or clearing flags of the  ``flags`` field of a
+struct :c:type:`v4l2_subdev_route`.
+
+All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called. This
+means that the userspace mut reconfigure all streams after calling the ioctl
+with e.g. ``VIDIOC_SUBDEV_S_FMT``.
 
 A special case for routing are routes marked with
 ``V4L2_SUBDEV_ROUTE_FL_SOURCE`` flag. These routes are used to describe
 source endpoints on sensors and the sink fields are unused.
 
-When inspecting routes through VIDIOC_SUBDEV_G_ROUTING and the application
+When inspecting routes through ``VIDIOC_SUBDEV_G_ROUTING`` and the application
 provided ``num_routes`` is not big enough to contain all the available routes
 the subdevice exposes, drivers return the ENOSPC error code and adjust the
 value of the ``num_routes`` field. Application should then reserve enough memory
-for all the route entries and call VIDIOC_SUBDEV_G_ROUTING again.
+for all the route entries and call ``VIDIOC_SUBDEV_G_ROUTING`` again.
 
 .. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
 
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 2a64ff003e4b..dca2bea180ec 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -175,7 +175,7 @@ static int check_state_pad_stream(struct v4l2_subdev *sd,
 	 * We need to take the state lock to access the format, but as we then
 	 * have to unlock, nothing prevents someone changing the state before
 	 * this call thread enters the driver's op and the driver has the
-	 * change to lock the state.
+	 * chance to lock the state.
 	 */
 	v4l2_subdev_lock_state(state);
 
@@ -462,7 +462,9 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 
 		memset(cap->reserved, 0, sizeof(cap->reserved));
 		cap->version = LINUX_VERSION_CODE;
-		cap->capabilities = ro_subdev ? V4L2_SUBDEV_CAP_RO_SUBDEV : 0;
+		cap->capabilities =
+			(ro_subdev ? V4L2_SUBDEV_CAP_RO_SUBDEV : 0) |
+			((sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED) ? V4L2_SUBDEV_CAP_MPLEXED : 0);
 
 		return 0;
 	}
@@ -1197,7 +1199,11 @@ int v4l2_subdev_link_validate(struct media_link *link)
 	if (ret)
 		goto free_source;
 
-	/* It is ok to have more source streams than sink streams */
+	/*
+	 * It is ok to have more source streams than sink streams as extra
+	 * source streams can just be ignored (i.e. they go nowhere), but extra
+	 * sink streams is an error.
+	 */
 	if (num_source_streams < num_sink_streams) {
 		dev_err(dev,
 			"Not enough source streams: %d < %d\n",
@@ -1358,7 +1364,7 @@ void __v4l2_subdev_state_free(struct v4l2_subdev_state *state)
 
 	mutex_destroy(&state->lock);
 
-	kvfree(state->routing.routes);
+	kfree(state->routing.routes);
 	kvfree(state->stream_configs.configs);
 	kvfree(state->pads);
 	kfree(state);
@@ -1403,7 +1409,7 @@ int __v4l2_subdev_init_finalize(struct v4l2_subdev *sd, const char *name,
 	if (IS_ERR(state))
 		return PTR_ERR(state);
 
-	sd->state = state;
+	sd->active_state = state;
 
 	return 0;
 }
@@ -1411,16 +1417,16 @@ EXPORT_SYMBOL_GPL(__v4l2_subdev_init_finalize);
 
 void v4l2_subdev_cleanup(struct v4l2_subdev *sd)
 {
-	__v4l2_subdev_state_free(sd->state);
-	sd->state = NULL;
+	__v4l2_subdev_state_free(sd->active_state);
+	sd->active_state = NULL;
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_cleanup);
 
 struct v4l2_subdev_state *v4l2_subdev_lock_active_state(struct v4l2_subdev *sd)
 {
-	mutex_lock(&sd->state->lock);
+	mutex_lock(&sd->active_state->lock);
 
-	return sd->state;
+	return sd->active_state;
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_lock_active_state);
 
@@ -1445,18 +1451,17 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
 
 	lockdep_assert_held(&state->lock);
 
-	kvfree(dst->routes);
+	kfree(dst->routes);
 	dst->routes = NULL;
 	dst->num_routes = 0;
 
 	if (src->num_routes > 0) {
-		dst->routes = kvmalloc_array(src->num_routes,
-					     sizeof(*src->routes), GFP_KERNEL);
+		dst->routes = kmemdup(src->routes,
+				      src->num_routes * sizeof(*src->routes),
+				      GFP_KERNEL);
 		if (!dst->routes)
 			return -ENOMEM;
 
-		memcpy(dst->routes, src->routes,
-		       src->num_routes * sizeof(*src->routes));
 		dst->num_routes = src->num_routes;
 	}
 
@@ -1507,8 +1512,9 @@ v4l2_state_get_stream_format(struct v4l2_subdev_state *state, unsigned int pad,
 }
 EXPORT_SYMBOL_GPL(v4l2_state_get_stream_format);
 
-int v4l2_state_find_opposite_end(struct v4l2_subdev_krouting *routing, u32 pad,
-				 u32 stream, u32 *other_pad, u32 *other_stream)
+int v4l2_subdev_routing_find_opposite_end(const struct v4l2_subdev_krouting *routing,
+					  u32 pad, u32 stream, u32 *other_pad,
+					  u32 *other_stream)
 {
 	unsigned int i;
 
@@ -1517,37 +1523,41 @@ int v4l2_state_find_opposite_end(struct v4l2_subdev_krouting *routing, u32 pad,
 
 		if (route->source_pad == pad &&
 		    route->source_stream == stream) {
-			*other_pad = route->sink_pad;
-			*other_stream = route->sink_stream;
+			if (other_pad)
+				*other_pad = route->sink_pad;
+			if (other_stream)
+				*other_stream = route->sink_stream;
 			return 0;
 		}
 
 		if (route->sink_pad == pad && route->sink_stream == stream) {
-			*other_pad = route->source_pad;
-			*other_stream = route->source_stream;
+			if (other_pad)
+				*other_pad = route->source_pad;
+			if (other_stream)
+				*other_stream = route->source_stream;
 			return 0;
 		}
 	}
 
 	return -EINVAL;
 }
-EXPORT_SYMBOL_GPL(v4l2_state_find_opposite_end);
+EXPORT_SYMBOL_GPL(v4l2_subdev_routing_find_opposite_end);
 
 struct v4l2_mbus_framefmt *
-v4l2_state_get_opposite_stream_format(struct v4l2_subdev_state *state, u32 pad,
-				      u32 stream)
+v4l2_subdev_state_get_opposite_stream_format(struct v4l2_subdev_state *state,
+					     u32 pad, u32 stream)
 {
 	u32 other_pad, other_stream;
 	int ret;
 
-	ret = v4l2_state_find_opposite_end(&state->routing, pad, stream,
-					   &other_pad, &other_stream);
+	ret = v4l2_subdev_routing_find_opposite_end(&state->routing, pad, stream,
+						    &other_pad, &other_stream);
 	if (ret)
 		return NULL;
 
 	return v4l2_state_get_stream_format(state, other_pad, other_stream);
 }
-EXPORT_SYMBOL_GPL(v4l2_state_get_opposite_stream_format);
+EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_opposite_stream_format);
 
 int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
 			struct v4l2_subdev_format *format)
@@ -1570,7 +1580,7 @@ int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_get_fmt);
 
-int v4l2_routing_simple_verify(const struct v4l2_subdev_krouting *routing)
+int v4l2_subdev_routing_validate_1_to_1(const struct v4l2_subdev_krouting *routing)
 {
 	unsigned int i, j;
 
@@ -1592,4 +1602,24 @@ int v4l2_routing_simple_verify(const struct v4l2_subdev_krouting *routing)
 
 	return 0;
 }
-EXPORT_SYMBOL_GPL(v4l2_routing_simple_verify);
+EXPORT_SYMBOL_GPL(v4l2_subdev_routing_validate_1_to_1);
+
+struct v4l2_subdev_route *
+__v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting *routing,
+				struct v4l2_subdev_route *route)
+{
+	if (route)
+		++route;
+	else
+		route = &routing->routes[0];
+
+	for (; route < routing->routes + routing->num_routes; ++route) {
+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
+			continue;
+
+		return route;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(__v4l2_subdev_next_active_route);
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 5e50f2ded653..9754913b34f8 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -687,6 +687,9 @@ struct v4l2_subdev_ir_ops {
  * This structure only needs to be passed to the pad op if the 'which' field
  * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
  * %V4L2_SUBDEV_FORMAT_ACTIVE it is safe to pass %NULL.
+ *
+ * Note: This struct is also used in active state, and the try_ prefix is
+ * historical and to be removed.
  */
 struct v4l2_subdev_pad_config {
 	struct v4l2_mbus_framefmt try_fmt;
@@ -751,6 +754,7 @@ struct v4l2_subdev_krouting {
  * %V4L2_SUBDEV_FORMAT_ACTIVE it is safe to pass %NULL.
  */
 struct v4l2_subdev_state {
+	/* lock for the struct v4l2_subdev_state fields */
 	struct mutex lock;
 	struct v4l2_subdev_pad_config *pads;
 	struct v4l2_subdev_krouting routing;
@@ -1008,8 +1012,8 @@ struct v4l2_subdev_platform_data {
  * @subdev_notifier: A sub-device notifier implicitly registered for the sub-
  *		     device using v4l2_async_register_subdev_sensor().
  * @pdata: common part of subdevice platform data
- * @state: active state for the subdev (NULL for subdevs tracking the state
- *	   internally)
+ * @active_state: active state for the subdev (NULL for subdevs tracking the
+ *                state internally)
  *
  * Each instance of a subdev driver should create this struct, either
  * stand-alone or embedded in a larger struct.
@@ -1053,7 +1057,7 @@ struct v4l2_subdev {
 	 * easily catch uses of state in the cases where the driver doesn't
 	 * support it.
 	 */
-	struct v4l2_subdev_state *state;
+	struct v4l2_subdev_state *active_state;
 };
 
 
@@ -1404,7 +1408,7 @@ void v4l2_subdev_cleanup(struct v4l2_subdev *sd);
 static inline struct v4l2_subdev_state *
 v4l2_subdev_get_active_state(struct v4l2_subdev *sd)
 {
-	return sd->state;
+	return sd->active_state;
 }
 
 /**
@@ -1438,8 +1442,7 @@ void v4l2_subdev_lock_state(struct v4l2_subdev_state *state);
 void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state);
 
 /**
- * v4l2_subdev_validate_and_lock_state() - Gets locked TRY or ACTIVE subdev
- *					   state
+ * v4l2_subdev_lock_and_return_state() - Gets locked TRY or ACTIVE subdev state
  * @sd: subdevice
  * @state: subdevice state as passed to the subdev op
  *
@@ -1448,22 +1451,21 @@ void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state);
  * state stored privately.
  *
  * However, newer state-aware subdev drivers, which store their active state in
- * a common place, subdev->state, expect to always get a proper state as a
- * parameter.
+ * a common place, subdev->active_state, expect to always get a proper state as
+ * a parameter.
  *
- * These state-aware drivers can use v4l2_subdev_validate_and_lock_state()
- * instead of v4l2_subdev_lock_state(). v4l2_subdev_validate_and_lock_state()
- * solves the issue by using subdev->state in case the passed state is
- * NULL.
+ * These state-aware drivers can use v4l2_subdev_lock_and_return_state() instead
+ * of v4l2_subdev_lock_state(). v4l2_subdev_lock_and_return_state() solves the
+ * issue by using subdev->state in case the passed state is NULL.
  *
  * This is a temporary helper function, and should be removed when we can ensure
  * that all drivers pass proper state when calling other subdevs.
  */
 static inline struct v4l2_subdev_state *
-v4l2_subdev_validate_and_lock_state(struct v4l2_subdev *sd,
-				    struct v4l2_subdev_state *state)
+v4l2_subdev_lock_and_return_state(struct v4l2_subdev *sd,
+				  struct v4l2_subdev_state *state)
 {
-	state = state ? state : sd->state;
+	state = state ? state : sd->active_state;
 
 	v4l2_subdev_lock_state(state);
 
@@ -1518,7 +1520,7 @@ v4l2_state_get_stream_format(struct v4l2_subdev_state *state, unsigned int pad,
 			     u32 stream);
 
 /**
- * v4l2_state_find_opposite_end() - Find the opposite stream
+ * v4l2_subdev_routing_find_opposite_end() - Find the opposite stream
  * @routing: routing used to find the opposite side
  * @pad: pad id
  * @stream: stream id
@@ -1528,14 +1530,18 @@ v4l2_state_get_stream_format(struct v4l2_subdev_state *state, unsigned int pad,
  * This function uses the routing table to find the pad + stream which is
  * opposite the given pad + stream.
  *
+ * @other_pad and/or @other_stream can be NULL if the caller does not need the
+ * value.
+ *
  * Returns 0 on success, or -EINVAL if no matching route is found.
  */
-int v4l2_state_find_opposite_end(struct v4l2_subdev_krouting *routing, u32 pad,
-				 u32 stream, u32 *other_pad, u32 *other_stream);
+int v4l2_subdev_routing_find_opposite_end(const struct v4l2_subdev_krouting *routing,
+					  u32 pad, u32 stream, u32 *other_pad,
+					  u32 *other_stream);
 
 /**
- * v4l2_state_get_opposite_stream_format() - Get pointer to opposite stream
- *					     format
+ * v4l2_subdev_state_get_opposite_stream_format() - Get pointer to opposite
+ *                                                  stream format
  * @state: subdevice state
  * @pad: pad id
  * @stream: stream id
@@ -1546,8 +1552,9 @@ int v4l2_state_find_opposite_end(struct v4l2_subdev_krouting *routing, u32 pad,
  * If the state does not contain the given pad + stream, NULL is returned.
  */
 struct v4l2_mbus_framefmt *
-v4l2_state_get_opposite_stream_format(struct v4l2_subdev_state *state, u32 pad,
-				      u32 stream);
+v4l2_subdev_state_get_opposite_stream_format(struct v4l2_subdev_state *state,
+					     u32 pad, u32 stream);
+
 /**
  * v4l2_subdev_get_fmt() - Fill format based on state
  * @sd: subdevice
@@ -1566,8 +1573,8 @@ int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
 			struct v4l2_subdev_format *format);
 
 /**
- * v4l2_routing_simple_verify() - Verify that all streams are non-overlapping
- *				  1-to-1 streams
+ * v4l2_subdev_routing_validate_1_to_1() - Verify that all streams are
+ *                                         non-overlapping 1-to-1 streams
  * @routing: routing to verify
  *
  * This verifies that the given routing contains only non-overlapping 1-to-1
@@ -1577,6 +1584,19 @@ int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
  *
  * Returns 0 on success, error value otherwise.
  */
-int v4l2_routing_simple_verify(const struct v4l2_subdev_krouting *routing);
+int v4l2_subdev_routing_validate_1_to_1(const struct v4l2_subdev_krouting *routing);
+
+struct v4l2_subdev_route *
+__v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting *routing,
+				struct v4l2_subdev_route *route);
+
+/**
+ * for_each_active_route - iterate on all active routes of a routing table
+ * @routing: The routing table
+ * @route: The route iterator
+ */
+#define for_each_active_route(routing, route) \
+	for ((route) = NULL;                  \
+	     ((route) = __v4l2_subdev_next_active_route((routing), (route)));)
 
 #endif
diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
index f61ace2b5755..480891dba193 100644
--- a/include/uapi/linux/v4l2-subdev.h
+++ b/include/uapi/linux/v4l2-subdev.h
@@ -202,17 +202,20 @@ struct v4l2_subdev_capability {
 /* The v4l2 sub-device video device node is registered in read-only mode. */
 #define V4L2_SUBDEV_CAP_RO_SUBDEV		0x00000001
 
+/* The v4l2 sub-device supports multiplexed streams. */
+#define V4L2_SUBDEV_CAP_MPLEXED			0x00000002
+
 /*
  * Is the route active? An active route will start when streaming is enabled
  * on a video node.
  */
-#define V4L2_SUBDEV_ROUTE_FL_ACTIVE		(1 << 0)
+#define V4L2_SUBDEV_ROUTE_FL_ACTIVE		_BITUL(0)
 
 /*
  * Is the route immutable, i.e. can it be activated and inactivated?
  * Set by the driver.
  */
-#define V4L2_SUBDEV_ROUTE_FL_IMMUTABLE		(1 << 1)
+#define V4L2_SUBDEV_ROUTE_FL_IMMUTABLE		_BITUL(1)
 
 /*
  * Is the route a source endpoint? A source endpoint route refers to a stream
@@ -221,7 +224,7 @@ struct v4l2_subdev_capability {
  * fields are unused.
  * Set by the driver.
  */
-#define V4L2_SUBDEV_ROUTE_FL_SOURCE		(1 << 2)
+#define V4L2_SUBDEV_ROUTE_FL_SOURCE		_BITUL(2)
 
 /**
  * struct v4l2_subdev_route - A route inside a subdev
@@ -246,15 +249,15 @@ struct v4l2_subdev_route {
  * struct v4l2_subdev_routing - Subdev routing information
  *
  * @which: configuration type (from enum v4l2_subdev_format_whence)
- * @routes: pointer to the routes array
  * @num_routes: the total number of routes in the routes array
+ * @routes: pointer to the routes array
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_routing {
 	__u32 which;
-	__u64 routes;
 	__u32 num_routes;
-	__u32 reserved[5];
+	__u64 routes;
+	__u32 reserved[6];
 };
 
 /* Backwards compatibility define --- to be removed */

Re: [PATCH v10 01/38] media: subdev: rename subdev-state alloc & free

[Jacopo Mondi]

Hi Tomi,

On Tue, Nov 30, 2021 at 04:14:59PM +0200, Tomi Valkeinen wrote:
> v4l2_subdev_alloc_state() and v4l2_subdev_free_state() are not supposed
> to be used by the drivers. However, we do have a few drivers that use
> those at the moment, so we need to expose these functions for the time
> being.
>
> Prefix the functions with __ to mark the functions as internal.
>
> At the same time, rename them to v4l2_subdev_state_alloc and
> v4l2_subdev_state_free to match the style used for other functions like
> video_device_alloc() and media_request_alloc().
>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>

Reviewed-by: Jacopo Mondi <jacopo@jmondi.org>

> ---
>  drivers/media/platform/rcar-vin/rcar-v4l2.c |  4 ++--
>  drivers/media/platform/vsp1/vsp1_entity.c   |  4 ++--
>  drivers/media/v4l2-core/v4l2-subdev.c       | 12 ++++++------
>  drivers/staging/media/tegra-video/vi.c      |  4 ++--
>  include/media/v4l2-subdev.h                 | 10 +++++-----
>  5 files changed, 17 insertions(+), 17 deletions(-)
>
> diff --git a/drivers/media/platform/rcar-vin/rcar-v4l2.c b/drivers/media/platform/rcar-vin/rcar-v4l2.c
> index 0d141155f0e3..ba1d16ab1651 100644
> --- a/drivers/media/platform/rcar-vin/rcar-v4l2.c
> +++ b/drivers/media/platform/rcar-vin/rcar-v4l2.c
> @@ -252,7 +252,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
>  	u32 width, height;
>  	int ret;
>
> -	sd_state = v4l2_subdev_alloc_state(sd);
> +	sd_state = __v4l2_subdev_state_alloc(sd);
>  	if (IS_ERR(sd_state))
>  		return PTR_ERR(sd_state);
>
> @@ -288,7 +288,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
>
>  	rvin_format_align(vin, pix);
>  done:
> -	v4l2_subdev_free_state(sd_state);
> +	__v4l2_subdev_state_free(sd_state);
>
>  	return ret;
>  }
> diff --git a/drivers/media/platform/vsp1/vsp1_entity.c b/drivers/media/platform/vsp1/vsp1_entity.c
> index 823c15facd1b..869cadc1468d 100644
> --- a/drivers/media/platform/vsp1/vsp1_entity.c
> +++ b/drivers/media/platform/vsp1/vsp1_entity.c
> @@ -675,7 +675,7 @@ int vsp1_entity_init(struct vsp1_device *vsp1, struct vsp1_entity *entity,
>  	 * Allocate the pad configuration to store formats and selection
>  	 * rectangles.
>  	 */
> -	entity->config = v4l2_subdev_alloc_state(&entity->subdev);
> +	entity->config = __v4l2_subdev_state_alloc(&entity->subdev);
>  	if (IS_ERR(entity->config)) {
>  		media_entity_cleanup(&entity->subdev.entity);
>  		return PTR_ERR(entity->config);
> @@ -690,6 +690,6 @@ void vsp1_entity_destroy(struct vsp1_entity *entity)
>  		entity->ops->destroy(entity);
>  	if (entity->subdev.ctrl_handler)
>  		v4l2_ctrl_handler_free(entity->subdev.ctrl_handler);
> -	v4l2_subdev_free_state(entity->config);
> +	__v4l2_subdev_state_free(entity->config);
>  	media_entity_cleanup(&entity->subdev.entity);
>  }
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 5d27a27cc2f2..fe49c86a9b02 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -28,7 +28,7 @@ static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
>  {
>  	struct v4l2_subdev_state *state;
>
> -	state = v4l2_subdev_alloc_state(sd);
> +	state = __v4l2_subdev_state_alloc(sd);
>  	if (IS_ERR(state))
>  		return PTR_ERR(state);
>
> @@ -39,7 +39,7 @@ static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
>
>  static void subdev_fh_free(struct v4l2_subdev_fh *fh)
>  {
> -	v4l2_subdev_free_state(fh->state);
> +	__v4l2_subdev_state_free(fh->state);
>  	fh->state = NULL;
>  }
>
> @@ -870,7 +870,7 @@ int v4l2_subdev_link_validate(struct media_link *link)
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
>
> -struct v4l2_subdev_state *v4l2_subdev_alloc_state(struct v4l2_subdev *sd)
> +struct v4l2_subdev_state *__v4l2_subdev_state_alloc(struct v4l2_subdev *sd)
>  {
>  	struct v4l2_subdev_state *state;
>  	int ret;
> @@ -903,9 +903,9 @@ struct v4l2_subdev_state *v4l2_subdev_alloc_state(struct v4l2_subdev *sd)
>
>  	return ERR_PTR(ret);
>  }
> -EXPORT_SYMBOL_GPL(v4l2_subdev_alloc_state);
> +EXPORT_SYMBOL_GPL(__v4l2_subdev_state_alloc);
>
> -void v4l2_subdev_free_state(struct v4l2_subdev_state *state)
> +void __v4l2_subdev_state_free(struct v4l2_subdev_state *state)
>  {
>  	if (!state)
>  		return;
> @@ -913,7 +913,7 @@ void v4l2_subdev_free_state(struct v4l2_subdev_state *state)
>  	kvfree(state->pads);
>  	kfree(state);
>  }
> -EXPORT_SYMBOL_GPL(v4l2_subdev_free_state);
> +EXPORT_SYMBOL_GPL(__v4l2_subdev_state_free);
>
>  #endif /* CONFIG_MEDIA_CONTROLLER */
>
> diff --git a/drivers/staging/media/tegra-video/vi.c b/drivers/staging/media/tegra-video/vi.c
> index d321790b07d9..66b9ce160472 100644
> --- a/drivers/staging/media/tegra-video/vi.c
> +++ b/drivers/staging/media/tegra-video/vi.c
> @@ -507,7 +507,7 @@ static int __tegra_channel_try_format(struct tegra_vi_channel *chan,
>  	if (!subdev)
>  		return -ENODEV;
>
> -	sd_state = v4l2_subdev_alloc_state(subdev);
> +	sd_state = __v4l2_subdev_state_alloc(subdev);
>  	if (IS_ERR(sd_state))
>  		return PTR_ERR(sd_state);
>  	/*
> @@ -558,7 +558,7 @@ static int __tegra_channel_try_format(struct tegra_vi_channel *chan,
>  	v4l2_fill_pix_format(pix, &fmt.format);
>  	tegra_channel_fmt_align(chan, pix, fmtinfo->bpp);
>
> -	v4l2_subdev_free_state(sd_state);
> +	__v4l2_subdev_state_free(sd_state);
>
>  	return 0;
>  }
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 95ec18c2f49c..e52bf508c75b 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -1135,20 +1135,20 @@ int v4l2_subdev_link_validate_default(struct v4l2_subdev *sd,
>  int v4l2_subdev_link_validate(struct media_link *link);
>
>  /**
> - * v4l2_subdev_alloc_state - allocate v4l2_subdev_state
> + * __v4l2_subdev_state_alloc - allocate v4l2_subdev_state
>   *
>   * @sd: pointer to &struct v4l2_subdev for which the state is being allocated.
>   *
> - * Must call v4l2_subdev_free_state() when state is no longer needed.
> + * Must call __v4l2_subdev_state_free() when state is no longer needed.
>   */
> -struct v4l2_subdev_state *v4l2_subdev_alloc_state(struct v4l2_subdev *sd);
> +struct v4l2_subdev_state *__v4l2_subdev_state_alloc(struct v4l2_subdev *sd);
>
>  /**
> - * v4l2_subdev_free_state - free a v4l2_subdev_state
> + * __v4l2_subdev_state_free - free a v4l2_subdev_state
>   *
>   * @state: v4l2_subdev_state to be freed.
>   */
> -void v4l2_subdev_free_state(struct v4l2_subdev_state *state);
> +void __v4l2_subdev_state_free(struct v4l2_subdev_state *state);
>
>  #endif /* CONFIG_MEDIA_CONTROLLER */
>
> --
> 2.25.1
>

Re: [PATCH v10 02/38] media: subdev: add active state to struct v4l2_subdev

[Jacopo Mondi]

Hi Tomi,

On Tue, Nov 30, 2021 at 04:15:00PM +0200, Tomi Valkeinen wrote:
> Add a new 'active_state' field to struct v4l2_subdev to which we can
> store the active state of a subdev. This will place the subdev
> configuration into a known place, allowing us to use the state directly
> from the v4l2 framework, thus simplifying the drivers.
>
> Also add functions v4l2_subdev_init_finalize() and
> v4l2_subdev_cleanup(), which will allocate and free the active state.
> The functions are named in a generic way so that they can be also used
> for other subdev initialization work.
>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 21 +++++++++++
>  include/media/v4l2-subdev.h           | 54 +++++++++++++++++++++++++++
>  2 files changed, 75 insertions(+)
>
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index fe49c86a9b02..de160140d63b 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -943,3 +943,24 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
>  	v4l2_subdev_notify(sd, V4L2_DEVICE_NOTIFY_EVENT, (void *)ev);
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
> +
> +int v4l2_subdev_init_finalize(struct v4l2_subdev *sd)
> +{
> +	struct v4l2_subdev_state *state;
> +
> +	state = __v4l2_subdev_state_alloc(sd);
> +	if (IS_ERR(state))
> +		return PTR_ERR(state);
> +
> +	sd->active_state = state;

You could spare the local var..

> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_init_finalize);
> +
> +void v4l2_subdev_cleanup(struct v4l2_subdev *sd)
> +{
> +	__v4l2_subdev_state_free(sd->active_state);
> +	sd->active_state = NULL;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_cleanup);
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index e52bf508c75b..87c3126b2754 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -645,6 +645,9 @@ struct v4l2_subdev_ir_ops {
>   * This structure only needs to be passed to the pad op if the 'which' field
>   * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
>   * %V4L2_SUBDEV_FORMAT_ACTIVE it is safe to pass %NULL.
> + *
> + * Note: This struct is also used in active state, and the try_ prefix is
> + * historical and to be removed.
>   */
>  struct v4l2_subdev_pad_config {
>  	struct v4l2_mbus_framefmt try_fmt;
> @@ -898,6 +901,8 @@ struct v4l2_subdev_platform_data {
>   * @subdev_notifier: A sub-device notifier implicitly registered for the sub-
>   *		     device using v4l2_async_register_subdev_sensor().
>   * @pdata: common part of subdevice platform data
> + * @active_state: active state for the subdev (NULL for subdevs tracking the
> + *                state internally)

I would suggest here that init_finalize() should be used

 * @active_state: subdev active state. Initialized by calling
                  v4l2_subdev_init_finalize() by drivers that support
                  state

I would have said "the state API" but then we would need to define
what the state API as a concept :)

>   *
>   * Each instance of a subdev driver should create this struct, either
>   * stand-alone or embedded in a larger struct.
> @@ -929,6 +934,19 @@ struct v4l2_subdev {
>  	struct v4l2_async_notifier *notifier;
>  	struct v4l2_async_notifier *subdev_notifier;
>  	struct v4l2_subdev_platform_data *pdata;
> +
> +	/*
> +	 * The fields below are private, and should only be accessed via
> +	 * appropriate functions.
> +	 */
> +
> +	/*
> +	 * TODO: state should most likely be changed from a pointer to an
> +	 * embedded field. For the time being it's kept as a pointer to more
> +	 * easily catch uses of state in the cases where the driver doesn't
> +	 * support it.

what about "... For the time being it's kept as a pointer to allow drivers
to support state as an opt-in feature."

> +	 */
> +	struct v4l2_subdev_state *active_state;
>  };
>
>
> @@ -1217,4 +1235,40 @@ extern const struct v4l2_subdev_ops v4l2_subdev_call_wrappers;
>  void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
>  			      const struct v4l2_event *ev);
>
> +/**
> + * v4l2_subdev_init_finalize() - Finalize the initialization of the subdevice
> + * @sd: The subdev
> + *
> + * This finalizes the initialization of the subdev, including allocation of
> + * the active state for the subdev.
> + *
> + * This must be called by the subdev drivers that use the centralized active
> + * state, after the subdev struct has been initialized and
> + * media_entity_pads_init() has been called.

Or just "It must be called by subdev drivers that support active state
as the last operation before registering the subdev."

> + *
> + * Must call v4l2_subdev_cleanup() when the subdev is being removed.
> + */
> +int v4l2_subdev_init_finalize(struct v4l2_subdev *sd);
> +
> +/**
> + * v4l2_subdev_cleanup() - Release the resources needed by the subdevice
> + * @sd: The subdevice
> + *
> + * This will release the resources allocated in v4l2_subdev_init_finalize.
> + */
> +void v4l2_subdev_cleanup(struct v4l2_subdev *sd);
> +
> +/**
> + * v4l2_subdev_get_active_state() - Return the active subdev state for subdevice
> + * @sd: The subdevice
> + *
> + * Return the active state for the subdevice, or NULL if the subdev does not

nit:
The other functions in the file are documented with "Returns" (and
also "This function" seems to be more commonly used that just "This").

> + * support active state.
> + */
> +static inline struct v4l2_subdev_state *
> +v4l2_subdev_get_active_state(struct v4l2_subdev *sd)
> +{
> +	return sd->active_state;
> +}
> +

This is only used by the core, and by top drivers when doing:

      ret = v4l2_subdev_call(sd, pad, enum_frame_size,
                             v4l2_subdev_get_active_state(sd), &fse);

To pass to the subdev its own state.

I wonder if a macro like

#define v4l2_subdev_call_state(sd, o, f, args)					\
	({									\
	 struct v4l2_subdev_state *st = v4l2_subdev_get_active_state(sd);	\
	 v4l2_subdev_call(sd, o, f, st, args);					\
	 })

Could help simplify the state API.

Please be patient if I digress a bit more on the API now..

Looking forward in the series we'll end up having 5 functions to
manage state, if I've not missed any

1) struct v4l2_subdev_state *v4l2_subdev_get_active_state(struct v4l2_subdev *sd);

2) struct v4l2_subdev_state *v4l2_subdev_lock_active_state(struct v4l2_subdev *sd);

3) void v4l2_subdev_lock_state(struct v4l2_subdev_state *state);

4) void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state);

5) struct v4l2_subdev_state *v4l2_subdev_lock_and_return_state(struct v4l2_subdev *sd,
				                               struct v4l2_subdev_state *state)

1) and 2) are helpers to get (locked/unlocked) v4l2_subdevice.state,
used in operations which implicitly operates on the active state
(s_stream etc)

I wonder if we shouldn't just have a locked v4l2_subdev_get_state()
and an optional unlocked version which can be used by the core in
place of v4l2_subdev_get_active_state() and by drivers which know what
they're doing by using the unlocked version

3) and 4) lock the state at hand (to be used in operations with a
'state' parameter)

5) is currently unused and it still feels a bit ill-defined. If the state
passed in as parameter is NULL then deflect it to sd->state, so I
guess it assumes that the user is a state-aware subdev driver which
needs to protect against in-kernel callers that do no pass in a valid state to
the operation call. This implies that if the caller passes in a NULL
state for the TRY format, this gets deflected to the subdev's active
state, something which should not happen, right ? I would be tempted
to just fail loud if !state and assume if you're moving your subdev to
use state you should be able to fix the caller as well.

What would you think of:

- v4l2_subdev_get_state(sd)
- v4l2_subdev_get_state_unlocked(sd)
  (or __v4l2_subdev_get_state(sd))
- v4l2_subdev_lock_state(state)
- v4l2_subdev_unlock_state(state)

>  #endif
> --
> 2.25.1
>

Re: [PATCH v10 03/38] media: subdev: pass also the active state to subdevs from ioctls

[Jacopo Mondi]

Hi Tomi

On Tue, Nov 30, 2021 at 04:15:01PM +0200, Tomi Valkeinen wrote:
> At the moment when a subdev op is called, the TRY subdev state
> (subdev_fh->state) is passed as a parameter even for the ACTIVE case, or
> alternatively a NULL can be passed for ACTIVE case. This used to make
> sense, as the ACTIVE state was handled internally by the subdev drivers.
>
> We now have a state for the ACTIVE case in a standard place, and can
> pass that also to the drivers. This patch changes the subdev ioctls to
> either pass the TRY or ACTIVE state to the subdev.
>
> Unfortunately many drivers call ops from other subdevs, and implicitly
> pass NULL as the state, so this is just a partial solution. A coccinelle
> spatch could perhaps be created which fixes the drivers' subdev calls.
>
> For all current upstream drivers this doesn't matter, as they do not
> expect to get a valid state for ACTIVE case. But future drivers which
> support multiplexed streaming and routing will depend on getting a state
> for both active and try cases.
>
> For new drivers we can mandate that the pipelines where the drivers are
> used need to pass the state properly, or preferably, not call such
> subdev ops at all.
>
> However, if an existing subdev driver is changed to support multiplexed
> streams, the driver has to consider cases where its ops will be called
> with NULL state. The problem can easily be solved by using the
> v4l2_subdev_validate_and_lock_state() helper, introduced in a follow up

Now called v4l2_subdev_lock_and_return_state() and introduced in a
previous patch.

I would still push for state-aware subdev drivers to BUG() on !state
and tell them to fix the caller. Is this too harsh ?

> patch.
>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 73 +++++++++++++++++++++++----
>  1 file changed, 63 insertions(+), 10 deletions(-)
>
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index de160140d63b..3289875d9ec1 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -353,6 +353,53 @@ const struct v4l2_subdev_ops v4l2_subdev_call_wrappers = {
>  EXPORT_SYMBOL(v4l2_subdev_call_wrappers);
>
>  #if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
> +
> +static struct v4l2_subdev_state *
> +subdev_ioctl_get_state(struct v4l2_subdev *sd, struct v4l2_subdev_fh *subdev_fh,
> +		       unsigned int cmd, void *arg)
> +{
> +	u32 which;
> +
> +	switch (cmd) {
> +	default:
> +		return NULL;
> +
> +	case VIDIOC_SUBDEV_G_FMT:
> +	case VIDIOC_SUBDEV_S_FMT: {
> +		which = ((struct v4l2_subdev_format *)arg)->which;
> +		break;
> +	}
> +	case VIDIOC_SUBDEV_G_CROP:
> +	case VIDIOC_SUBDEV_S_CROP: {
> +		which = ((struct v4l2_subdev_crop *)arg)->which;
> +		break;
> +	}
> +	case VIDIOC_SUBDEV_ENUM_MBUS_CODE: {
> +		which = ((struct v4l2_subdev_mbus_code_enum *)arg)->which;
> +		break;
> +	}
> +	case VIDIOC_SUBDEV_ENUM_FRAME_SIZE: {
> +		which = ((struct v4l2_subdev_frame_size_enum *)arg)->which;
> +		break;
> +	}

No need for braces if I'm not mistaken, and also some blocks are
followed by an empty line some or not.

> +
> +	case VIDIOC_SUBDEV_ENUM_FRAME_INTERVAL: {
> +		which = ((struct v4l2_subdev_frame_interval_enum *)arg)->which;
> +		break;
> +	}
> +
> +	case VIDIOC_SUBDEV_G_SELECTION:
> +	case VIDIOC_SUBDEV_S_SELECTION: {
> +		which = ((struct v4l2_subdev_selection *)arg)->which;
> +		break;
> +	}
> +	}
> +
> +	return which == V4L2_SUBDEV_FORMAT_TRY ?
> +			     subdev_fh->state :
> +			     v4l2_subdev_get_active_state(sd);
> +}
> +
>  static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  {
>  	struct video_device *vdev = video_devdata(file);
> @@ -360,8 +407,11 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  	struct v4l2_fh *vfh = file->private_data;
>  	struct v4l2_subdev_fh *subdev_fh = to_v4l2_subdev_fh(vfh);
>  	bool ro_subdev = test_bit(V4L2_FL_SUBDEV_RO_DEVNODE, &vdev->flags);
> +	struct v4l2_subdev_state *state;
>  	int rval;
>
> +	state = subdev_ioctl_get_state(sd, subdev_fh, cmd, arg);
> +

There's a possibility NULL is returned if a new subdev_fh is added and
the above not updated. Should we BUG_ON() ?

>  	switch (cmd) {
>  	case VIDIOC_SUBDEV_QUERYCAP: {
>  		struct v4l2_subdev_capability *cap = arg;
> @@ -484,7 +534,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>
>  		memset(format->reserved, 0, sizeof(format->reserved));
>  		memset(format->format.reserved, 0, sizeof(format->format.reserved));
> -		return v4l2_subdev_call(sd, pad, get_fmt, subdev_fh->state, format);
> +		return v4l2_subdev_call(sd, pad, get_fmt, state, format);
>  	}
>
>  	case VIDIOC_SUBDEV_S_FMT: {
> @@ -495,7 +545,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>
>  		memset(format->reserved, 0, sizeof(format->reserved));
>  		memset(format->format.reserved, 0, sizeof(format->format.reserved));
> -		return v4l2_subdev_call(sd, pad, set_fmt, subdev_fh->state, format);
> +		return v4l2_subdev_call(sd, pad, set_fmt, state, format);
>  	}
>
>  	case VIDIOC_SUBDEV_G_CROP: {
> @@ -509,7 +559,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  		sel.target = V4L2_SEL_TGT_CROP;
>
>  		rval = v4l2_subdev_call(
> -			sd, pad, get_selection, subdev_fh->state, &sel);
> +			sd, pad, get_selection, state, &sel);
>
>  		crop->rect = sel.r;
>
> @@ -531,7 +581,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  		sel.r = crop->rect;
>
>  		rval = v4l2_subdev_call(
> -			sd, pad, set_selection, subdev_fh->state, &sel);
> +			sd, pad, set_selection, state, &sel);
>
>  		crop->rect = sel.r;
>
> @@ -542,7 +592,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  		struct v4l2_subdev_mbus_code_enum *code = arg;
>
>  		memset(code->reserved, 0, sizeof(code->reserved));
> -		return v4l2_subdev_call(sd, pad, enum_mbus_code, subdev_fh->state,
> +		return v4l2_subdev_call(sd, pad, enum_mbus_code, state,
>  					code);
>  	}
>
> @@ -550,7 +600,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  		struct v4l2_subdev_frame_size_enum *fse = arg;
>
>  		memset(fse->reserved, 0, sizeof(fse->reserved));
> -		return v4l2_subdev_call(sd, pad, enum_frame_size, subdev_fh->state,
> +		return v4l2_subdev_call(sd, pad, enum_frame_size, state,
>  					fse);
>  	}
>
> @@ -575,7 +625,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  		struct v4l2_subdev_frame_interval_enum *fie = arg;
>
>  		memset(fie->reserved, 0, sizeof(fie->reserved));
> -		return v4l2_subdev_call(sd, pad, enum_frame_interval, subdev_fh->state,
> +		return v4l2_subdev_call(sd, pad, enum_frame_interval, state,
>  					fie);
>  	}
>
> @@ -584,7 +634,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>
>  		memset(sel->reserved, 0, sizeof(sel->reserved));
>  		return v4l2_subdev_call(
> -			sd, pad, get_selection, subdev_fh->state, sel);
> +			sd, pad, get_selection, state, sel);
>  	}
>
>  	case VIDIOC_SUBDEV_S_SELECTION: {
> @@ -595,7 +645,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>
>  		memset(sel->reserved, 0, sizeof(sel->reserved));
>  		return v4l2_subdev_call(
> -			sd, pad, set_selection, subdev_fh->state, sel);
> +			sd, pad, set_selection, state, sel);
>  	}
>
>  	case VIDIOC_G_EDID: {
> @@ -829,10 +879,13 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
>  	if (is_media_entity_v4l2_subdev(pad->entity)) {
>  		struct v4l2_subdev *sd =
>  			media_entity_to_v4l2_subdev(pad->entity);
> +		struct v4l2_subdev_state *state;
> +
> +		state = v4l2_subdev_get_active_state(sd);
>
>  		fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
>  		fmt->pad = pad->index;
> -		return v4l2_subdev_call(sd, pad, get_fmt, NULL, fmt);
> +		return v4l2_subdev_call(sd, pad, get_fmt, state, fmt);

Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>

Thanks
   j
>  	}
>
>  	WARN(pad->entity->function != MEDIA_ENT_F_IO_V4L,
> --
> 2.25.1
>

Re: [PATCH v10 04/38] media: subdev: add subdev state locking

[Jacopo Mondi]

Hi Tomi,

On Tue, Nov 30, 2021 at 04:15:02PM +0200, Tomi Valkeinen wrote:
> The V4L2 subdevs have managed without centralized locking for the state
> (previously pad_config), as the TRY state is supposedly safe (although I
> believe two TRY ioctls for the same fd would race), and the ACTIVE
> state, and its locking, is managed by the drivers internally.
>
> We now have ACTIVE state in a centralized position, and need locking.

I would use 'active' instead of ACTIVE

> Strictly speaking the locking is only needed for new drivers that use
> the new state, as the current drivers continue behaving as they used to.
>
> Add a mutex to the struct v4l2_subdev_state, along with a few helper
> functions for locking/unlocking.
>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/platform/rcar-vin/rcar-v4l2.c |  3 +-
>  drivers/media/platform/vsp1/vsp1_entity.c   |  4 +-
>  drivers/media/v4l2-core/v4l2-subdev.c       | 38 +++++++++++++---
>  drivers/staging/media/tegra-video/vi.c      |  4 +-
>  include/media/v4l2-subdev.h                 | 50 ++++++++++++++++++++-
>  5 files changed, 89 insertions(+), 10 deletions(-)
>
> diff --git a/drivers/media/platform/rcar-vin/rcar-v4l2.c b/drivers/media/platform/rcar-vin/rcar-v4l2.c
> index ba1d16ab1651..e6bd94d63e4f 100644
> --- a/drivers/media/platform/rcar-vin/rcar-v4l2.c
> +++ b/drivers/media/platform/rcar-vin/rcar-v4l2.c
> @@ -244,6 +244,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
>  {
>  	struct v4l2_subdev *sd = vin_to_source(vin);
>  	struct v4l2_subdev_state *sd_state;
> +	static struct lock_class_key key;
>  	struct v4l2_subdev_format format = {
>  		.which = which,
>  		.pad = vin->parallel.source_pad,
> @@ -252,7 +253,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
>  	u32 width, height;
>  	int ret;
>
> -	sd_state = __v4l2_subdev_state_alloc(sd);
> +	sd_state = __v4l2_subdev_state_alloc(sd, "rvin:state->lock", &key);

Is key needed in the callers ? can it be moved to
__v4l2_subdev_state_alloc() ?

This apart
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>

Thanks
   j

>  	if (IS_ERR(sd_state))
>  		return PTR_ERR(sd_state);
>
> diff --git a/drivers/media/platform/vsp1/vsp1_entity.c b/drivers/media/platform/vsp1/vsp1_entity.c
> index 869cadc1468d..e607c3ae2520 100644
> --- a/drivers/media/platform/vsp1/vsp1_entity.c
> +++ b/drivers/media/platform/vsp1/vsp1_entity.c
> @@ -613,6 +613,7 @@ int vsp1_entity_init(struct vsp1_device *vsp1, struct vsp1_entity *entity,
>  		     const char *name, unsigned int num_pads,
>  		     const struct v4l2_subdev_ops *ops, u32 function)
>  {
> +	static struct lock_class_key key;
>  	struct v4l2_subdev *subdev;
>  	unsigned int i;
>  	int ret;
> @@ -675,7 +676,8 @@ int vsp1_entity_init(struct vsp1_device *vsp1, struct vsp1_entity *entity,
>  	 * Allocate the pad configuration to store formats and selection
>  	 * rectangles.
>  	 */
> -	entity->config = __v4l2_subdev_state_alloc(&entity->subdev);
> +	entity->config = __v4l2_subdev_state_alloc(&entity->subdev,
> +						   "vsp1:config->lock", &key);
>  	if (IS_ERR(entity->config)) {
>  		media_entity_cleanup(&entity->subdev.entity);
>  		return PTR_ERR(entity->config);
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 3289875d9ec1..2053fe1cd67d 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -27,8 +27,9 @@
>  static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
>  {
>  	struct v4l2_subdev_state *state;
> +	static struct lock_class_key key;
>
> -	state = __v4l2_subdev_state_alloc(sd);
> +	state = __v4l2_subdev_state_alloc(sd, "fh->state->lock", &key);
>  	if (IS_ERR(state))
>  		return PTR_ERR(state);
>
> @@ -923,7 +924,9 @@ int v4l2_subdev_link_validate(struct media_link *link)
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
>
> -struct v4l2_subdev_state *__v4l2_subdev_state_alloc(struct v4l2_subdev *sd)
> +struct v4l2_subdev_state *
> +__v4l2_subdev_state_alloc(struct v4l2_subdev *sd, const char *lock_name,
> +			  struct lock_class_key *lock_key)
>  {
>  	struct v4l2_subdev_state *state;
>  	int ret;
> @@ -932,6 +935,8 @@ struct v4l2_subdev_state *__v4l2_subdev_state_alloc(struct v4l2_subdev *sd)
>  	if (!state)
>  		return ERR_PTR(-ENOMEM);
>
> +	__mutex_init(&state->lock, lock_name, lock_key);
> +
>  	if (sd->entity.num_pads) {
>  		state->pads = kvmalloc_array(sd->entity.num_pads,
>  					     sizeof(*state->pads),
> @@ -963,6 +968,8 @@ void __v4l2_subdev_state_free(struct v4l2_subdev_state *state)
>  	if (!state)
>  		return;
>
> +	mutex_destroy(&state->lock);
> +
>  	kvfree(state->pads);
>  	kfree(state);
>  }
> @@ -997,11 +1004,12 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
>
> -int v4l2_subdev_init_finalize(struct v4l2_subdev *sd)
> +int __v4l2_subdev_init_finalize(struct v4l2_subdev *sd, const char *name,
> +				struct lock_class_key *key)
>  {
>  	struct v4l2_subdev_state *state;
>
> -	state = __v4l2_subdev_state_alloc(sd);
> +	state = __v4l2_subdev_state_alloc(sd, name, key);
>  	if (IS_ERR(state))
>  		return PTR_ERR(state);
>
> @@ -1009,7 +1017,7 @@ int v4l2_subdev_init_finalize(struct v4l2_subdev *sd)
>
>  	return 0;
>  }
> -EXPORT_SYMBOL_GPL(v4l2_subdev_init_finalize);
> +EXPORT_SYMBOL_GPL(__v4l2_subdev_init_finalize);
>
>  void v4l2_subdev_cleanup(struct v4l2_subdev *sd)
>  {
> @@ -1017,3 +1025,23 @@ void v4l2_subdev_cleanup(struct v4l2_subdev *sd)
>  	sd->active_state = NULL;
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_cleanup);
> +
> +struct v4l2_subdev_state *v4l2_subdev_lock_active_state(struct v4l2_subdev *sd)
> +{
> +	mutex_lock(&sd->active_state->lock);
> +
> +	return sd->active_state;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_lock_active_state);
> +
> +void v4l2_subdev_lock_state(struct v4l2_subdev_state *state)
> +{
> +	mutex_lock(&state->lock);
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_lock_state);
> +
> +void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state)
> +{
> +	mutex_unlock(&state->lock);
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_unlock_state);
> diff --git a/drivers/staging/media/tegra-video/vi.c b/drivers/staging/media/tegra-video/vi.c
> index 66b9ce160472..3516735f0cad 100644
> --- a/drivers/staging/media/tegra-video/vi.c
> +++ b/drivers/staging/media/tegra-video/vi.c
> @@ -491,6 +491,7 @@ static int __tegra_channel_try_format(struct tegra_vi_channel *chan,
>  				      struct v4l2_pix_format *pix)
>  {
>  	const struct tegra_video_format *fmtinfo;
> +	static struct lock_class_key key;
>  	struct v4l2_subdev *subdev;
>  	struct v4l2_subdev_format fmt;
>  	struct v4l2_subdev_state *sd_state;
> @@ -507,7 +508,8 @@ static int __tegra_channel_try_format(struct tegra_vi_channel *chan,
>  	if (!subdev)
>  		return -ENODEV;
>
> -	sd_state = __v4l2_subdev_state_alloc(subdev);
> +	sd_state = __v4l2_subdev_state_alloc(subdev, "tegra:state->lock",
> +					     &key);
>  	if (IS_ERR(sd_state))
>  		return PTR_ERR(sd_state);
>  	/*
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 87c3126b2754..1810dde9c7fc 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -658,6 +658,7 @@ struct v4l2_subdev_pad_config {
>  /**
>   * struct v4l2_subdev_state - Used for storing subdev state information.
>   *
> + * @lock: mutex for the state
>   * @pads: &struct v4l2_subdev_pad_config array
>   *
>   * This structure only needs to be passed to the pad op if the 'which' field
> @@ -665,6 +666,8 @@ struct v4l2_subdev_pad_config {
>   * %V4L2_SUBDEV_FORMAT_ACTIVE it is safe to pass %NULL.
>   */
>  struct v4l2_subdev_state {
> +	/* lock for the struct v4l2_subdev_state fields */
> +	struct mutex lock;
>  	struct v4l2_subdev_pad_config *pads;
>  };
>
> @@ -1156,10 +1159,14 @@ int v4l2_subdev_link_validate(struct media_link *link);
>   * __v4l2_subdev_state_alloc - allocate v4l2_subdev_state
>   *
>   * @sd: pointer to &struct v4l2_subdev for which the state is being allocated.
> + * @lock_name: name of the state lock
> + * @key: lock_class_key for the lock
>   *
>   * Must call __v4l2_subdev_state_free() when state is no longer needed.
>   */
> -struct v4l2_subdev_state *__v4l2_subdev_state_alloc(struct v4l2_subdev *sd);
> +struct v4l2_subdev_state *__v4l2_subdev_state_alloc(struct v4l2_subdev *sd,
> +						    const char *lock_name,
> +						    struct lock_class_key *key);
>
>  /**
>   * __v4l2_subdev_state_free - free a v4l2_subdev_state
> @@ -1248,7 +1255,16 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
>   *
>   * Must call v4l2_subdev_cleanup() when the subdev is being removed.
>   */
> -int v4l2_subdev_init_finalize(struct v4l2_subdev *sd);
> +#define v4l2_subdev_init_finalize(sd)                                          \
> +	({                                                                     \
> +		static struct lock_class_key __key;                            \
> +		const char *name = KBUILD_BASENAME                             \
> +			":" __stringify(__LINE__) ":subdev->state->lock";      \
> +		__v4l2_subdev_init_finalize(sd, name, &__key);                 \
> +	})
> +
> +int __v4l2_subdev_init_finalize(struct v4l2_subdev *sd, const char *name,
> +				struct lock_class_key *key);
>
>  /**
>   * v4l2_subdev_cleanup() - Release the resources needed by the subdevice
> @@ -1271,4 +1287,34 @@ v4l2_subdev_get_active_state(struct v4l2_subdev *sd)
>  	return sd->active_state;
>  }
>
> +/**
> + * v4l2_subdev_lock_active_state() - Lock and return the active subdev state for
> + *				     subdevice
> + * @sd: The subdevice
> + *
> + * Return the locked active state for the subdevice, or NULL if the subdev
> + * does not support active state.
> + *
> + * Must be unlocked with v4l2_subdev_unlock_state() after use.
> + */
> +struct v4l2_subdev_state *v4l2_subdev_lock_active_state(struct v4l2_subdev *sd);
> +
> +/**
> + * v4l2_subdev_lock_state() - Lock the subdev state
> + * @state: The subdevice state
> + *
> + * Lock the given subdev state.
> + *
> + * Must be unlocked with v4l2_subdev_unlock_state() after use.
> + */
> +void v4l2_subdev_lock_state(struct v4l2_subdev_state *state);
> +
> +/**
> + * v4l2_subdev_unlock_state() - Unlock the subdev state
> + * @state: The subdevice state
> + *
> + * Unlock the given subdev state.
> + */
> +void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state);
> +
>  #endif
> --
> 2.25.1
>

Re: [PATCH v10 06/38] media: Documentation: add documentation about subdev state

[Jacopo Mondi]

Hi Tomi,

On Tue, Nov 30, 2021 at 04:15:04PM +0200, Tomi Valkeinen wrote:
> Add documentation about centrally managed subdev state.
>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  .../driver-api/media/v4l2-subdev.rst          | 28 +++++++++++++++++++
>  1 file changed, 28 insertions(+)
>
> diff --git a/Documentation/driver-api/media/v4l2-subdev.rst b/Documentation/driver-api/media/v4l2-subdev.rst
> index 7736da077fb8..db012729f8e3 100644
> --- a/Documentation/driver-api/media/v4l2-subdev.rst
> +++ b/Documentation/driver-api/media/v4l2-subdev.rst
> @@ -518,6 +518,34 @@ The :c:func:`v4l2_i2c_new_subdev` function will call
>  :c:type:`i2c_board_info` structure using the ``client_type`` and the
>  ``addr`` to fill it.
>
> +Centrally managed subdev active state
> +-------------------------------------
> +
> +Traditionally V4L2 subdev drivers maintained internal state for the active
> +configuration for the subdev. This is often implemented as an array of struct
> +v4l2_mbus_framefmt, one entry for each pad.

Not only v4l2_mbus_framefmt, but the 'active' configuration also
contains cropping rectangles. I would drop the last statement maybe ?

> +
> +In addition to the active configuration, each subdev filehandle has an array of
> +struct v4l2_subdev_pad_config, managed by V4L2 core, which contains the TRY
> +configuration.
> +
> +To simplify the subdev drivers the V4L2 subdev API now optionally supports a
> +centrally managed active configuration. A subdev driver must use

I would here explain what active and try states are and where they
live.

"a centrally managed active configuration represented by
:c:type:`v4l2_subdev_state`. One instance of state, which contains
the active device configuration, is associated with the sub-device itself as
part of the :c:type:`v4l2_subdev` structure, while the core associates to
each open file handle a try state, which contains the configuration
valid in the file-handle context only."

> +v4l2_subdev_init_finalize() to initialize the active state between calls to
> +media_entity_pads_init() and v4l2_*_register_subdev(), and must call
> +v4l2_subdev_cleanup() to free the state.

"Sub-device drivers can opt-in and use state to manage their active
configuration by initializing the subdevice state with a call to
v4l2_subdev_init_finalize() as the last operation before registering
the sub-device. They must also call v4l2_subdev_cleanup() to release
all the acquired resources before unregistering the sub-device. The
core automatically initializes a state for each open file handle where
to store the try configurations and releases them at file handle
closing time."

> +
> +The active state must be locked before access, and that can be done with
> +v4l2_subdev_lock_state() or v4l2_subdev_lock_active_state().

I would also explain which state to use and when.

"V4L2 sub-device operations that operates on both the :ref:`ACTIVE and
TRY formats <v4l2-subdev-format-whence>` receive by the core the
correct state to operate on an operation parameter. The sub-device driver
can access and modify the configuration stored in the provided state
after having exclusively acquired it by calling
:c:func:`v4l2_subdev_lock_state()` and released it with
:c:func:`v4l2_subdev_unlock_state()` when done.

Operations that do not receive a state parameter implicitly operates
on the subdevice active state, which drivers can exclusively access by
calling :c:func:`v4l2_subdev_lock_active_state()`. The sub-device active
state should equally be released by calling
:c:func:`v4l2_subdev_unlock_state()`.

In no occasions driver should try to manually access the state stored
in the :c:type`v4l2_subdev` or in the file handle without going
through the designated helpers".

> +
> +The V4L2 core will pass either the TRY or ACTIVE state to various subdev ops.
> +Unfortunately not all the subdev drivers comply with this yet, and may pass

If I'm not mistaken it's not the subdev drivers but the top drivers
which might not comply.

> +NULL for the ACTIVE case. This is only a problem for subdev drivers which use
> +the centrally managed active state and are used in media pipelines with older
> +subdev drivers. In these cases the called subdev ops must also handle the NULL
> +case. This can be easily managed by the use of
> +v4l2_subdev_validate_and_lock_state() helper.

depending on the previous suggestion to fail hard if !state, this
might be removed ?

Thanks
   j

> +
>  V4L2 sub-device functions and data structures
>  ---------------------------------------------
>
> --
> 2.25.1
>

Re: [PATCH v10 24/38] media: subdev: increase V4L2_FRAME_DESC_ENTRY_MAX to 8

[Jacopo Mondi]

Hi Tomi

On Tue, Nov 30, 2021 at 04:15:22PM +0200, Tomi Valkeinen wrote:
> V4L2_FRAME_DESC_ENTRY_MAX is currently set to 4. In theory it's possible
> to have an arbitrary amount of streams in a single pad, so preferably
> there should be no hardcoded maximum number.
>
> However, I believe a reasonable max is 8, which would cover a CSI-2 pad
> with 4 streams of pixel data and 4 streams of metadata.
>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>

If I'm not mistaken C-PHY supports up to 32 virtual channels ?

Can be increased later...

Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>

Thanks
  j

> ---
>  include/media/v4l2-subdev.h | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 2921885eb390..a82fc74f4646 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -360,7 +360,11 @@ struct v4l2_mbus_frame_desc_entry {
>  	} bus;
>  };
>
> -#define V4L2_FRAME_DESC_ENTRY_MAX	4
> + /*
> +  * FIXME: If this number is too small, it should be dropped altogether and the
> +  * API switched to a dynamic number of frame descriptor entries.
> +  */
> +#define V4L2_FRAME_DESC_ENTRY_MAX	8
>
>  /**
>   * enum v4l2_mbus_frame_desc_type - media bus frame description type
> --
> 2.25.1
>

Re: [PATCH v10 26/38] media: add V4L2_SUBDEV_CAP_MPLEXED

[Jacopo Mondi]

Hi Tomi,

On Tue, Nov 30, 2021 at 04:15:24PM +0200, Tomi Valkeinen wrote:
> Add a subdev capability flag to expose to userspace if a subdev supports
> multiplexed streams.
>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 4 +++-
>  include/uapi/linux/v4l2-subdev.h      | 3 +++
>  2 files changed, 6 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 2053fe1cd67d..721148e35624 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -419,7 +419,9 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>
>  		memset(cap->reserved, 0, sizeof(cap->reserved));
>  		cap->version = LINUX_VERSION_CODE;
> -		cap->capabilities = ro_subdev ? V4L2_SUBDEV_CAP_RO_SUBDEV : 0;
> +		cap->capabilities =
> +			(ro_subdev ? V4L2_SUBDEV_CAP_RO_SUBDEV : 0) |
> +			((sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED) ? V4L2_SUBDEV_CAP_MPLEXED : 0);

I had been suggested to go for:

	bool ro_subdev = test_bit(V4L2_FL_SUBDEV_RO_DEVNODE, &vdev->flags);

when introducing V4L2_SUBDEV_CAP_RO_SUBDEV.

To me it doesn't make much difference

Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>

Thanks
   j

>
>  		return 0;
>  	}
> diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
> index 658106f5b5dc..d91ab6f22fa7 100644
> --- a/include/uapi/linux/v4l2-subdev.h
> +++ b/include/uapi/linux/v4l2-subdev.h
> @@ -188,6 +188,9 @@ struct v4l2_subdev_capability {
>  /* The v4l2 sub-device video device node is registered in read-only mode. */
>  #define V4L2_SUBDEV_CAP_RO_SUBDEV		0x00000001
>
> +/* The v4l2 sub-device supports multiplexed streams. */
> +#define V4L2_SUBDEV_CAP_MPLEXED			0x00000002
> +
>  /* Backwards compatibility define --- to be removed */
>  #define v4l2_subdev_edid v4l2_edid
>
> --
> 2.25.1
>

Re: [PATCH v10 25/38] media: add V4L2_SUBDEV_FL_MULTIPLEXED

[Jacopo Mondi]

Hi Tomi,

On Tue, Nov 30, 2021 at 04:15:23PM +0200, Tomi Valkeinen wrote:
> Add subdev flag V4L2_SUBDEV_FL_MULTIPLEXED. It is used to indicate that
> the subdev supports the new API with multiplexed streams (routing,
> stream configs).
>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  include/media/v4l2-subdev.h | 11 +++++++++++
>  1 file changed, 11 insertions(+)
>
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index a82fc74f4646..45861bcdccf5 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -892,6 +892,17 @@ struct v4l2_subdev_internal_ops {
>   * should set this flag.
>   */
>  #define V4L2_SUBDEV_FL_HAS_EVENTS		(1U << 3)
> +/*
> + * Set this flag if this subdev supports multiplexed streams. This means
> + * that the driver supports routing and handles the stream parameter in its
> + * v4l2_subdev_pad_ops handlers. More specifically, this means:
> + *
> + * - Centrally managed active state is enabled
> + * - Legacy pad config is _not_ supported (state->pads)

Does this still apply after pad config has been replaced by try states ?

Otherwise
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>

Thanks
   j

> + * - Routing ioctls are available
> + * - Multiple streams per pad are supported
> + */
> +#define V4L2_SUBDEV_FL_MULTIPLEXED		(1U << 4)
>
>  struct regulator_bulk_data;
>
> --
> 2.25.1
>

Re: [PATCH v10 02/38] media: subdev: add active state to struct v4l2_subdev

[Tomi Valkeinen]

Hi,

On 13/12/2021 23:30, Jacopo Mondi wrote:
> Hi Tomi,
> 
> On Tue, Nov 30, 2021 at 04:15:00PM +0200, Tomi Valkeinen wrote:
>> Add a new 'active_state' field to struct v4l2_subdev to which we can
>> store the active state of a subdev. This will place the subdev
>> configuration into a known place, allowing us to use the state directly
>> from the v4l2 framework, thus simplifying the drivers.
>>
>> Also add functions v4l2_subdev_init_finalize() and
>> v4l2_subdev_cleanup(), which will allocate and free the active state.
>> The functions are named in a generic way so that they can be also used
>> for other subdev initialization work.
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>>   drivers/media/v4l2-core/v4l2-subdev.c | 21 +++++++++++
>>   include/media/v4l2-subdev.h           | 54 +++++++++++++++++++++++++++
>>   2 files changed, 75 insertions(+)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>> index fe49c86a9b02..de160140d63b 100644
>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>> @@ -943,3 +943,24 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
>>   	v4l2_subdev_notify(sd, V4L2_DEVICE_NOTIFY_EVENT, (void *)ev);
>>   }
>>   EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
>> +
>> +int v4l2_subdev_init_finalize(struct v4l2_subdev *sd)
>> +{
>> +	struct v4l2_subdev_state *state;
>> +
>> +	state = __v4l2_subdev_state_alloc(sd);
>> +	if (IS_ERR(state))
>> +		return PTR_ERR(state);
>> +
>> +	sd->active_state = state;
> 
> You could spare the local var..

I don't like to write error values to storage fields. Here I only touch 
the field when succesfull.

>> +
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_subdev_init_finalize);
>> +
>> +void v4l2_subdev_cleanup(struct v4l2_subdev *sd)
>> +{
>> +	__v4l2_subdev_state_free(sd->active_state);
>> +	sd->active_state = NULL;
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_subdev_cleanup);
>> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
>> index e52bf508c75b..87c3126b2754 100644
>> --- a/include/media/v4l2-subdev.h
>> +++ b/include/media/v4l2-subdev.h
>> @@ -645,6 +645,9 @@ struct v4l2_subdev_ir_ops {
>>    * This structure only needs to be passed to the pad op if the 'which' field
>>    * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
>>    * %V4L2_SUBDEV_FORMAT_ACTIVE it is safe to pass %NULL.
>> + *
>> + * Note: This struct is also used in active state, and the try_ prefix is
>> + * historical and to be removed.
>>    */
>>   struct v4l2_subdev_pad_config {
>>   	struct v4l2_mbus_framefmt try_fmt;
>> @@ -898,6 +901,8 @@ struct v4l2_subdev_platform_data {
>>    * @subdev_notifier: A sub-device notifier implicitly registered for the sub-
>>    *		     device using v4l2_async_register_subdev_sensor().
>>    * @pdata: common part of subdevice platform data
>> + * @active_state: active state for the subdev (NULL for subdevs tracking the
>> + *                state internally)
> 
> I would suggest here that init_finalize() should be used
> 
>   * @active_state: subdev active state. Initialized by calling
>                    v4l2_subdev_init_finalize() by drivers that support
>                    state

Ok.

> I would have said "the state API" but then we would need to define
> what the state API as a concept :)
> 
>>    *
>>    * Each instance of a subdev driver should create this struct, either
>>    * stand-alone or embedded in a larger struct.
>> @@ -929,6 +934,19 @@ struct v4l2_subdev {
>>   	struct v4l2_async_notifier *notifier;
>>   	struct v4l2_async_notifier *subdev_notifier;
>>   	struct v4l2_subdev_platform_data *pdata;
>> +
>> +	/*
>> +	 * The fields below are private, and should only be accessed via
>> +	 * appropriate functions.
>> +	 */
>> +
>> +	/*
>> +	 * TODO: state should most likely be changed from a pointer to an
>> +	 * embedded field. For the time being it's kept as a pointer to more
>> +	 * easily catch uses of state in the cases where the driver doesn't
>> +	 * support it.
> 
> what about "... For the time being it's kept as a pointer to allow drivers
> to support state as an opt-in feature."

Well, that's not quite correct. Even if it was an embedded field, it 
would be an opt-in feature. The point with the pointer is only that if 
an opt-out driver accidentally tries to access the field, it'll get 
NULL, and hopefully fail.

>> +	 */
>> +	struct v4l2_subdev_state *active_state;
>>   };
>>
>>
>> @@ -1217,4 +1235,40 @@ extern const struct v4l2_subdev_ops v4l2_subdev_call_wrappers;
>>   void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
>>   			      const struct v4l2_event *ev);
>>
>> +/**
>> + * v4l2_subdev_init_finalize() - Finalize the initialization of the subdevice
>> + * @sd: The subdev
>> + *
>> + * This finalizes the initialization of the subdev, including allocation of
>> + * the active state for the subdev.
>> + *
>> + * This must be called by the subdev drivers that use the centralized active
>> + * state, after the subdev struct has been initialized and
>> + * media_entity_pads_init() has been called.
> 
> Or just "It must be called by subdev drivers that support active state
> as the last operation before registering the subdev."

That's not quite true either. The driver may do setup operations 
requiring the active state before registering the subdev.

>> + *
>> + * Must call v4l2_subdev_cleanup() when the subdev is being removed.
>> + */
>> +int v4l2_subdev_init_finalize(struct v4l2_subdev *sd);
>> +
>> +/**
>> + * v4l2_subdev_cleanup() - Release the resources needed by the subdevice
>> + * @sd: The subdevice
>> + *
>> + * This will release the resources allocated in v4l2_subdev_init_finalize.
>> + */
>> +void v4l2_subdev_cleanup(struct v4l2_subdev *sd);
>> +
>> +/**
>> + * v4l2_subdev_get_active_state() - Return the active subdev state for subdevice
>> + * @sd: The subdevice
>> + *
>> + * Return the active state for the subdevice, or NULL if the subdev does not
> 
> nit:
> The other functions in the file are documented with "Returns" (and
> also "This function" seems to be more commonly used that just "This").

Ok, I'll try to get these consistent.

>> + * support active state.
>> + */
>> +static inline struct v4l2_subdev_state *
>> +v4l2_subdev_get_active_state(struct v4l2_subdev *sd)
>> +{
>> +	return sd->active_state;
>> +}
>> +
> 
> This is only used by the core, and by top drivers when doing:
> 
>        ret = v4l2_subdev_call(sd, pad, enum_frame_size,
>                               v4l2_subdev_get_active_state(sd), &fse);
> 
> To pass to the subdev its own state.
> 
> I wonder if a macro like
> 
> #define v4l2_subdev_call_state(sd, o, f, args)					\
> 	({									\
> 	 struct v4l2_subdev_state *st = v4l2_subdev_get_active_state(sd);	\
> 	 v4l2_subdev_call(sd, o, f, st, args);					\
> 	 })
> 
> Could help simplify the state API.

Possibly, but we need to support the TRY case also. So I would call it 
something more specific, v4l2_subdev_call_active(), or such.

> Please be patient if I digress a bit more on the API now..
> 
> Looking forward in the series we'll end up having 5 functions to
> manage state, if I've not missed any
> 
> 1) struct v4l2_subdev_state *v4l2_subdev_get_active_state(struct v4l2_subdev *sd);
> 
> 2) struct v4l2_subdev_state *v4l2_subdev_lock_active_state(struct v4l2_subdev *sd);
> 
> 3) void v4l2_subdev_lock_state(struct v4l2_subdev_state *state);
> 
> 4) void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state);
> 
> 5) struct v4l2_subdev_state *v4l2_subdev_lock_and_return_state(struct v4l2_subdev *sd,
> 				                               struct v4l2_subdev_state *state)
> 
> 1) and 2) are helpers to get (locked/unlocked) v4l2_subdevice.state,
> used in operations which implicitly operates on the active state
> (s_stream etc)
> 
> I wonder if we shouldn't just have a locked v4l2_subdev_get_state()
> and an optional unlocked version which can be used by the core in
> place of v4l2_subdev_get_active_state() and by drivers which know what
> they're doing by using the unlocked version

I think we need to always make it clear if we're dealing with the active 
or try state (when it's relevant). And I also want to be clear if a lock 
is taken. v4l2_subdev_get_state() would hide both of those.

> 3) and 4) lock the state at hand (to be used in operations with a
> 'state' parameter)
> 
> 5) is currently unused and it still feels a bit ill-defined. If the state
> passed in as parameter is NULL then deflect it to sd->state, so I
> guess it assumes that the user is a state-aware subdev driver which
> needs to protect against in-kernel callers that do no pass in a valid state to

Correct.

> the operation call. This implies that if the caller passes in a NULL
> state for the TRY format, this gets deflected to the subdev's active
> state, something which should not happen, right ? I would be tempted

Yes. That's an illegal call, isn't it? Or does it happen and we need to 
protect against it?

> to just fail loud if !state and assume if you're moving your subdev to
> use state you should be able to fix the caller as well.

That would be nice, but I think it's not realistic. If you convert a 
sensor driver to multiplexed streams, are you going to fix (convert to 
multiplexed streams) all the bridges and SoC drivers that may use that 
sensor? How do you even find all those bridges and SoCs...

  Tomi

Re: [PATCH v10 03/38] media: subdev: pass also the active state to subdevs from ioctls

[Tomi Valkeinen]

Hi,

On 14/12/2021 09:13, Jacopo Mondi wrote:
> Hi Tomi
> 
> On Tue, Nov 30, 2021 at 04:15:01PM +0200, Tomi Valkeinen wrote:
>> At the moment when a subdev op is called, the TRY subdev state
>> (subdev_fh->state) is passed as a parameter even for the ACTIVE case, or
>> alternatively a NULL can be passed for ACTIVE case. This used to make
>> sense, as the ACTIVE state was handled internally by the subdev drivers.
>>
>> We now have a state for the ACTIVE case in a standard place, and can
>> pass that also to the drivers. This patch changes the subdev ioctls to
>> either pass the TRY or ACTIVE state to the subdev.
>>
>> Unfortunately many drivers call ops from other subdevs, and implicitly
>> pass NULL as the state, so this is just a partial solution. A coccinelle
>> spatch could perhaps be created which fixes the drivers' subdev calls.
>>
>> For all current upstream drivers this doesn't matter, as they do not
>> expect to get a valid state for ACTIVE case. But future drivers which
>> support multiplexed streaming and routing will depend on getting a state
>> for both active and try cases.
>>
>> For new drivers we can mandate that the pipelines where the drivers are
>> used need to pass the state properly, or preferably, not call such
>> subdev ops at all.
>>
>> However, if an existing subdev driver is changed to support multiplexed
>> streams, the driver has to consider cases where its ops will be called
>> with NULL state. The problem can easily be solved by using the
>> v4l2_subdev_validate_and_lock_state() helper, introduced in a follow up
> 
> Now called v4l2_subdev_lock_and_return_state() and introduced in a
> previous patch.

Ah, right.

> I would still push for state-aware subdev drivers to BUG() on !state
> and tell them to fix the caller. Is this too harsh ?

I think so. We can't break currently working drivers.

>> patch.
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
>> ---
>>   drivers/media/v4l2-core/v4l2-subdev.c | 73 +++++++++++++++++++++++----
>>   1 file changed, 63 insertions(+), 10 deletions(-)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>> index de160140d63b..3289875d9ec1 100644
>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>> @@ -353,6 +353,53 @@ const struct v4l2_subdev_ops v4l2_subdev_call_wrappers = {
>>   EXPORT_SYMBOL(v4l2_subdev_call_wrappers);
>>
>>   #if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
>> +
>> +static struct v4l2_subdev_state *
>> +subdev_ioctl_get_state(struct v4l2_subdev *sd, struct v4l2_subdev_fh *subdev_fh,
>> +		       unsigned int cmd, void *arg)
>> +{
>> +	u32 which;
>> +
>> +	switch (cmd) {
>> +	default:
>> +		return NULL;
>> +
>> +	case VIDIOC_SUBDEV_G_FMT:
>> +	case VIDIOC_SUBDEV_S_FMT: {
>> +		which = ((struct v4l2_subdev_format *)arg)->which;
>> +		break;
>> +	}
>> +	case VIDIOC_SUBDEV_G_CROP:
>> +	case VIDIOC_SUBDEV_S_CROP: {
>> +		which = ((struct v4l2_subdev_crop *)arg)->which;
>> +		break;
>> +	}
>> +	case VIDIOC_SUBDEV_ENUM_MBUS_CODE: {
>> +		which = ((struct v4l2_subdev_mbus_code_enum *)arg)->which;
>> +		break;
>> +	}
>> +	case VIDIOC_SUBDEV_ENUM_FRAME_SIZE: {
>> +		which = ((struct v4l2_subdev_frame_size_enum *)arg)->which;
>> +		break;
>> +	}
> 
> No need for braces if I'm not mistaken, and also some blocks are
> followed by an empty line some or not.

Ok.

>> +
>> +	case VIDIOC_SUBDEV_ENUM_FRAME_INTERVAL: {
>> +		which = ((struct v4l2_subdev_frame_interval_enum *)arg)->which;
>> +		break;
>> +	}
>> +
>> +	case VIDIOC_SUBDEV_G_SELECTION:
>> +	case VIDIOC_SUBDEV_S_SELECTION: {
>> +		which = ((struct v4l2_subdev_selection *)arg)->which;
>> +		break;
>> +	}
>> +	}
>> +
>> +	return which == V4L2_SUBDEV_FORMAT_TRY ?
>> +			     subdev_fh->state :
>> +			     v4l2_subdev_get_active_state(sd);
>> +}
>> +
>>   static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>>   {
>>   	struct video_device *vdev = video_devdata(file);
>> @@ -360,8 +407,11 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>>   	struct v4l2_fh *vfh = file->private_data;
>>   	struct v4l2_subdev_fh *subdev_fh = to_v4l2_subdev_fh(vfh);
>>   	bool ro_subdev = test_bit(V4L2_FL_SUBDEV_RO_DEVNODE, &vdev->flags);
>> +	struct v4l2_subdev_state *state;
>>   	int rval;
>>
>> +	state = subdev_ioctl_get_state(sd, subdev_fh, cmd, arg);
>> +
> 
> There's a possibility NULL is returned if a new subdev_fh is added and
> the above not updated. Should we BUG_ON() ?

NULL is fine, as not all ioctls need a state.

  Tomi

Re: [PATCH v10 04/38] media: subdev: add subdev state locking

[Tomi Valkeinen]

On 14/12/2021 09:42, Jacopo Mondi wrote:
> Hi Tomi,
> 
> On Tue, Nov 30, 2021 at 04:15:02PM +0200, Tomi Valkeinen wrote:
>> The V4L2 subdevs have managed without centralized locking for the state
>> (previously pad_config), as the TRY state is supposedly safe (although I
>> believe two TRY ioctls for the same fd would race), and the ACTIVE
>> state, and its locking, is managed by the drivers internally.
>>
>> We now have ACTIVE state in a centralized position, and need locking.
> 
> I would use 'active' instead of ACTIVE

Hmm, yes, the naming I use varies, I guess =). I often use capitalized 
ACTIVE/TRY to make them stand out, and not get mixed with normal words.

With active/try-state it's clear, I believe, especially if I have a dash 
there.

>> Strictly speaking the locking is only needed for new drivers that use
>> the new state, as the current drivers continue behaving as they used to.
>>
>> Add a mutex to the struct v4l2_subdev_state, along with a few helper
>> functions for locking/unlocking.
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>>   drivers/media/platform/rcar-vin/rcar-v4l2.c |  3 +-
>>   drivers/media/platform/vsp1/vsp1_entity.c   |  4 +-
>>   drivers/media/v4l2-core/v4l2-subdev.c       | 38 +++++++++++++---
>>   drivers/staging/media/tegra-video/vi.c      |  4 +-
>>   include/media/v4l2-subdev.h                 | 50 ++++++++++++++++++++-
>>   5 files changed, 89 insertions(+), 10 deletions(-)
>>
>> diff --git a/drivers/media/platform/rcar-vin/rcar-v4l2.c b/drivers/media/platform/rcar-vin/rcar-v4l2.c
>> index ba1d16ab1651..e6bd94d63e4f 100644
>> --- a/drivers/media/platform/rcar-vin/rcar-v4l2.c
>> +++ b/drivers/media/platform/rcar-vin/rcar-v4l2.c
>> @@ -244,6 +244,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
>>   {
>>   	struct v4l2_subdev *sd = vin_to_source(vin);
>>   	struct v4l2_subdev_state *sd_state;
>> +	static struct lock_class_key key;
>>   	struct v4l2_subdev_format format = {
>>   		.which = which,
>>   		.pad = vin->parallel.source_pad,
>> @@ -252,7 +253,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
>>   	u32 width, height;
>>   	int ret;
>>
>> -	sd_state = __v4l2_subdev_state_alloc(sd);
>> +	sd_state = __v4l2_subdev_state_alloc(sd, "rvin:state->lock", &key);
> 
> Is key needed in the callers ? can it be moved to
> __v4l2_subdev_state_alloc() ?

These drivers shouldn't really be using the functions, so I didn't even 
try to make it look neat.

  Tomi

Re: [PATCH v10 25/38] media: add V4L2_SUBDEV_FL_MULTIPLEXED

[Tomi Valkeinen]

On 14/12/2021 10:41, Jacopo Mondi wrote:
> Hi Tomi,
> 
> On Tue, Nov 30, 2021 at 04:15:23PM +0200, Tomi Valkeinen wrote:
>> Add subdev flag V4L2_SUBDEV_FL_MULTIPLEXED. It is used to indicate that
>> the subdev supports the new API with multiplexed streams (routing,
>> stream configs).
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>>   include/media/v4l2-subdev.h | 11 +++++++++++
>>   1 file changed, 11 insertions(+)
>>
>> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
>> index a82fc74f4646..45861bcdccf5 100644
>> --- a/include/media/v4l2-subdev.h
>> +++ b/include/media/v4l2-subdev.h
>> @@ -892,6 +892,17 @@ struct v4l2_subdev_internal_ops {
>>    * should set this flag.
>>    */
>>   #define V4L2_SUBDEV_FL_HAS_EVENTS		(1U << 3)
>> +/*
>> + * Set this flag if this subdev supports multiplexed streams. This means
>> + * that the driver supports routing and handles the stream parameter in its
>> + * v4l2_subdev_pad_ops handlers. More specifically, this means:
>> + *
>> + * - Centrally managed active state is enabled
>> + * - Legacy pad config is _not_ supported (state->pads)
> 
> Does this still apply after pad config has been replaced by try states ?

What do you mean?

If a subdev is V4L2_SUBDEV_FL_MULTIPLEXED, state->pads is NULL, so 
there's no legacy pad config available.

  Tomi

Re: [PATCH v10 02/38] media: subdev: add active state to struct v4l2_subdev

[Jacopo Mondi]

Hi Tomi

On Wed, Dec 15, 2021 at 10:06:16AM +0200, Tomi Valkeinen wrote:
> Hi,
>
> On 13/12/2021 23:30, Jacopo Mondi wrote:
> > Hi Tomi,
> >
> > On Tue, Nov 30, 2021 at 04:15:00PM +0200, Tomi Valkeinen wrote:
> > > Add a new 'active_state' field to struct v4l2_subdev to which we can
> > > store the active state of a subdev. This will place the subdev
> > > configuration into a known place, allowing us to use the state directly
> > > from the v4l2 framework, thus simplifying the drivers.
> > >
> > > Also add functions v4l2_subdev_init_finalize() and
> > > v4l2_subdev_cleanup(), which will allocate and free the active state.
> > > The functions are named in a generic way so that they can be also used
> > > for other subdev initialization work.
> > >
> > > Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > > ---
> > >   drivers/media/v4l2-core/v4l2-subdev.c | 21 +++++++++++
> > >   include/media/v4l2-subdev.h           | 54 +++++++++++++++++++++++++++
> > >   2 files changed, 75 insertions(+)
> > >
> > > diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> > > index fe49c86a9b02..de160140d63b 100644
> > > --- a/drivers/media/v4l2-core/v4l2-subdev.c
> > > +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> > > @@ -943,3 +943,24 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
> > >   	v4l2_subdev_notify(sd, V4L2_DEVICE_NOTIFY_EVENT, (void *)ev);
> > >   }
> > >   EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
> > > +
> > > +int v4l2_subdev_init_finalize(struct v4l2_subdev *sd)
> > > +{
> > > +	struct v4l2_subdev_state *state;
> > > +
> > > +	state = __v4l2_subdev_state_alloc(sd);
> > > +	if (IS_ERR(state))
> > > +		return PTR_ERR(state);
> > > +
> > > +	sd->active_state = state;
> >
> > You could spare the local var..
>
> I don't like to write error values to storage fields. Here I only touch the
> field when succesfull.
>
> > > +
> > > +	return 0;
> > > +}
> > > +EXPORT_SYMBOL_GPL(v4l2_subdev_init_finalize);
> > > +
> > > +void v4l2_subdev_cleanup(struct v4l2_subdev *sd)
> > > +{
> > > +	__v4l2_subdev_state_free(sd->active_state);
> > > +	sd->active_state = NULL;
> > > +}
> > > +EXPORT_SYMBOL_GPL(v4l2_subdev_cleanup);
> > > diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> > > index e52bf508c75b..87c3126b2754 100644
> > > --- a/include/media/v4l2-subdev.h
> > > +++ b/include/media/v4l2-subdev.h
> > > @@ -645,6 +645,9 @@ struct v4l2_subdev_ir_ops {
> > >    * This structure only needs to be passed to the pad op if the 'which' field
> > >    * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
> > >    * %V4L2_SUBDEV_FORMAT_ACTIVE it is safe to pass %NULL.
> > > + *
> > > + * Note: This struct is also used in active state, and the try_ prefix is
> > > + * historical and to be removed.
> > >    */
> > >   struct v4l2_subdev_pad_config {
> > >   	struct v4l2_mbus_framefmt try_fmt;
> > > @@ -898,6 +901,8 @@ struct v4l2_subdev_platform_data {
> > >    * @subdev_notifier: A sub-device notifier implicitly registered for the sub-
> > >    *		     device using v4l2_async_register_subdev_sensor().
> > >    * @pdata: common part of subdevice platform data
> > > + * @active_state: active state for the subdev (NULL for subdevs tracking the
> > > + *                state internally)
> >
> > I would suggest here that init_finalize() should be used
> >
> >   * @active_state: subdev active state. Initialized by calling
> >                    v4l2_subdev_init_finalize() by drivers that support
> >                    state
>
> Ok.
>
> > I would have said "the state API" but then we would need to define
> > what the state API as a concept :)
> >
> > >    *
> > >    * Each instance of a subdev driver should create this struct, either
> > >    * stand-alone or embedded in a larger struct.
> > > @@ -929,6 +934,19 @@ struct v4l2_subdev {
> > >   	struct v4l2_async_notifier *notifier;
> > >   	struct v4l2_async_notifier *subdev_notifier;
> > >   	struct v4l2_subdev_platform_data *pdata;
> > > +
> > > +	/*
> > > +	 * The fields below are private, and should only be accessed via
> > > +	 * appropriate functions.
> > > +	 */
> > > +
> > > +	/*
> > > +	 * TODO: state should most likely be changed from a pointer to an
> > > +	 * embedded field. For the time being it's kept as a pointer to more
> > > +	 * easily catch uses of state in the cases where the driver doesn't
> > > +	 * support it.
> >
> > what about "... For the time being it's kept as a pointer to allow drivers
> > to support state as an opt-in feature."
>
> Well, that's not quite correct. Even if it was an embedded field, it would
> be an opt-in feature. The point with the pointer is only that if an opt-out
> driver accidentally tries to access the field, it'll get NULL, and hopefully
> fail.

Ok, what I mean is that being a pointer, it's easy to identify if a
driver has opted-in. Whatever, we're saying the same thing

>
> > > +	 */
> > > +	struct v4l2_subdev_state *active_state;
> > >   };
> > >
> > >
> > > @@ -1217,4 +1235,40 @@ extern const struct v4l2_subdev_ops v4l2_subdev_call_wrappers;
> > >   void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
> > >   			      const struct v4l2_event *ev);
> > >
> > > +/**
> > > + * v4l2_subdev_init_finalize() - Finalize the initialization of the subdevice
> > > + * @sd: The subdev
> > > + *
> > > + * This finalizes the initialization of the subdev, including allocation of
> > > + * the active state for the subdev.
> > > + *
> > > + * This must be called by the subdev drivers that use the centralized active
> > > + * state, after the subdev struct has been initialized and
> > > + * media_entity_pads_init() has been called.
> >
> > Or just "It must be called by subdev drivers that support active state
> > as the last operation before registering the subdev."
>
> That's not quite true either. The driver may do setup operations requiring
> the active state before registering the subdev.

Are there other setup operations on the subdev besides the usual
v4l2_.._subdev_init(), media_pads_init() etc that have to be called
after a function that by its name 'finalizes' the initialization ?
Anyway, just a nit, whatever is fine

> > > + *
> > > + * Must call v4l2_subdev_cleanup() when the subdev is being removed.
> > > + */
> > > +int v4l2_subdev_init_finalize(struct v4l2_subdev *sd);
> > > +
> > > +/**
> > > + * v4l2_subdev_cleanup() - Release the resources needed by the subdevice
> > > + * @sd: The subdevice
> > > + *
> > > + * This will release the resources allocated in v4l2_subdev_init_finalize.
> > > + */
> > > +void v4l2_subdev_cleanup(struct v4l2_subdev *sd);
> > > +
> > > +/**
> > > + * v4l2_subdev_get_active_state() - Return the active subdev state for subdevice
> > > + * @sd: The subdevice
> > > + *
> > > + * Return the active state for the subdevice, or NULL if the subdev does not
> >
> > nit:
> > The other functions in the file are documented with "Returns" (and
> > also "This function" seems to be more commonly used that just "This").
>
> Ok, I'll try to get these consistent.
>
> > > + * support active state.
> > > + */
> > > +static inline struct v4l2_subdev_state *
> > > +v4l2_subdev_get_active_state(struct v4l2_subdev *sd)
> > > +{
> > > +	return sd->active_state;
> > > +}
> > > +
> >
> > This is only used by the core, and by top drivers when doing:
> >
> >        ret = v4l2_subdev_call(sd, pad, enum_frame_size,
> >                               v4l2_subdev_get_active_state(sd), &fse);
> >
> > To pass to the subdev its own state.
> >
> > I wonder if a macro like
> >
> > #define v4l2_subdev_call_state(sd, o, f, args)					\
> > 	({									\
> > 	 struct v4l2_subdev_state *st = v4l2_subdev_get_active_state(sd);	\
> > 	 v4l2_subdev_call(sd, o, f, st, args);					\
> > 	 })
> >
> > Could help simplify the state API.
>
> Possibly, but we need to support the TRY case also. So I would call it
> something more specific, v4l2_subdev_call_active(), or such.
>

If the caller intends to pass is a try state (which it has to allocate
explicitly) shouldn't it just use v4l2_subdev_call() ?

> > Please be patient if I digress a bit more on the API now..
> >
> > Looking forward in the series we'll end up having 5 functions to
> > manage state, if I've not missed any
> >
> > 1) struct v4l2_subdev_state *v4l2_subdev_get_active_state(struct v4l2_subdev *sd);
> >
> > 2) struct v4l2_subdev_state *v4l2_subdev_lock_active_state(struct v4l2_subdev *sd);
> >
> > 3) void v4l2_subdev_lock_state(struct v4l2_subdev_state *state);
> >
> > 4) void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state);
> >
> > 5) struct v4l2_subdev_state *v4l2_subdev_lock_and_return_state(struct v4l2_subdev *sd,
> > 				                               struct v4l2_subdev_state *state)
> >
> > 1) and 2) are helpers to get (locked/unlocked) v4l2_subdevice.state,
> > used in operations which implicitly operates on the active state
> > (s_stream etc)
> >
> > I wonder if we shouldn't just have a locked v4l2_subdev_get_state()
> > and an optional unlocked version which can be used by the core in
> > place of v4l2_subdev_get_active_state() and by drivers which know what
> > they're doing by using the unlocked version
>
> I think we need to always make it clear if we're dealing with the active or
> try state (when it's relevant). And I also want to be clear if a lock is
> taken. v4l2_subdev_get_state() would hide both of those.

It is clear you're dealing with active state if the only argument is
a struct v4l2_subdevice *.

If you do not receive a state as an operation parameter you can only
get the one is the subdev, which is by definition the active one.

When it comes to locked, if the usage pattern is that the state is to
be locked before being accessed, and the unlocked case is the
exception (as it is only used by the core and by the top drivers) then
I don't feel so strongly as you about having _locked() in the name.

Anyway, your code, your names..

>
> > 3) and 4) lock the state at hand (to be used in operations with a
> > 'state' parameter)
> >
> > 5) is currently unused and it still feels a bit ill-defined. If the state
> > passed in as parameter is NULL then deflect it to sd->state, so I
> > guess it assumes that the user is a state-aware subdev driver which
> > needs to protect against in-kernel callers that do no pass in a valid state to
>
> Correct.
>
> > the operation call. This implies that if the caller passes in a NULL
> > state for the TRY format, this gets deflected to the subdev's active
> > state, something which should not happen, right ? I would be tempted
>
> Yes. That's an illegal call, isn't it? Or does it happen and we need to
> protect against it?
>
> > to just fail loud if !state and assume if you're moving your subdev to
> > use state you should be able to fix the caller as well.
>
> That would be nice, but I think it's not realistic. If you convert a sensor
> driver to multiplexed streams, are you going to fix (convert to multiplexed
> streams) all the bridges and SoC drivers that may use that sensor? How do
> you even find all those bridges and SoCs...

Of course no. You fix the one you're using. You're converting a sensor
driver, you can fix the top driver too. Other users of the sensor
driver will probably scream when moving to the next driver release
that has been made state aware, so they'll hopefully fix their top driver
too. After all, this applies equally to downstrems as well and
providing an helper to work around issues is not the best idea in my
opinion. Also the helper should be used in the subdev driver in place
of the regular v4l2_subdev_lock_state() to protect just in case
against legacy callers. When will they be moved to use the regular
v4l2_subdev_lock_state() ?

Once a subdev driver has been moved to be state aware callers that
passes in a NULL state should be fixed. As we can't fix them all,
screaming loud and clearly say what has to be done to move forward is
in my opinion better than introducing a temporary function that
replaces the regular API and that subdev should decide to use just in
case (and which can lead to subtle errors like using the active state
in a TRY context).

If you want to protect against broken callers then what about
doing the "state = state ? : sd->active_state;" dance in
v4l2_subdev_lock_state() with a WARN_ONCE(!state) so that
subdev drivers can use the regular API from day 0 ?

>
>  Tomi

Re: [PATCH v10 02/38] media: subdev: add active state to struct v4l2_subdev

[Tomi Valkeinen]

On 15/12/2021 11:38, Jacopo Mondi wrote:
> Hi Tomi
> 
> On Wed, Dec 15, 2021 at 10:06:16AM +0200, Tomi Valkeinen wrote:
>> Hi,
>>
>> On 13/12/2021 23:30, Jacopo Mondi wrote:
>>> Hi Tomi,
>>>
>>> On Tue, Nov 30, 2021 at 04:15:00PM +0200, Tomi Valkeinen wrote:
>>>> Add a new 'active_state' field to struct v4l2_subdev to which we can
>>>> store the active state of a subdev. This will place the subdev
>>>> configuration into a known place, allowing us to use the state directly
>>>> from the v4l2 framework, thus simplifying the drivers.
>>>>
>>>> Also add functions v4l2_subdev_init_finalize() and
>>>> v4l2_subdev_cleanup(), which will allocate and free the active state.
>>>> The functions are named in a generic way so that they can be also used
>>>> for other subdev initialization work.
>>>>
>>>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>>>> ---
>>>>    drivers/media/v4l2-core/v4l2-subdev.c | 21 +++++++++++
>>>>    include/media/v4l2-subdev.h           | 54 +++++++++++++++++++++++++++
>>>>    2 files changed, 75 insertions(+)
>>>>
>>>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>>>> index fe49c86a9b02..de160140d63b 100644
>>>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>>>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>>>> @@ -943,3 +943,24 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
>>>>    	v4l2_subdev_notify(sd, V4L2_DEVICE_NOTIFY_EVENT, (void *)ev);
>>>>    }
>>>>    EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
>>>> +
>>>> +int v4l2_subdev_init_finalize(struct v4l2_subdev *sd)
>>>> +{
>>>> +	struct v4l2_subdev_state *state;
>>>> +
>>>> +	state = __v4l2_subdev_state_alloc(sd);
>>>> +	if (IS_ERR(state))
>>>> +		return PTR_ERR(state);
>>>> +
>>>> +	sd->active_state = state;
>>>
>>> You could spare the local var..
>>
>> I don't like to write error values to storage fields. Here I only touch the
>> field when succesfull.
>>
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +EXPORT_SYMBOL_GPL(v4l2_subdev_init_finalize);
>>>> +
>>>> +void v4l2_subdev_cleanup(struct v4l2_subdev *sd)
>>>> +{
>>>> +	__v4l2_subdev_state_free(sd->active_state);
>>>> +	sd->active_state = NULL;
>>>> +}
>>>> +EXPORT_SYMBOL_GPL(v4l2_subdev_cleanup);
>>>> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
>>>> index e52bf508c75b..87c3126b2754 100644
>>>> --- a/include/media/v4l2-subdev.h
>>>> +++ b/include/media/v4l2-subdev.h
>>>> @@ -645,6 +645,9 @@ struct v4l2_subdev_ir_ops {
>>>>     * This structure only needs to be passed to the pad op if the 'which' field
>>>>     * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
>>>>     * %V4L2_SUBDEV_FORMAT_ACTIVE it is safe to pass %NULL.
>>>> + *
>>>> + * Note: This struct is also used in active state, and the try_ prefix is
>>>> + * historical and to be removed.
>>>>     */
>>>>    struct v4l2_subdev_pad_config {
>>>>    	struct v4l2_mbus_framefmt try_fmt;
>>>> @@ -898,6 +901,8 @@ struct v4l2_subdev_platform_data {
>>>>     * @subdev_notifier: A sub-device notifier implicitly registered for the sub-
>>>>     *		     device using v4l2_async_register_subdev_sensor().
>>>>     * @pdata: common part of subdevice platform data
>>>> + * @active_state: active state for the subdev (NULL for subdevs tracking the
>>>> + *                state internally)
>>>
>>> I would suggest here that init_finalize() should be used
>>>
>>>    * @active_state: subdev active state. Initialized by calling
>>>                     v4l2_subdev_init_finalize() by drivers that support
>>>                     state
>>
>> Ok.
>>
>>> I would have said "the state API" but then we would need to define
>>> what the state API as a concept :)
>>>
>>>>     *
>>>>     * Each instance of a subdev driver should create this struct, either
>>>>     * stand-alone or embedded in a larger struct.
>>>> @@ -929,6 +934,19 @@ struct v4l2_subdev {
>>>>    	struct v4l2_async_notifier *notifier;
>>>>    	struct v4l2_async_notifier *subdev_notifier;
>>>>    	struct v4l2_subdev_platform_data *pdata;
>>>> +
>>>> +	/*
>>>> +	 * The fields below are private, and should only be accessed via
>>>> +	 * appropriate functions.
>>>> +	 */
>>>> +
>>>> +	/*
>>>> +	 * TODO: state should most likely be changed from a pointer to an
>>>> +	 * embedded field. For the time being it's kept as a pointer to more
>>>> +	 * easily catch uses of state in the cases where the driver doesn't
>>>> +	 * support it.
>>>
>>> what about "... For the time being it's kept as a pointer to allow drivers
>>> to support state as an opt-in feature."
>>
>> Well, that's not quite correct. Even if it was an embedded field, it would
>> be an opt-in feature. The point with the pointer is only that if an opt-out
>> driver accidentally tries to access the field, it'll get NULL, and hopefully
>> fail.
> 
> Ok, what I mean is that being a pointer, it's easy to identify if a
> driver has opted-in. Whatever, we're saying the same thing
> 
>>
>>>> +	 */
>>>> +	struct v4l2_subdev_state *active_state;
>>>>    };
>>>>
>>>>
>>>> @@ -1217,4 +1235,40 @@ extern const struct v4l2_subdev_ops v4l2_subdev_call_wrappers;
>>>>    void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
>>>>    			      const struct v4l2_event *ev);
>>>>
>>>> +/**
>>>> + * v4l2_subdev_init_finalize() - Finalize the initialization of the subdevice
>>>> + * @sd: The subdev
>>>> + *
>>>> + * This finalizes the initialization of the subdev, including allocation of
>>>> + * the active state for the subdev.
>>>> + *
>>>> + * This must be called by the subdev drivers that use the centralized active
>>>> + * state, after the subdev struct has been initialized and
>>>> + * media_entity_pads_init() has been called.
>>>
>>> Or just "It must be called by subdev drivers that support active state
>>> as the last operation before registering the subdev."
>>
>> That's not quite true either. The driver may do setup operations requiring
>> the active state before registering the subdev.
> 
> Are there other setup operations on the subdev besides the usual
> v4l2_.._subdev_init(), media_pads_init() etc that have to be called
> after a function that by its name 'finalizes' the initialization ?
> Anyway, just a nit, whatever is fine

I was only thinking about the finalization of the state. But as it's now 
"init finalize", I think it makes sense that it would be just before 
subdev registration. But I'm actually setting up the notifiers between 
the v4l2_subdev_init_finalize and v4l2_async_register_subdev... I need 
to see if there's any reason for that, or can the 
v4l2_subdev_init_finalize be moved to be just before subdev registration.

>>>> + *
>>>> + * Must call v4l2_subdev_cleanup() when the subdev is being removed.
>>>> + */
>>>> +int v4l2_subdev_init_finalize(struct v4l2_subdev *sd);
>>>> +
>>>> +/**
>>>> + * v4l2_subdev_cleanup() - Release the resources needed by the subdevice
>>>> + * @sd: The subdevice
>>>> + *
>>>> + * This will release the resources allocated in v4l2_subdev_init_finalize.
>>>> + */
>>>> +void v4l2_subdev_cleanup(struct v4l2_subdev *sd);
>>>> +
>>>> +/**
>>>> + * v4l2_subdev_get_active_state() - Return the active subdev state for subdevice
>>>> + * @sd: The subdevice
>>>> + *
>>>> + * Return the active state for the subdevice, or NULL if the subdev does not
>>>
>>> nit:
>>> The other functions in the file are documented with "Returns" (and
>>> also "This function" seems to be more commonly used that just "This").
>>
>> Ok, I'll try to get these consistent.
>>
>>>> + * support active state.
>>>> + */
>>>> +static inline struct v4l2_subdev_state *
>>>> +v4l2_subdev_get_active_state(struct v4l2_subdev *sd)
>>>> +{
>>>> +	return sd->active_state;
>>>> +}
>>>> +
>>>
>>> This is only used by the core, and by top drivers when doing:
>>>
>>>         ret = v4l2_subdev_call(sd, pad, enum_frame_size,
>>>                                v4l2_subdev_get_active_state(sd), &fse);
>>>
>>> To pass to the subdev its own state.
>>>
>>> I wonder if a macro like
>>>
>>> #define v4l2_subdev_call_state(sd, o, f, args)					\
>>> 	({									\
>>> 	 struct v4l2_subdev_state *st = v4l2_subdev_get_active_state(sd);	\
>>> 	 v4l2_subdev_call(sd, o, f, st, args);					\
>>> 	 })
>>>
>>> Could help simplify the state API.
>>
>> Possibly, but we need to support the TRY case also. So I would call it
>> something more specific, v4l2_subdev_call_active(), or such.
>>
> 
> If the caller intends to pass is a try state (which it has to allocate
> explicitly) shouldn't it just use v4l2_subdev_call() ?

Yes. But I think this is pretty confusing already, and in my opinion 
adding v4l2_subdev_call_state() would confuse this even more =). I think 
at least it has to make it clear that the subdev is being called using 
its active state.

But maybe just using v4l2_subdev_call() for now and seeing how this 
evolves is best. It's trivial to add the helper later.

>>> Please be patient if I digress a bit more on the API now..
>>>
>>> Looking forward in the series we'll end up having 5 functions to
>>> manage state, if I've not missed any
>>>
>>> 1) struct v4l2_subdev_state *v4l2_subdev_get_active_state(struct v4l2_subdev *sd);
>>>
>>> 2) struct v4l2_subdev_state *v4l2_subdev_lock_active_state(struct v4l2_subdev *sd);
>>>
>>> 3) void v4l2_subdev_lock_state(struct v4l2_subdev_state *state);
>>>
>>> 4) void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state);
>>>
>>> 5) struct v4l2_subdev_state *v4l2_subdev_lock_and_return_state(struct v4l2_subdev *sd,
>>> 				                               struct v4l2_subdev_state *state)
>>>
>>> 1) and 2) are helpers to get (locked/unlocked) v4l2_subdevice.state,
>>> used in operations which implicitly operates on the active state
>>> (s_stream etc)
>>>
>>> I wonder if we shouldn't just have a locked v4l2_subdev_get_state()
>>> and an optional unlocked version which can be used by the core in
>>> place of v4l2_subdev_get_active_state() and by drivers which know what
>>> they're doing by using the unlocked version
>>
>> I think we need to always make it clear if we're dealing with the active or
>> try state (when it's relevant). And I also want to be clear if a lock is
>> taken. v4l2_subdev_get_state() would hide both of those.
> 
> It is clear you're dealing with active state if the only argument is
> a struct v4l2_subdevice *.

Perhaps, if you understand the internals here. I'd rather be more 
explicit and clear in the function names. Again, we can easily change 
these later if they don't feel necessary.

> If you do not receive a state as an operation parameter you can only
> get the one is the subdev, which is by definition the active one.
> 
> When it comes to locked, if the usage pattern is that the state is to
> be locked before being accessed, and the unlocked case is the
> exception (as it is only used by the core and by the top drivers) then
> I don't feel so strongly as you about having _locked() in the name.

My reasoning here is that we can have function pairs like get_xxx, 
put_xxx, and then it's clear that those match and they need to be paired.

But if we have just a get_xxx, at least my thinking is that it just 
returns something, and I'll use it and forget about it when I don't need 
it anymore.

But lock_xxx makes it clear I have to unlock it afterwards.

That said, maybe instead of v4l2_subdev_lock_active_state we should have 
v4l2_subdev_lock_and_get_active_state. But it gets a bit long...

> Anyway, your code, your names..
> 
>>
>>> 3) and 4) lock the state at hand (to be used in operations with a
>>> 'state' parameter)
>>>
>>> 5) is currently unused and it still feels a bit ill-defined. If the state
>>> passed in as parameter is NULL then deflect it to sd->state, so I
>>> guess it assumes that the user is a state-aware subdev driver which
>>> needs to protect against in-kernel callers that do no pass in a valid state to
>>
>> Correct.
>>
>>> the operation call. This implies that if the caller passes in a NULL
>>> state for the TRY format, this gets deflected to the subdev's active
>>> state, something which should not happen, right ? I would be tempted
>>
>> Yes. That's an illegal call, isn't it? Or does it happen and we need to
>> protect against it?
>>
>>> to just fail loud if !state and assume if you're moving your subdev to
>>> use state you should be able to fix the caller as well.
>>
>> That would be nice, but I think it's not realistic. If you convert a sensor
>> driver to multiplexed streams, are you going to fix (convert to multiplexed
>> streams) all the bridges and SoC drivers that may use that sensor? How do
>> you even find all those bridges and SoCs...
> 
> Of course no. You fix the one you're using. You're converting a sensor
> driver, you can fix the top driver too. Other users of the sensor
> driver will probably scream when moving to the next driver release
> that has been made state aware, so they'll hopefully fix their top driver
> too. After all, this applies equally to downstrems as well and

Well, I'm not a maintainer in linux-media, but I would nack that 
approach =). We can't just go breaking other platforms, and hoping other 
devs will fix them.

> providing an helper to work around issues is not the best idea in my
> opinion. Also the helper should be used in the subdev driver in place
> of the regular v4l2_subdev_lock_state() to protect just in case
> against legacy callers. When will they be moved to use the regular
> v4l2_subdev_lock_state() ?

Note that this is needed only when porting an existing and presumably 
in-use subdev driver. You don't need this if you write a new driver.

The users of v4l2_subdev_lock_and_return_state are easy to find and easy 
to change to v4l2_subdev_lock_state when we decide a particular driver 
doesn't need to support legacy subdevs.

I don't like this at all but, afaics, we can't break the existing 
platforms. This function is a very simple work-around for the time 
being, and easy to drop later.

> Once a subdev driver has been moved to be state aware callers that
> passes in a NULL state should be fixed. As we can't fix them all,
> screaming loud and clearly say what has to be done to move forward is
> in my opinion better than introducing a temporary function that
> replaces the regular API and that subdev should decide to use just in
> case (and which can lead to subtle errors like using the active state
> in a TRY context).
> 
> If you want to protect against broken callers then what about
> doing the "state = state ? : sd->active_state;" dance in
> v4l2_subdev_lock_state() with a WARN_ONCE(!state) so that
> subdev drivers can use the regular API from day 0 ?

Hmm, I think that is an option. I didn't implement "state = state ? : 
sd->active_state;" in the v4l2_subdev_lock_state() as I didn't want 
v4l2_subdev_lock_state() to hide an invalid NULL parameter. But adding 
WARN_ONCE() would warn about it.

I'm still undecided, though. The WARN_ONCE would come only once for the 
whole kernel, per boot, wouldn't it? We could have a macro for 
v4l2_subdev_lock_state, but then we'd get lots of warnings. And a full 
WARN just because a driver hasn't been updated is a bit harsh. Maybe we 
can start with just a normal warning print.

  Tomi

Re: [PATCH v10 02/38] media: subdev: add active state to struct v4l2_subdev

[Jacopo Mondi]

Hi Tomi

On Wed, Dec 15, 2021 at 05:35:53PM +0200, Tomi Valkeinen wrote:
> On 15/12/2021 11:38, Jacopo Mondi wrote:
> > Hi Tomi
> >
> > On Wed, Dec 15, 2021 at 10:06:16AM +0200, Tomi Valkeinen wrote:
> > > Hi,
> > >
> > > On 13/12/2021 23:30, Jacopo Mondi wrote:
> > > > Hi Tomi,
> > > >
> > > > On Tue, Nov 30, 2021 at 04:15:00PM +0200, Tomi Valkeinen wrote:
> > > > > Add a new 'active_state' field to struct v4l2_subdev to which we can
> > > > > store the active state of a subdev. This will place the subdev
> > > > > configuration into a known place, allowing us to use the state directly
> > > > > from the v4l2 framework, thus simplifying the drivers.
> > > > >
> > > > > Also add functions v4l2_subdev_init_finalize() and
> > > > > v4l2_subdev_cleanup(), which will allocate and free the active state.
> > > > > The functions are named in a generic way so that they can be also used
> > > > > for other subdev initialization work.
> > > > >
> > > > > Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > > > > ---
> > > > >    drivers/media/v4l2-core/v4l2-subdev.c | 21 +++++++++++
> > > > >    include/media/v4l2-subdev.h           | 54 +++++++++++++++++++++++++++
> > > > >    2 files changed, 75 insertions(+)
> > > > >
> > > > > diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> > > > > index fe49c86a9b02..de160140d63b 100644
> > > > > --- a/drivers/media/v4l2-core/v4l2-subdev.c
> > > > > +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> > > > > @@ -943,3 +943,24 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
> > > > >    	v4l2_subdev_notify(sd, V4L2_DEVICE_NOTIFY_EVENT, (void *)ev);
> > > > >    }
> > > > >    EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
> > > > > +
> > > > > +int v4l2_subdev_init_finalize(struct v4l2_subdev *sd)
> > > > > +{
> > > > > +	struct v4l2_subdev_state *state;
> > > > > +
> > > > > +	state = __v4l2_subdev_state_alloc(sd);
> > > > > +	if (IS_ERR(state))
> > > > > +		return PTR_ERR(state);
> > > > > +
> > > > > +	sd->active_state = state;
> > > >
> > > > You could spare the local var..
> > >
> > > I don't like to write error values to storage fields. Here I only touch the
> > > field when succesfull.
> > >
> > > > > +
> > > > > +	return 0;
> > > > > +}
> > > > > +EXPORT_SYMBOL_GPL(v4l2_subdev_init_finalize);
> > > > > +
> > > > > +void v4l2_subdev_cleanup(struct v4l2_subdev *sd)
> > > > > +{
> > > > > +	__v4l2_subdev_state_free(sd->active_state);
> > > > > +	sd->active_state = NULL;
> > > > > +}
> > > > > +EXPORT_SYMBOL_GPL(v4l2_subdev_cleanup);
> > > > > diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> > > > > index e52bf508c75b..87c3126b2754 100644
> > > > > --- a/include/media/v4l2-subdev.h
> > > > > +++ b/include/media/v4l2-subdev.h
> > > > > @@ -645,6 +645,9 @@ struct v4l2_subdev_ir_ops {
> > > > >     * This structure only needs to be passed to the pad op if the 'which' field
> > > > >     * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
> > > > >     * %V4L2_SUBDEV_FORMAT_ACTIVE it is safe to pass %NULL.
> > > > > + *
> > > > > + * Note: This struct is also used in active state, and the try_ prefix is
> > > > > + * historical and to be removed.
> > > > >     */
> > > > >    struct v4l2_subdev_pad_config {
> > > > >    	struct v4l2_mbus_framefmt try_fmt;
> > > > > @@ -898,6 +901,8 @@ struct v4l2_subdev_platform_data {
> > > > >     * @subdev_notifier: A sub-device notifier implicitly registered for the sub-
> > > > >     *		     device using v4l2_async_register_subdev_sensor().
> > > > >     * @pdata: common part of subdevice platform data
> > > > > + * @active_state: active state for the subdev (NULL for subdevs tracking the
> > > > > + *                state internally)
> > > >
> > > > I would suggest here that init_finalize() should be used
> > > >
> > > >    * @active_state: subdev active state. Initialized by calling
> > > >                     v4l2_subdev_init_finalize() by drivers that support
> > > >                     state
> > >
> > > Ok.
> > >
> > > > I would have said "the state API" but then we would need to define
> > > > what the state API as a concept :)
> > > >
> > > > >     *
> > > > >     * Each instance of a subdev driver should create this struct, either
> > > > >     * stand-alone or embedded in a larger struct.
> > > > > @@ -929,6 +934,19 @@ struct v4l2_subdev {
> > > > >    	struct v4l2_async_notifier *notifier;
> > > > >    	struct v4l2_async_notifier *subdev_notifier;
> > > > >    	struct v4l2_subdev_platform_data *pdata;
> > > > > +
> > > > > +	/*
> > > > > +	 * The fields below are private, and should only be accessed via
> > > > > +	 * appropriate functions.
> > > > > +	 */
> > > > > +
> > > > > +	/*
> > > > > +	 * TODO: state should most likely be changed from a pointer to an
> > > > > +	 * embedded field. For the time being it's kept as a pointer to more
> > > > > +	 * easily catch uses of state in the cases where the driver doesn't
> > > > > +	 * support it.
> > > >
> > > > what about "... For the time being it's kept as a pointer to allow drivers
> > > > to support state as an opt-in feature."
> > >
> > > Well, that's not quite correct. Even if it was an embedded field, it would
> > > be an opt-in feature. The point with the pointer is only that if an opt-out
> > > driver accidentally tries to access the field, it'll get NULL, and hopefully
> > > fail.
> >
> > Ok, what I mean is that being a pointer, it's easy to identify if a
> > driver has opted-in. Whatever, we're saying the same thing
> >
> > >
> > > > > +	 */
> > > > > +	struct v4l2_subdev_state *active_state;
> > > > >    };
> > > > >
> > > > >
> > > > > @@ -1217,4 +1235,40 @@ extern const struct v4l2_subdev_ops v4l2_subdev_call_wrappers;
> > > > >    void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
> > > > >    			      const struct v4l2_event *ev);
> > > > >
> > > > > +/**
> > > > > + * v4l2_subdev_init_finalize() - Finalize the initialization of the subdevice
> > > > > + * @sd: The subdev
> > > > > + *
> > > > > + * This finalizes the initialization of the subdev, including allocation of
> > > > > + * the active state for the subdev.
> > > > > + *
> > > > > + * This must be called by the subdev drivers that use the centralized active
> > > > > + * state, after the subdev struct has been initialized and
> > > > > + * media_entity_pads_init() has been called.
> > > >
> > > > Or just "It must be called by subdev drivers that support active state
> > > > as the last operation before registering the subdev."
> > >
> > > That's not quite true either. The driver may do setup operations requiring
> > > the active state before registering the subdev.
> >
> > Are there other setup operations on the subdev besides the usual
> > v4l2_.._subdev_init(), media_pads_init() etc that have to be called
> > after a function that by its name 'finalizes' the initialization ?
> > Anyway, just a nit, whatever is fine
>
> I was only thinking about the finalization of the state. But as it's now
> "init finalize", I think it makes sense that it would be just before subdev
> registration. But I'm actually setting up the notifiers between the
> v4l2_subdev_init_finalize and v4l2_async_register_subdev... I need to see if
> there's any reason for that, or can the v4l2_subdev_init_finalize be moved
> to be just before subdev registration.
>

I see, of course there might be other operations going on between
init_finalize() and subdev_register(), what I meant was to only
discuss about helper function provided by the core, not driver
specific functions. Intrpreted this way I understand why my suggestion
is confusing, feel free to ignore it!

> > > > > + *
> > > > > + * Must call v4l2_subdev_cleanup() when the subdev is being removed.
> > > > > + */
> > > > > +int v4l2_subdev_init_finalize(struct v4l2_subdev *sd);
> > > > > +
> > > > > +/**
> > > > > + * v4l2_subdev_cleanup() - Release the resources needed by the subdevice
> > > > > + * @sd: The subdevice
> > > > > + *
> > > > > + * This will release the resources allocated in v4l2_subdev_init_finalize.
> > > > > + */
> > > > > +void v4l2_subdev_cleanup(struct v4l2_subdev *sd);
> > > > > +
> > > > > +/**
> > > > > + * v4l2_subdev_get_active_state() - Return the active subdev state for subdevice
> > > > > + * @sd: The subdevice
> > > > > + *
> > > > > + * Return the active state for the subdevice, or NULL if the subdev does not
> > > >
> > > > nit:
> > > > The other functions in the file are documented with "Returns" (and
> > > > also "This function" seems to be more commonly used that just "This").
> > >
> > > Ok, I'll try to get these consistent.
> > >
> > > > > + * support active state.
> > > > > + */
> > > > > +static inline struct v4l2_subdev_state *
> > > > > +v4l2_subdev_get_active_state(struct v4l2_subdev *sd)
> > > > > +{
> > > > > +	return sd->active_state;
> > > > > +}
> > > > > +
> > > >
> > > > This is only used by the core, and by top drivers when doing:
> > > >
> > > >         ret = v4l2_subdev_call(sd, pad, enum_frame_size,
> > > >                                v4l2_subdev_get_active_state(sd), &fse);
> > > >
> > > > To pass to the subdev its own state.
> > > >
> > > > I wonder if a macro like
> > > >
> > > > #define v4l2_subdev_call_state(sd, o, f, args)					\
> > > > 	({									\
> > > > 	 struct v4l2_subdev_state *st = v4l2_subdev_get_active_state(sd);	\
> > > > 	 v4l2_subdev_call(sd, o, f, st, args);					\
> > > > 	 })
> > > >
> > > > Could help simplify the state API.
> > >
> > > Possibly, but we need to support the TRY case also. So I would call it
> > > something more specific, v4l2_subdev_call_active(), or such.
> > >
> >
> > If the caller intends to pass is a try state (which it has to allocate
> > explicitly) shouldn't it just use v4l2_subdev_call() ?
>
> Yes. But I think this is pretty confusing already, and in my opinion adding
> v4l2_subdev_call_state() would confuse this even more =). I think at least
> it has to make it clear that the subdev is being called using its active
> state.
>
> But maybe just using v4l2_subdev_call() for now and seeing how this evolves
> is best. It's trivial to add the helper later.
>

Make sense

> > > > Please be patient if I digress a bit more on the API now..
> > > >
> > > > Looking forward in the series we'll end up having 5 functions to
> > > > manage state, if I've not missed any
> > > >
> > > > 1) struct v4l2_subdev_state *v4l2_subdev_get_active_state(struct v4l2_subdev *sd);
> > > >
> > > > 2) struct v4l2_subdev_state *v4l2_subdev_lock_active_state(struct v4l2_subdev *sd);
> > > >
> > > > 3) void v4l2_subdev_lock_state(struct v4l2_subdev_state *state);
> > > >
> > > > 4) void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state);
> > > >
> > > > 5) struct v4l2_subdev_state *v4l2_subdev_lock_and_return_state(struct v4l2_subdev *sd,
> > > > 				                               struct v4l2_subdev_state *state)
> > > >
> > > > 1) and 2) are helpers to get (locked/unlocked) v4l2_subdevice.state,
> > > > used in operations which implicitly operates on the active state
> > > > (s_stream etc)
> > > >
> > > > I wonder if we shouldn't just have a locked v4l2_subdev_get_state()
> > > > and an optional unlocked version which can be used by the core in
> > > > place of v4l2_subdev_get_active_state() and by drivers which know what
> > > > they're doing by using the unlocked version
> > >
> > > I think we need to always make it clear if we're dealing with the active or
> > > try state (when it's relevant). And I also want to be clear if a lock is
> > > taken. v4l2_subdev_get_state() would hide both of those.
> >
> > It is clear you're dealing with active state if the only argument is
> > a struct v4l2_subdevice *.
>
> Perhaps, if you understand the internals here. I'd rather be more explicit
> and clear in the function names. Again, we can easily change these later if
> they don't feel necessary.
>
> > If you do not receive a state as an operation parameter you can only
> > get the one is the subdev, which is by definition the active one.
> >
> > When it comes to locked, if the usage pattern is that the state is to
> > be locked before being accessed, and the unlocked case is the
> > exception (as it is only used by the core and by the top drivers) then
> > I don't feel so strongly as you about having _locked() in the name.
>
> My reasoning here is that we can have function pairs like get_xxx, put_xxx,
> and then it's clear that those match and they need to be paired.

Like:

v4l2_subdev_lock_state(state)
v4l2_subdev_unlock_state(state)

>
> But if we have just a get_xxx, at least my thinking is that it just returns
> something, and I'll use it and forget about it when I don't need it anymore.
>
> But lock_xxx makes it clear I have to unlock it afterwards.
>

so it gets

state = v4l2_subdev_lock_active_state(sd)
v4l2_subdev_unlock_state(state)

> That said, maybe instead of v4l2_subdev_lock_active_state we should have
> v4l2_subdev_lock_and_get_active_state. But it gets a bit long...

No no please :)

You bought me with:
v4l2_subdev_lock_active_state() + v4l2_subdev_unlock_state()

>
> > Anyway, your code, your names..
> >
> > >
> > > > 3) and 4) lock the state at hand (to be used in operations with a
> > > > 'state' parameter)
> > > >
> > > > 5) is currently unused and it still feels a bit ill-defined. If the state
> > > > passed in as parameter is NULL then deflect it to sd->state, so I
> > > > guess it assumes that the user is a state-aware subdev driver which
> > > > needs to protect against in-kernel callers that do no pass in a valid state to
> > >
> > > Correct.
> > >
> > > > the operation call. This implies that if the caller passes in a NULL
> > > > state for the TRY format, this gets deflected to the subdev's active
> > > > state, something which should not happen, right ? I would be tempted
> > >
> > > Yes. That's an illegal call, isn't it? Or does it happen and we need to
> > > protect against it?
> > >
> > > > to just fail loud if !state and assume if you're moving your subdev to
> > > > use state you should be able to fix the caller as well.
> > >
> > > That would be nice, but I think it's not realistic. If you convert a sensor
> > > driver to multiplexed streams, are you going to fix (convert to multiplexed
> > > streams) all the bridges and SoC drivers that may use that sensor? How do
> > > you even find all those bridges and SoCs...
> >
> > Of course no. You fix the one you're using. You're converting a sensor
> > driver, you can fix the top driver too. Other users of the sensor
> > driver will probably scream when moving to the next driver release
> > that has been made state aware, so they'll hopefully fix their top driver
> > too. After all, this applies equally to downstrems as well and
>
> Well, I'm not a maintainer in linux-media, but I would nack that approach
> =). We can't just go breaking other platforms, and hoping other devs will
> fix them.
>
> > providing an helper to work around issues is not the best idea in my
> > opinion. Also the helper should be used in the subdev driver in place
> > of the regular v4l2_subdev_lock_state() to protect just in case
> > against legacy callers. When will they be moved to use the regular
> > v4l2_subdev_lock_state() ?
>
> Note that this is needed only when porting an existing and presumably in-use
> subdev driver. You don't need this if you write a new driver.
>
> The users of v4l2_subdev_lock_and_return_state are easy to find and easy to
> change to v4l2_subdev_lock_state when we decide a particular driver doesn't
> need to support legacy subdevs.

I'm just concerned that as long as we offer an helper to work this
around (and the helper might introduce subtle issues like mixing
try/active context) once we
s/v4l2_subdev_lock_and_return_state/v4l2_subdev_lock_state we'll be
anyway breaking users.

>
> I don't like this at all but, afaics, we can't break the existing platforms.
> This function is a very simple work-around for the time being, and easy to
> drop later.
>
> > Once a subdev driver has been moved to be state aware callers that
> > passes in a NULL state should be fixed. As we can't fix them all,
> > screaming loud and clearly say what has to be done to move forward is
> > in my opinion better than introducing a temporary function that
> > replaces the regular API and that subdev should decide to use just in
> > case (and which can lead to subtle errors like using the active state
> > in a TRY context).
> >
> > If you want to protect against broken callers then what about
> > doing the "state = state ? : sd->active_state;" dance in
> > v4l2_subdev_lock_state() with a WARN_ONCE(!state) so that
> > subdev drivers can use the regular API from day 0 ?
>
> Hmm, I think that is an option. I didn't implement "state = state ? :
> sd->active_state;" in the v4l2_subdev_lock_state() as I didn't want
> v4l2_subdev_lock_state() to hide an invalid NULL parameter. But adding
> WARN_ONCE() would warn about it.
>
> I'm still undecided, though. The WARN_ONCE would come only once for the
> whole kernel, per boot, wouldn't it? We could have a macro for
> v4l2_subdev_lock_state, but then we'd get lots of warnings. And a full WARN
> just because a driver hasn't been updated is a bit harsh. Maybe we can start
> with just a normal warning print.

There is a precendet I can think of: async matching on device or
endpoints. Initially all async subdevs where matched on the remote end
device node. For more complex setups this didn't scale, as the same
remote can have multiple endpoints, and matching on the device node
would have created false positives. So v4l2-async was moved to match on
endpoints with some legacy compatibility code

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/media/v4l2-core/v4l2-async.c#n69

	if (dev && dev->driver) {
		if (sd_fwnode_is_ep)
			dev_warn(dev, "Driver %s uses device fwnode, incorrect match may occur\n",
				 dev->driver->name);
		dev_notice(dev, "Consider updating driver %s to match on endpoints\n",
			   dev->driver->name);
	}

Can't we have something like this in v4l2_subdev_lock_state() ?

Thanks
   j

>
>  Tomi

Re: [PATCH v10 04/38] media: subdev: add subdev state locking

[Laurent Pinchart]

Hi Tomi,

Thank you for the patch.

On Tue, Nov 30, 2021 at 04:15:02PM +0200, Tomi Valkeinen wrote:
> The V4L2 subdevs have managed without centralized locking for the state
> (previously pad_config), as the TRY state is supposedly safe (although I
> believe two TRY ioctls for the same fd would race), and the ACTIVE
> state, and its locking, is managed by the drivers internally.
> 
> We now have ACTIVE state in a centralized position, and need locking.
> Strictly speaking the locking is only needed for new drivers that use
> the new state, as the current drivers continue behaving as they used to.
> 
> Add a mutex to the struct v4l2_subdev_state, along with a few helper
> functions for locking/unlocking.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/platform/rcar-vin/rcar-v4l2.c |  3 +-
>  drivers/media/platform/vsp1/vsp1_entity.c   |  4 +-
>  drivers/media/v4l2-core/v4l2-subdev.c       | 38 +++++++++++++---
>  drivers/staging/media/tegra-video/vi.c      |  4 +-
>  include/media/v4l2-subdev.h                 | 50 ++++++++++++++++++++-
>  5 files changed, 89 insertions(+), 10 deletions(-)
> 
> diff --git a/drivers/media/platform/rcar-vin/rcar-v4l2.c b/drivers/media/platform/rcar-vin/rcar-v4l2.c
> index ba1d16ab1651..e6bd94d63e4f 100644
> --- a/drivers/media/platform/rcar-vin/rcar-v4l2.c
> +++ b/drivers/media/platform/rcar-vin/rcar-v4l2.c
> @@ -244,6 +244,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
>  {
>  	struct v4l2_subdev *sd = vin_to_source(vin);
>  	struct v4l2_subdev_state *sd_state;
> +	static struct lock_class_key key;
>  	struct v4l2_subdev_format format = {
>  		.which = which,
>  		.pad = vin->parallel.source_pad,
> @@ -252,7 +253,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
>  	u32 width, height;
>  	int ret;
>  
> -	sd_state = __v4l2_subdev_state_alloc(sd);

I would add here a

	/*
	 * FIXME: Drop this call, drivers are not supposed to use
	 * __v4l2_subdev_state_alloc().
	 */

Same below.

> +	sd_state = __v4l2_subdev_state_alloc(sd, "rvin:state->lock", &key);
>  	if (IS_ERR(sd_state))
>  		return PTR_ERR(sd_state);
>  
> diff --git a/drivers/media/platform/vsp1/vsp1_entity.c b/drivers/media/platform/vsp1/vsp1_entity.c
> index 869cadc1468d..e607c3ae2520 100644
> --- a/drivers/media/platform/vsp1/vsp1_entity.c
> +++ b/drivers/media/platform/vsp1/vsp1_entity.c
> @@ -613,6 +613,7 @@ int vsp1_entity_init(struct vsp1_device *vsp1, struct vsp1_entity *entity,
>  		     const char *name, unsigned int num_pads,
>  		     const struct v4l2_subdev_ops *ops, u32 function)
>  {
> +	static struct lock_class_key key;
>  	struct v4l2_subdev *subdev;
>  	unsigned int i;
>  	int ret;
> @@ -675,7 +676,8 @@ int vsp1_entity_init(struct vsp1_device *vsp1, struct vsp1_entity *entity,
>  	 * Allocate the pad configuration to store formats and selection
>  	 * rectangles.
>  	 */
> -	entity->config = __v4l2_subdev_state_alloc(&entity->subdev);
> +	entity->config = __v4l2_subdev_state_alloc(&entity->subdev,
> +						   "vsp1:config->lock", &key);
>  	if (IS_ERR(entity->config)) {
>  		media_entity_cleanup(&entity->subdev.entity);
>  		return PTR_ERR(entity->config);
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 3289875d9ec1..2053fe1cd67d 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -27,8 +27,9 @@
>  static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
>  {
>  	struct v4l2_subdev_state *state;
> +	static struct lock_class_key key;
>  
> -	state = __v4l2_subdev_state_alloc(sd);
> +	state = __v4l2_subdev_state_alloc(sd, "fh->state->lock", &key);
>  	if (IS_ERR(state))
>  		return PTR_ERR(state);
>  
> @@ -923,7 +924,9 @@ int v4l2_subdev_link_validate(struct media_link *link)
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
>  
> -struct v4l2_subdev_state *__v4l2_subdev_state_alloc(struct v4l2_subdev *sd)
> +struct v4l2_subdev_state *
> +__v4l2_subdev_state_alloc(struct v4l2_subdev *sd, const char *lock_name,
> +			  struct lock_class_key *lock_key)
>  {
>  	struct v4l2_subdev_state *state;
>  	int ret;
> @@ -932,6 +935,8 @@ struct v4l2_subdev_state *__v4l2_subdev_state_alloc(struct v4l2_subdev *sd)
>  	if (!state)
>  		return ERR_PTR(-ENOMEM);
>  
> +	__mutex_init(&state->lock, lock_name, lock_key);
> +
>  	if (sd->entity.num_pads) {
>  		state->pads = kvmalloc_array(sd->entity.num_pads,
>  					     sizeof(*state->pads),
> @@ -963,6 +968,8 @@ void __v4l2_subdev_state_free(struct v4l2_subdev_state *state)
>  	if (!state)
>  		return;
>  
> +	mutex_destroy(&state->lock);
> +
>  	kvfree(state->pads);
>  	kfree(state);
>  }
> @@ -997,11 +1004,12 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
>  
> -int v4l2_subdev_init_finalize(struct v4l2_subdev *sd)
> +int __v4l2_subdev_init_finalize(struct v4l2_subdev *sd, const char *name,
> +				struct lock_class_key *key)
>  {
>  	struct v4l2_subdev_state *state;
>  
> -	state = __v4l2_subdev_state_alloc(sd);
> +	state = __v4l2_subdev_state_alloc(sd, name, key);
>  	if (IS_ERR(state))
>  		return PTR_ERR(state);
>  
> @@ -1009,7 +1017,7 @@ int v4l2_subdev_init_finalize(struct v4l2_subdev *sd)
>  
>  	return 0;
>  }
> -EXPORT_SYMBOL_GPL(v4l2_subdev_init_finalize);
> +EXPORT_SYMBOL_GPL(__v4l2_subdev_init_finalize);
>  
>  void v4l2_subdev_cleanup(struct v4l2_subdev *sd)
>  {
> @@ -1017,3 +1025,23 @@ void v4l2_subdev_cleanup(struct v4l2_subdev *sd)
>  	sd->active_state = NULL;
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_cleanup);
> +
> +struct v4l2_subdev_state *v4l2_subdev_lock_active_state(struct v4l2_subdev *sd)
> +{
> +	mutex_lock(&sd->active_state->lock);
> +
> +	return sd->active_state;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_lock_active_state);
> +
> +void v4l2_subdev_lock_state(struct v4l2_subdev_state *state)
> +{
> +	mutex_lock(&state->lock);
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_lock_state);
> +
> +void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state)
> +{
> +	mutex_unlock(&state->lock);
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_unlock_state);
> diff --git a/drivers/staging/media/tegra-video/vi.c b/drivers/staging/media/tegra-video/vi.c
> index 66b9ce160472..3516735f0cad 100644
> --- a/drivers/staging/media/tegra-video/vi.c
> +++ b/drivers/staging/media/tegra-video/vi.c
> @@ -491,6 +491,7 @@ static int __tegra_channel_try_format(struct tegra_vi_channel *chan,
>  				      struct v4l2_pix_format *pix)
>  {
>  	const struct tegra_video_format *fmtinfo;
> +	static struct lock_class_key key;
>  	struct v4l2_subdev *subdev;
>  	struct v4l2_subdev_format fmt;
>  	struct v4l2_subdev_state *sd_state;
> @@ -507,7 +508,8 @@ static int __tegra_channel_try_format(struct tegra_vi_channel *chan,
>  	if (!subdev)
>  		return -ENODEV;
>  
> -	sd_state = __v4l2_subdev_state_alloc(subdev);
> +	sd_state = __v4l2_subdev_state_alloc(subdev, "tegra:state->lock",
> +					     &key);
>  	if (IS_ERR(sd_state))
>  		return PTR_ERR(sd_state);
>  	/*
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 87c3126b2754..1810dde9c7fc 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -658,6 +658,7 @@ struct v4l2_subdev_pad_config {
>  /**
>   * struct v4l2_subdev_state - Used for storing subdev state information.
>   *
> + * @lock: mutex for the state
>   * @pads: &struct v4l2_subdev_pad_config array
>   *
>   * This structure only needs to be passed to the pad op if the 'which' field
> @@ -665,6 +666,8 @@ struct v4l2_subdev_pad_config {
>   * %V4L2_SUBDEV_FORMAT_ACTIVE it is safe to pass %NULL.
>   */
>  struct v4l2_subdev_state {
> +	/* lock for the struct v4l2_subdev_state fields */
> +	struct mutex lock;

I'm looking forward to the first AB-BA deadlock when subdevs will call
each other's operations in different orders :-) I'm pretty sure we'll
need a ww-mutex in v4l2_device to solve that. It's for later though.

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

>  	struct v4l2_subdev_pad_config *pads;
>  };
>  
> @@ -1156,10 +1159,14 @@ int v4l2_subdev_link_validate(struct media_link *link);
>   * __v4l2_subdev_state_alloc - allocate v4l2_subdev_state
>   *
>   * @sd: pointer to &struct v4l2_subdev for which the state is being allocated.
> + * @lock_name: name of the state lock
> + * @key: lock_class_key for the lock
>   *
>   * Must call __v4l2_subdev_state_free() when state is no longer needed.
>   */
> -struct v4l2_subdev_state *__v4l2_subdev_state_alloc(struct v4l2_subdev *sd);
> +struct v4l2_subdev_state *__v4l2_subdev_state_alloc(struct v4l2_subdev *sd,
> +						    const char *lock_name,
> +						    struct lock_class_key *key);
>  
>  /**
>   * __v4l2_subdev_state_free - free a v4l2_subdev_state
> @@ -1248,7 +1255,16 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
>   *
>   * Must call v4l2_subdev_cleanup() when the subdev is being removed.
>   */
> -int v4l2_subdev_init_finalize(struct v4l2_subdev *sd);
> +#define v4l2_subdev_init_finalize(sd)                                          \
> +	({                                                                     \
> +		static struct lock_class_key __key;                            \
> +		const char *name = KBUILD_BASENAME                             \
> +			":" __stringify(__LINE__) ":subdev->state->lock";      \
> +		__v4l2_subdev_init_finalize(sd, name, &__key);                 \
> +	})
> +
> +int __v4l2_subdev_init_finalize(struct v4l2_subdev *sd, const char *name,
> +				struct lock_class_key *key);
>  
>  /**
>   * v4l2_subdev_cleanup() - Release the resources needed by the subdevice
> @@ -1271,4 +1287,34 @@ v4l2_subdev_get_active_state(struct v4l2_subdev *sd)
>  	return sd->active_state;
>  }
>  
> +/**
> + * v4l2_subdev_lock_active_state() - Lock and return the active subdev state for
> + *				     subdevice
> + * @sd: The subdevice
> + *
> + * Return the locked active state for the subdevice, or NULL if the subdev
> + * does not support active state.
> + *
> + * Must be unlocked with v4l2_subdev_unlock_state() after use.
> + */
> +struct v4l2_subdev_state *v4l2_subdev_lock_active_state(struct v4l2_subdev *sd);
> +
> +/**
> + * v4l2_subdev_lock_state() - Lock the subdev state
> + * @state: The subdevice state
> + *
> + * Lock the given subdev state.
> + *
> + * Must be unlocked with v4l2_subdev_unlock_state() after use.
> + */
> +void v4l2_subdev_lock_state(struct v4l2_subdev_state *state);
> +
> +/**
> + * v4l2_subdev_unlock_state() - Unlock the subdev state
> + * @state: The subdevice state
> + *
> + * Unlock the given subdev state.
> + */
> +void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state);
> +
>  #endif

-- 
Regards,

Laurent Pinchart

Re: [PATCH v10 03/38] media: subdev: pass also the active state to subdevs from ioctls

[Laurent Pinchart]

Hi Tomi,

Thank you for the patch.

On Tue, Nov 30, 2021 at 04:15:01PM +0200, Tomi Valkeinen wrote:
> At the moment when a subdev op is called, the TRY subdev state
> (subdev_fh->state) is passed as a parameter even for the ACTIVE case, or
> alternatively a NULL can be passed for ACTIVE case. This used to make
> sense, as the ACTIVE state was handled internally by the subdev drivers.
> 
> We now have a state for the ACTIVE case in a standard place, and can
> pass that also to the drivers. This patch changes the subdev ioctls to
> either pass the TRY or ACTIVE state to the subdev.
> 
> Unfortunately many drivers call ops from other subdevs, and implicitly
> pass NULL as the state, so this is just a partial solution. A coccinelle
> spatch could perhaps be created which fixes the drivers' subdev calls.
> 
> For all current upstream drivers this doesn't matter, as they do not
> expect to get a valid state for ACTIVE case. But future drivers which
> support multiplexed streaming and routing will depend on getting a state
> for both active and try cases.
> 
> For new drivers we can mandate that the pipelines where the drivers are
> used need to pass the state properly, or preferably, not call such
> subdev ops at all.

I'm not sure we can really get rid of all those calls, but I believe we
could at least provide wrapper functions that should be used instead,
and that will handle state management.

> However, if an existing subdev driver is changed to support multiplexed
> streams, the driver has to consider cases where its ops will be called
> with NULL state. The problem can easily be solved by using the
> v4l2_subdev_validate_and_lock_state() helper, introduced in a follow up
> patch.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 73 +++++++++++++++++++++++----
>  1 file changed, 63 insertions(+), 10 deletions(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index de160140d63b..3289875d9ec1 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -353,6 +353,53 @@ const struct v4l2_subdev_ops v4l2_subdev_call_wrappers = {
>  EXPORT_SYMBOL(v4l2_subdev_call_wrappers);
>  
>  #if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
> +
> +static struct v4l2_subdev_state *
> +subdev_ioctl_get_state(struct v4l2_subdev *sd, struct v4l2_subdev_fh *subdev_fh,
> +		       unsigned int cmd, void *arg)
> +{
> +	u32 which;
> +
> +	switch (cmd) {
> +	default:
> +		return NULL;
> +
> +	case VIDIOC_SUBDEV_G_FMT:
> +	case VIDIOC_SUBDEV_S_FMT: {
> +		which = ((struct v4l2_subdev_format *)arg)->which;
> +		break;
> +	}
> +	case VIDIOC_SUBDEV_G_CROP:
> +	case VIDIOC_SUBDEV_S_CROP: {
> +		which = ((struct v4l2_subdev_crop *)arg)->which;
> +		break;
> +	}
> +	case VIDIOC_SUBDEV_ENUM_MBUS_CODE: {
> +		which = ((struct v4l2_subdev_mbus_code_enum *)arg)->which;
> +		break;
> +	}
> +	case VIDIOC_SUBDEV_ENUM_FRAME_SIZE: {
> +		which = ((struct v4l2_subdev_frame_size_enum *)arg)->which;
> +		break;
> +	}
> +
> +	case VIDIOC_SUBDEV_ENUM_FRAME_INTERVAL: {
> +		which = ((struct v4l2_subdev_frame_interval_enum *)arg)->which;
> +		break;
> +	}
> +
> +	case VIDIOC_SUBDEV_G_SELECTION:
> +	case VIDIOC_SUBDEV_S_SELECTION: {
> +		which = ((struct v4l2_subdev_selection *)arg)->which;
> +		break;
> +	}
> +	}
> +
> +	return which == V4L2_SUBDEV_FORMAT_TRY ?
> +			     subdev_fh->state :
> +			     v4l2_subdev_get_active_state(sd);
> +}
> +
>  static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  {
>  	struct video_device *vdev = video_devdata(file);
> @@ -360,8 +407,11 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  	struct v4l2_fh *vfh = file->private_data;
>  	struct v4l2_subdev_fh *subdev_fh = to_v4l2_subdev_fh(vfh);
>  	bool ro_subdev = test_bit(V4L2_FL_SUBDEV_RO_DEVNODE, &vdev->flags);
> +	struct v4l2_subdev_state *state;
>  	int rval;
>  
> +	state = subdev_ioctl_get_state(sd, subdev_fh, cmd, arg);
> +
>  	switch (cmd) {
>  	case VIDIOC_SUBDEV_QUERYCAP: {
>  		struct v4l2_subdev_capability *cap = arg;
> @@ -484,7 +534,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  
>  		memset(format->reserved, 0, sizeof(format->reserved));
>  		memset(format->format.reserved, 0, sizeof(format->format.reserved));
> -		return v4l2_subdev_call(sd, pad, get_fmt, subdev_fh->state, format);
> +		return v4l2_subdev_call(sd, pad, get_fmt, state, format);
>  	}
>  
>  	case VIDIOC_SUBDEV_S_FMT: {
> @@ -495,7 +545,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  
>  		memset(format->reserved, 0, sizeof(format->reserved));
>  		memset(format->format.reserved, 0, sizeof(format->format.reserved));
> -		return v4l2_subdev_call(sd, pad, set_fmt, subdev_fh->state, format);
> +		return v4l2_subdev_call(sd, pad, set_fmt, state, format);
>  	}
>  
>  	case VIDIOC_SUBDEV_G_CROP: {
> @@ -509,7 +559,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  		sel.target = V4L2_SEL_TGT_CROP;
>  
>  		rval = v4l2_subdev_call(
> -			sd, pad, get_selection, subdev_fh->state, &sel);
> +			sd, pad, get_selection, state, &sel);
>  
>  		crop->rect = sel.r;
>  
> @@ -531,7 +581,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  		sel.r = crop->rect;
>  
>  		rval = v4l2_subdev_call(
> -			sd, pad, set_selection, subdev_fh->state, &sel);
> +			sd, pad, set_selection, state, &sel);
>  
>  		crop->rect = sel.r;
>  
> @@ -542,7 +592,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  		struct v4l2_subdev_mbus_code_enum *code = arg;
>  
>  		memset(code->reserved, 0, sizeof(code->reserved));
> -		return v4l2_subdev_call(sd, pad, enum_mbus_code, subdev_fh->state,
> +		return v4l2_subdev_call(sd, pad, enum_mbus_code, state,
>  					code);
>  	}
>  
> @@ -550,7 +600,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  		struct v4l2_subdev_frame_size_enum *fse = arg;
>  
>  		memset(fse->reserved, 0, sizeof(fse->reserved));
> -		return v4l2_subdev_call(sd, pad, enum_frame_size, subdev_fh->state,
> +		return v4l2_subdev_call(sd, pad, enum_frame_size, state,
>  					fse);
>  	}
>  
> @@ -575,7 +625,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  		struct v4l2_subdev_frame_interval_enum *fie = arg;
>  
>  		memset(fie->reserved, 0, sizeof(fie->reserved));
> -		return v4l2_subdev_call(sd, pad, enum_frame_interval, subdev_fh->state,
> +		return v4l2_subdev_call(sd, pad, enum_frame_interval, state,
>  					fie);
>  	}
>  
> @@ -584,7 +634,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  
>  		memset(sel->reserved, 0, sizeof(sel->reserved));
>  		return v4l2_subdev_call(
> -			sd, pad, get_selection, subdev_fh->state, sel);
> +			sd, pad, get_selection, state, sel);
>  	}
>  
>  	case VIDIOC_SUBDEV_S_SELECTION: {
> @@ -595,7 +645,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  
>  		memset(sel->reserved, 0, sizeof(sel->reserved));
>  		return v4l2_subdev_call(
> -			sd, pad, set_selection, subdev_fh->state, sel);
> +			sd, pad, set_selection, state, sel);
>  	}
>  
>  	case VIDIOC_G_EDID: {
> @@ -829,10 +879,13 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
>  	if (is_media_entity_v4l2_subdev(pad->entity)) {
>  		struct v4l2_subdev *sd =
>  			media_entity_to_v4l2_subdev(pad->entity);
> +		struct v4l2_subdev_state *state;
> +
> +		state = v4l2_subdev_get_active_state(sd);
>  
>  		fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
>  		fmt->pad = pad->index;
> -		return v4l2_subdev_call(sd, pad, get_fmt, NULL, fmt);
> +		return v4l2_subdev_call(sd, pad, get_fmt, state, fmt);
>  	}
>  
>  	WARN(pad->entity->function != MEDIA_ENT_F_IO_V4L,

-- 
Regards,

Laurent Pinchart

Re: [PATCH v10 02/38] media: subdev: add active state to struct v4l2_subdev

[Laurent Pinchart]

Hello,

On Wed, Dec 15, 2021 at 05:25:12PM +0100, Jacopo Mondi wrote:
> On Wed, Dec 15, 2021 at 05:35:53PM +0200, Tomi Valkeinen wrote:
> > On 15/12/2021 11:38, Jacopo Mondi wrote:
> > > On Wed, Dec 15, 2021 at 10:06:16AM +0200, Tomi Valkeinen wrote:
> > > > On 13/12/2021 23:30, Jacopo Mondi wrote:
> > > > > On Tue, Nov 30, 2021 at 04:15:00PM +0200, Tomi Valkeinen wrote:
> > > > > > Add a new 'active_state' field to struct v4l2_subdev to which we can
> > > > > > store the active state of a subdev. This will place the subdev
> > > > > > configuration into a known place, allowing us to use the state directly
> > > > > > from the v4l2 framework, thus simplifying the drivers.
> > > > > >
> > > > > > Also add functions v4l2_subdev_init_finalize() and
> > > > > > v4l2_subdev_cleanup(), which will allocate and free the active state.
> > > > > > The functions are named in a generic way so that they can be also used
> > > > > > for other subdev initialization work.
> > > > > >
> > > > > > Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > > > > > ---
> > > > > >    drivers/media/v4l2-core/v4l2-subdev.c | 21 +++++++++++
> > > > > >    include/media/v4l2-subdev.h           | 54 +++++++++++++++++++++++++++
> > > > > >    2 files changed, 75 insertions(+)
> > > > > >
> > > > > > diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> > > > > > index fe49c86a9b02..de160140d63b 100644
> > > > > > --- a/drivers/media/v4l2-core/v4l2-subdev.c
> > > > > > +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> > > > > > @@ -943,3 +943,24 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
> > > > > >    	v4l2_subdev_notify(sd, V4L2_DEVICE_NOTIFY_EVENT, (void *)ev);
> > > > > >    }
> > > > > >    EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
> > > > > > +
> > > > > > +int v4l2_subdev_init_finalize(struct v4l2_subdev *sd)
> > > > > > +{
> > > > > > +	struct v4l2_subdev_state *state;
> > > > > > +
> > > > > > +	state = __v4l2_subdev_state_alloc(sd);
> > > > > > +	if (IS_ERR(state))
> > > > > > +		return PTR_ERR(state);
> > > > > > +
> > > > > > +	sd->active_state = state;
> > > > >
> > > > > You could spare the local var..
> > > >
> > > > I don't like to write error values to storage fields. Here I only touch the
> > > > field when succesfull.
> > > >
> > > > > > +
> > > > > > +	return 0;
> > > > > > +}
> > > > > > +EXPORT_SYMBOL_GPL(v4l2_subdev_init_finalize);
> > > > > > +
> > > > > > +void v4l2_subdev_cleanup(struct v4l2_subdev *sd)
> > > > > > +{
> > > > > > +	__v4l2_subdev_state_free(sd->active_state);
> > > > > > +	sd->active_state = NULL;
> > > > > > +}
> > > > > > +EXPORT_SYMBOL_GPL(v4l2_subdev_cleanup);
> > > > > > diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> > > > > > index e52bf508c75b..87c3126b2754 100644
> > > > > > --- a/include/media/v4l2-subdev.h
> > > > > > +++ b/include/media/v4l2-subdev.h
> > > > > > @@ -645,6 +645,9 @@ struct v4l2_subdev_ir_ops {
> > > > > >     * This structure only needs to be passed to the pad op if the 'which' field
> > > > > >     * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
> > > > > >     * %V4L2_SUBDEV_FORMAT_ACTIVE it is safe to pass %NULL.
> > > > > > + *
> > > > > > + * Note: This struct is also used in active state, and the try_ prefix is
> > > > > > + * historical and to be removed.
> > > > > >     */
> > > > > >    struct v4l2_subdev_pad_config {
> > > > > >    	struct v4l2_mbus_framefmt try_fmt;
> > > > > > @@ -898,6 +901,8 @@ struct v4l2_subdev_platform_data {
> > > > > >     * @subdev_notifier: A sub-device notifier implicitly registered for the sub-
> > > > > >     *		     device using v4l2_async_register_subdev_sensor().
> > > > > >     * @pdata: common part of subdevice platform data
> > > > > > + * @active_state: active state for the subdev (NULL for subdevs tracking the
> > > > > > + *                state internally)
> > > > >
> > > > > I would suggest here that init_finalize() should be used
> > > > >
> > > > >    * @active_state: subdev active state. Initialized by calling
> > > > >                     v4l2_subdev_init_finalize() by drivers that support
> > > > >                     state
> > > >
> > > > Ok.
> > > >
> > > > > I would have said "the state API" but then we would need to define
> > > > > what the state API as a concept :)
> > > > >
> > > > > >     *
> > > > > >     * Each instance of a subdev driver should create this struct, either
> > > > > >     * stand-alone or embedded in a larger struct.
> > > > > > @@ -929,6 +934,19 @@ struct v4l2_subdev {
> > > > > >    	struct v4l2_async_notifier *notifier;
> > > > > >    	struct v4l2_async_notifier *subdev_notifier;
> > > > > >    	struct v4l2_subdev_platform_data *pdata;
> > > > > > +
> > > > > > +	/*
> > > > > > +	 * The fields below are private, and should only be accessed via
> > > > > > +	 * appropriate functions.
> > > > > > +	 */
> > > > > > +
> > > > > > +	/*
> > > > > > +	 * TODO: state should most likely be changed from a pointer to an
> > > > > > +	 * embedded field. For the time being it's kept as a pointer to more
> > > > > > +	 * easily catch uses of state in the cases where the driver doesn't
> > > > > > +	 * support it.
> > > > >
> > > > > what about "... For the time being it's kept as a pointer to allow drivers
> > > > > to support state as an opt-in feature."
> > > >
> > > > Well, that's not quite correct. Even if it was an embedded field, it would
> > > > be an opt-in feature. The point with the pointer is only that if an opt-out
> > > > driver accidentally tries to access the field, it'll get NULL, and hopefully
> > > > fail.
> > >
> > > Ok, what I mean is that being a pointer, it's easy to identify if a
> > > driver has opted-in. Whatever, we're saying the same thing
> > >
> > > > > > +	 */
> > > > > > +	struct v4l2_subdev_state *active_state;
> > > > > >    };
> > > > > >
> > > > > >
> > > > > > @@ -1217,4 +1235,40 @@ extern const struct v4l2_subdev_ops v4l2_subdev_call_wrappers;
> > > > > >    void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
> > > > > >    			      const struct v4l2_event *ev);
> > > > > >
> > > > > > +/**
> > > > > > + * v4l2_subdev_init_finalize() - Finalize the initialization of the subdevice
> > > > > > + * @sd: The subdev
> > > > > > + *
> > > > > > + * This finalizes the initialization of the subdev, including allocation of
> > > > > > + * the active state for the subdev.
> > > > > > + *
> > > > > > + * This must be called by the subdev drivers that use the centralized active
> > > > > > + * state, after the subdev struct has been initialized and
> > > > > > + * media_entity_pads_init() has been called.
> > > > >
> > > > > Or just "It must be called by subdev drivers that support active state
> > > > > as the last operation before registering the subdev."
> > > >
> > > > That's not quite true either. The driver may do setup operations requiring
> > > > the active state before registering the subdev.
> > >
> > > Are there other setup operations on the subdev besides the usual
> > > v4l2_.._subdev_init(), media_pads_init() etc that have to be called
> > > after a function that by its name 'finalizes' the initialization ?
> > > Anyway, just a nit, whatever is fine
> >
> > I was only thinking about the finalization of the state. But as it's now
> > "init finalize", I think it makes sense that it would be just before subdev
> > registration. But I'm actually setting up the notifiers between the
> > v4l2_subdev_init_finalize and v4l2_async_register_subdev... I need to see if
> > there's any reason for that, or can the v4l2_subdev_init_finalize be moved
> > to be just before subdev registration.
> 
> I see, of course there might be other operations going on between
> init_finalize() and subdev_register(), what I meant was to only
> discuss about helper function provided by the core, not driver
> specific functions. Intrpreted this way I understand why my suggestion
> is confusing, feel free to ignore it!
> 
> > > > > > + *
> > > > > > + * Must call v4l2_subdev_cleanup() when the subdev is being removed.
> > > > > > + */
> > > > > > +int v4l2_subdev_init_finalize(struct v4l2_subdev *sd);
> > > > > > +
> > > > > > +/**
> > > > > > + * v4l2_subdev_cleanup() - Release the resources needed by the subdevice
> > > > > > + * @sd: The subdevice
> > > > > > + *
> > > > > > + * This will release the resources allocated in v4l2_subdev_init_finalize.
> > > > > > + */
> > > > > > +void v4l2_subdev_cleanup(struct v4l2_subdev *sd);
> > > > > > +
> > > > > > +/**
> > > > > > + * v4l2_subdev_get_active_state() - Return the active subdev state for subdevice
> > > > > > + * @sd: The subdevice
> > > > > > + *
> > > > > > + * Return the active state for the subdevice, or NULL if the subdev does not
> > > > >
> > > > > nit:
> > > > > The other functions in the file are documented with "Returns" (and
> > > > > also "This function" seems to be more commonly used that just "This").

kerneldoc actually documents usage of "Return:", but seems to support
other syntaxes as well. That's something that should be fixed through
the file first if needed, for now let's just be consistent.

> > > > Ok, I'll try to get these consistent.
> > > >
> > > > > > + * support active state.
> > > > > > + */
> > > > > > +static inline struct v4l2_subdev_state *
> > > > > > +v4l2_subdev_get_active_state(struct v4l2_subdev *sd)
> > > > > > +{
> > > > > > +	return sd->active_state;
> > > > > > +}
> > > > > > +
> > > > >
> > > > > This is only used by the core, and by top drivers when doing:
> > > > >
> > > > >         ret = v4l2_subdev_call(sd, pad, enum_frame_size,
> > > > >                                v4l2_subdev_get_active_state(sd), &fse);
> > > > >
> > > > > To pass to the subdev its own state.
> > > > >
> > > > > I wonder if a macro like
> > > > >
> > > > > #define v4l2_subdev_call_state(sd, o, f, args)					\
> > > > > 	({									\
> > > > > 	 struct v4l2_subdev_state *st = v4l2_subdev_get_active_state(sd);	\
> > > > > 	 v4l2_subdev_call(sd, o, f, st, args);					\
> > > > > 	 })
> > > > >
> > > > > Could help simplify the state API.
> > > >
> > > > Possibly, but we need to support the TRY case also. So I would call it
> > > > something more specific, v4l2_subdev_call_active(), or such.
> > >
> > > If the caller intends to pass is a try state (which it has to allocate
> > > explicitly) shouldn't it just use v4l2_subdev_call() ?
> >
> > Yes. But I think this is pretty confusing already, and in my opinion adding
> > v4l2_subdev_call_state() would confuse this even more =). I think at least
> > it has to make it clear that the subdev is being called using its active
> > state.
> >
> > But maybe just using v4l2_subdev_call() for now and seeing how this evolves
> > is best. It's trivial to add the helper later.

As mentioned separately, I think we should have wrappers for individual
operations. We'll see about that later.

> Make sense
> 
> > > > > Please be patient if I digress a bit more on the API now..
> > > > >
> > > > > Looking forward in the series we'll end up having 5 functions to
> > > > > manage state, if I've not missed any
> > > > >
> > > > > 1) struct v4l2_subdev_state *v4l2_subdev_get_active_state(struct v4l2_subdev *sd);
> > > > >
> > > > > 2) struct v4l2_subdev_state *v4l2_subdev_lock_active_state(struct v4l2_subdev *sd);
> > > > >
> > > > > 3) void v4l2_subdev_lock_state(struct v4l2_subdev_state *state);
> > > > >
> > > > > 4) void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state);
> > > > >
> > > > > 5) struct v4l2_subdev_state *v4l2_subdev_lock_and_return_state(struct v4l2_subdev *sd,
> > > > > 				                               struct v4l2_subdev_state *state)
> > > > >
> > > > > 1) and 2) are helpers to get (locked/unlocked) v4l2_subdevice.state,
> > > > > used in operations which implicitly operates on the active state
> > > > > (s_stream etc)
> > > > >
> > > > > I wonder if we shouldn't just have a locked v4l2_subdev_get_state()
> > > > > and an optional unlocked version which can be used by the core in
> > > > > place of v4l2_subdev_get_active_state() and by drivers which know what
> > > > > they're doing by using the unlocked version
> > > >
> > > > I think we need to always make it clear if we're dealing with the active or
> > > > try state (when it's relevant). And I also want to be clear if a lock is
> > > > taken. v4l2_subdev_get_state() would hide both of those.
> > >
> > > It is clear you're dealing with active state if the only argument is
> > > a struct v4l2_subdevice *.
> >
> > Perhaps, if you understand the internals here. I'd rather be more explicit
> > and clear in the function names. Again, we can easily change these later if
> > they don't feel necessary.
> >
> > > If you do not receive a state as an operation parameter you can only
> > > get the one is the subdev, which is by definition the active one.
> > >
> > > When it comes to locked, if the usage pattern is that the state is to
> > > be locked before being accessed, and the unlocked case is the
> > > exception (as it is only used by the core and by the top drivers) then
> > > I don't feel so strongly as you about having _locked() in the name.
> >
> > My reasoning here is that we can have function pairs like get_xxx, put_xxx,
> > and then it's clear that those match and they need to be paired.
> 
> Like:
> 
> v4l2_subdev_lock_state(state)
> v4l2_subdev_unlock_state(state)
> 
> > But if we have just a get_xxx, at least my thinking is that it just returns
> > something, and I'll use it and forget about it when I don't need it anymore.
> >
> > But lock_xxx makes it clear I have to unlock it afterwards.
> 
> so it gets
> 
> state = v4l2_subdev_lock_active_state(sd)
> v4l2_subdev_unlock_state(state)
> 
> > That said, maybe instead of v4l2_subdev_lock_active_state we should have
> > v4l2_subdev_lock_and_get_active_state. But it gets a bit long...
> 
> No no please :)
> 
> You bought me with:
> v4l2_subdev_lock_active_state() + v4l2_subdev_unlock_state()
> 
> > > Anyway, your code, your names..
> > >
> > > > > 3) and 4) lock the state at hand (to be used in operations with a
> > > > > 'state' parameter)
> > > > >
> > > > > 5) is currently unused and it still feels a bit ill-defined. If the state
> > > > > passed in as parameter is NULL then deflect it to sd->state, so I
> > > > > guess it assumes that the user is a state-aware subdev driver which
> > > > > needs to protect against in-kernel callers that do no pass in a valid state to
> > > >
> > > > Correct.
> > > >
> > > > > the operation call. This implies that if the caller passes in a NULL
> > > > > state for the TRY format, this gets deflected to the subdev's active
> > > > > state, something which should not happen, right ? I would be tempted
> > > >
> > > > Yes. That's an illegal call, isn't it? Or does it happen and we need to
> > > > protect against it?
> > > >
> > > > > to just fail loud if !state and assume if you're moving your subdev to
> > > > > use state you should be able to fix the caller as well.
> > > >
> > > > That would be nice, but I think it's not realistic. If you convert a sensor
> > > > driver to multiplexed streams, are you going to fix (convert to multiplexed
> > > > streams) all the bridges and SoC drivers that may use that sensor? How do
> > > > you even find all those bridges and SoCs...
> > >
> > > Of course no. You fix the one you're using. You're converting a sensor
> > > driver, you can fix the top driver too. Other users of the sensor
> > > driver will probably scream when moving to the next driver release
> > > that has been made state aware, so they'll hopefully fix their top driver
> > > too. After all, this applies equally to downstrems as well and
> >
> > Well, I'm not a maintainer in linux-media, but I would nack that approach
> > =). We can't just go breaking other platforms, and hoping other devs will
> > fix them.

I'd love to agree with Jacopo here and break things all the time to get
other people to fix them, but I doubt that I'd make many friends while
doing so :-)

This being said, the more we can push for conversions to happen quickly,
the better. For instance, new sensor drivers should rely on the active
state being passed to all subdev operations, so that any existing host
driver that wants to use them will need to be converted.
v4l2_subdev_lock_and_return_state() in subdev drivers should only be
used as a transition tool.

Similarly, a way to enforce that new host drivers only work with sensor
drivers that have been converted would be good.

We should aim for the removal of v4l2_subdev_lock_and_return_state(). A
WARN_ON() there will be useful at some point, but it's of course too
early. A bit of yak shaving may also help, by asking maintainers and
contributors to sensor and host drivers to migrate to the new API.

> > > providing an helper to work around issues is not the best idea in my
> > > opinion. Also the helper should be used in the subdev driver in place
> > > of the regular v4l2_subdev_lock_state() to protect just in case
> > > against legacy callers. When will they be moved to use the regular
> > > v4l2_subdev_lock_state() ?
> >
> > Note that this is needed only when porting an existing and presumably in-use
> > subdev driver. You don't need this if you write a new driver.
> >
> > The users of v4l2_subdev_lock_and_return_state are easy to find and easy to
> > change to v4l2_subdev_lock_state when we decide a particular driver doesn't
> > need to support legacy subdevs.
> 
> I'm just concerned that as long as we offer an helper to work this
> around (and the helper might introduce subtle issues like mixing
> try/active context) once we
> s/v4l2_subdev_lock_and_return_state/v4l2_subdev_lock_state we'll be
> anyway breaking users.
> 
> > I don't like this at all but, afaics, we can't break the existing platforms.
> > This function is a very simple work-around for the time being, and easy to
> > drop later.
> >
> > > Once a subdev driver has been moved to be state aware callers that
> > > passes in a NULL state should be fixed. As we can't fix them all,
> > > screaming loud and clearly say what has to be done to move forward is
> > > in my opinion better than introducing a temporary function that
> > > replaces the regular API and that subdev should decide to use just in
> > > case (and which can lead to subtle errors like using the active state
> > > in a TRY context).
> > >
> > > If you want to protect against broken callers then what about
> > > doing the "state = state ? : sd->active_state;" dance in
> > > v4l2_subdev_lock_state() with a WARN_ONCE(!state) so that
> > > subdev drivers can use the regular API from day 0 ?
> >
> > Hmm, I think that is an option. I didn't implement "state = state ? :
> > sd->active_state;" in the v4l2_subdev_lock_state() as I didn't want
> > v4l2_subdev_lock_state() to hide an invalid NULL parameter. But adding
> > WARN_ONCE() would warn about it.

As we can't WARN() unconditionally yet when encountering a driver that
hasn't been converted, we need to keep v4l2_subdev_lock_and_return_state()
as an alternative that won't WARN. Do I understand that adding
"state = state ? : sd->active_state;" + WARN_ON in
v4l2_subdev_lock_state() would then be used only to catch invalid
combinations with a warning instead of a crash ? What would it help with
?

> > I'm still undecided, though. The WARN_ONCE would come only once for the
> > whole kernel, per boot, wouldn't it? We could have a macro for
> > v4l2_subdev_lock_state, but then we'd get lots of warnings. And a full WARN
> > just because a driver hasn't been updated is a bit harsh. Maybe we can start
> > with just a normal warning print.
> 
> There is a precendet I can think of: async matching on device or
> endpoints. Initially all async subdevs where matched on the remote end
> device node. For more complex setups this didn't scale, as the same
> remote can have multiple endpoints, and matching on the device node
> would have created false positives. So v4l2-async was moved to match on
> endpoints with some legacy compatibility code
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/media/v4l2-core/v4l2-async.c#n69
> 
> 	if (dev && dev->driver) {
> 		if (sd_fwnode_is_ep)
> 			dev_warn(dev, "Driver %s uses device fwnode, incorrect match may occur\n",
> 				 dev->driver->name);
> 		dev_notice(dev, "Consider updating driver %s to match on endpoints\n",
> 			   dev->driver->name);
> 	}
> 
> Can't we have something like this in v4l2_subdev_lock_state() ?

I think we need to start by being silent first, then move to dev_warn(),
then to WARN(), and finally drop the option. This can be done with
v4l2_subdev_lock_and_return_state() I believe, which would be used by
driver that have been converted but still need to support legacy
combinations. Drivers that don't want to support legacy combinations
would use v4l2_subdev_lock_state(). Usage of
v4l2_subdev_lock_and_return_state() in a driver would indicate more work
is needed, and would be a useful indication of how far we are in our
conversion effort.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v10 05/38] media: subdev: Add v4l2_subdev_lock_and_return_state()

[Laurent Pinchart]

Hi Tomi,

Thank you for the patch.

On Tue, Nov 30, 2021 at 04:15:03PM +0200, Tomi Valkeinen wrote:
> All suitable subdev ops are now passed either the TRY or the ACTIVE
> state by the v4l2 core. However, other subdev drivers can still call the
> ops passing NULL as the state, implying the active case.
> 
> For all current upstream drivers this doesn't matter, as they do not
> expect to get a valid state for ACTIVE case. But future drivers which
> support multiplexed streaming and routing will depend on getting a state
> for both active and try cases.
> 
> For new drivers we can mandate that the pipelines where the drivers are
> used need to pass the state properly, or preferably, not call such
> subdev ops at all.
> 
> However, if an existing subdev driver is changed to support multiplexed
> streams, the driver has to consider cases where its ops will be called
> with NULL state. The problem can easily be solved by using the
> v4l2_subdev_lock_and_return_state() helper, introduced here.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  include/media/v4l2-subdev.h | 31 +++++++++++++++++++++++++++++++
>  1 file changed, 31 insertions(+)
> 
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 1810dde9c7fc..873bbe0686e3 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -1317,4 +1317,35 @@ void v4l2_subdev_lock_state(struct v4l2_subdev_state *state);
>   */
>  void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state);
>  
> +/**
> + * v4l2_subdev_lock_and_return_state() - Gets locked TRY or ACTIVE subdev state
> + * @sd: subdevice
> + * @state: subdevice state as passed to the subdev op
> + *
> + * Due to legacy reasons, when subdev drivers call ops in other subdevs they use
> + * NULL as the state parameter, as subdevs always used to have their active
> + * state stored privately.
> + *
> + * However, newer state-aware subdev drivers, which store their active state in
> + * a common place, subdev->active_state, expect to always get a proper state as
> + * a parameter.
> + *
> + * These state-aware drivers can use v4l2_subdev_lock_and_return_state() instead
> + * of v4l2_subdev_lock_state(). v4l2_subdev_lock_and_return_state() solves the
> + * issue by using subdev->state in case the passed state is NULL.
> + *
> + * This is a temporary helper function, and should be removed when we can ensure
> + * that all drivers pass proper state when calling other subdevs.
> + */
> +static inline struct v4l2_subdev_state *
> +v4l2_subdev_lock_and_return_state(struct v4l2_subdev *sd,
> +				  struct v4l2_subdev_state *state)
> +{
> +	state = state ? state : sd->active_state;

Can we add a dev_warn() when state is NULL ? This will help speeding up
the transition.

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> +
> +	v4l2_subdev_lock_state(state);
> +
> +	return state;
> +}
> +
>  #endif

-- 
Regards,

Laurent Pinchart

Re: [PATCH v10 02/38] media: subdev: add active state to struct v4l2_subdev

[Laurent Pinchart]

On Thu, Dec 16, 2021 at 04:14:53PM +0200, Laurent Pinchart wrote:
> On Wed, Dec 15, 2021 at 05:25:12PM +0100, Jacopo Mondi wrote:
> > On Wed, Dec 15, 2021 at 05:35:53PM +0200, Tomi Valkeinen wrote:
> > > On 15/12/2021 11:38, Jacopo Mondi wrote:
> > > > On Wed, Dec 15, 2021 at 10:06:16AM +0200, Tomi Valkeinen wrote:
> > > > > On 13/12/2021 23:30, Jacopo Mondi wrote:
> > > > > > On Tue, Nov 30, 2021 at 04:15:00PM +0200, Tomi Valkeinen wrote:
> > > > > > > Add a new 'active_state' field to struct v4l2_subdev to which we can
> > > > > > > store the active state of a subdev. This will place the subdev
> > > > > > > configuration into a known place, allowing us to use the state directly
> > > > > > > from the v4l2 framework, thus simplifying the drivers.
> > > > > > >
> > > > > > > Also add functions v4l2_subdev_init_finalize() and
> > > > > > > v4l2_subdev_cleanup(), which will allocate and free the active state.
> > > > > > > The functions are named in a generic way so that they can be also used
> > > > > > > for other subdev initialization work.
> > > > > > >
> > > > > > > Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > > > > > > ---
> > > > > > >    drivers/media/v4l2-core/v4l2-subdev.c | 21 +++++++++++
> > > > > > >    include/media/v4l2-subdev.h           | 54 +++++++++++++++++++++++++++
> > > > > > >    2 files changed, 75 insertions(+)
> > > > > > >
> > > > > > > diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> > > > > > > index fe49c86a9b02..de160140d63b 100644
> > > > > > > --- a/drivers/media/v4l2-core/v4l2-subdev.c
> > > > > > > +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> > > > > > > @@ -943,3 +943,24 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
> > > > > > >    	v4l2_subdev_notify(sd, V4L2_DEVICE_NOTIFY_EVENT, (void *)ev);
> > > > > > >    }
> > > > > > >    EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
> > > > > > > +
> > > > > > > +int v4l2_subdev_init_finalize(struct v4l2_subdev *sd)
> > > > > > > +{
> > > > > > > +	struct v4l2_subdev_state *state;
> > > > > > > +
> > > > > > > +	state = __v4l2_subdev_state_alloc(sd);
> > > > > > > +	if (IS_ERR(state))
> > > > > > > +		return PTR_ERR(state);
> > > > > > > +
> > > > > > > +	sd->active_state = state;
> > > > > >
> > > > > > You could spare the local var..
> > > > >
> > > > > I don't like to write error values to storage fields. Here I only touch the
> > > > > field when succesfull.
> > > > >
> > > > > > > +
> > > > > > > +	return 0;
> > > > > > > +}
> > > > > > > +EXPORT_SYMBOL_GPL(v4l2_subdev_init_finalize);
> > > > > > > +
> > > > > > > +void v4l2_subdev_cleanup(struct v4l2_subdev *sd)
> > > > > > > +{
> > > > > > > +	__v4l2_subdev_state_free(sd->active_state);
> > > > > > > +	sd->active_state = NULL;
> > > > > > > +}
> > > > > > > +EXPORT_SYMBOL_GPL(v4l2_subdev_cleanup);
> > > > > > > diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> > > > > > > index e52bf508c75b..87c3126b2754 100644
> > > > > > > --- a/include/media/v4l2-subdev.h
> > > > > > > +++ b/include/media/v4l2-subdev.h
> > > > > > > @@ -645,6 +645,9 @@ struct v4l2_subdev_ir_ops {
> > > > > > >     * This structure only needs to be passed to the pad op if the 'which' field
> > > > > > >     * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
> > > > > > >     * %V4L2_SUBDEV_FORMAT_ACTIVE it is safe to pass %NULL.
> > > > > > > + *
> > > > > > > + * Note: This struct is also used in active state, and the try_ prefix is
> > > > > > > + * historical and to be removed.
> > > > > > >     */
> > > > > > >    struct v4l2_subdev_pad_config {
> > > > > > >    	struct v4l2_mbus_framefmt try_fmt;
> > > > > > > @@ -898,6 +901,8 @@ struct v4l2_subdev_platform_data {
> > > > > > >     * @subdev_notifier: A sub-device notifier implicitly registered for the sub-
> > > > > > >     *		     device using v4l2_async_register_subdev_sensor().
> > > > > > >     * @pdata: common part of subdevice platform data
> > > > > > > + * @active_state: active state for the subdev (NULL for subdevs tracking the
> > > > > > > + *                state internally)
> > > > > >
> > > > > > I would suggest here that init_finalize() should be used
> > > > > >
> > > > > >    * @active_state: subdev active state. Initialized by calling
> > > > > >                     v4l2_subdev_init_finalize() by drivers that support
> > > > > >                     state
> > > > >
> > > > > Ok.
> > > > >
> > > > > > I would have said "the state API" but then we would need to define
> > > > > > what the state API as a concept :)
> > > > > >
> > > > > > >     *
> > > > > > >     * Each instance of a subdev driver should create this struct, either
> > > > > > >     * stand-alone or embedded in a larger struct.
> > > > > > > @@ -929,6 +934,19 @@ struct v4l2_subdev {
> > > > > > >    	struct v4l2_async_notifier *notifier;
> > > > > > >    	struct v4l2_async_notifier *subdev_notifier;
> > > > > > >    	struct v4l2_subdev_platform_data *pdata;
> > > > > > > +
> > > > > > > +	/*
> > > > > > > +	 * The fields below are private, and should only be accessed via
> > > > > > > +	 * appropriate functions.
> > > > > > > +	 */
> > > > > > > +
> > > > > > > +	/*
> > > > > > > +	 * TODO: state should most likely be changed from a pointer to an
> > > > > > > +	 * embedded field. For the time being it's kept as a pointer to more
> > > > > > > +	 * easily catch uses of state in the cases where the driver doesn't
> > > > > > > +	 * support it.
> > > > > >
> > > > > > what about "... For the time being it's kept as a pointer to allow drivers
> > > > > > to support state as an opt-in feature."
> > > > >
> > > > > Well, that's not quite correct. Even if it was an embedded field, it would
> > > > > be an opt-in feature. The point with the pointer is only that if an opt-out
> > > > > driver accidentally tries to access the field, it'll get NULL, and hopefully
> > > > > fail.
> > > >
> > > > Ok, what I mean is that being a pointer, it's easy to identify if a
> > > > driver has opted-in. Whatever, we're saying the same thing
> > > >
> > > > > > > +	 */
> > > > > > > +	struct v4l2_subdev_state *active_state;
> > > > > > >    };
> > > > > > >
> > > > > > >
> > > > > > > @@ -1217,4 +1235,40 @@ extern const struct v4l2_subdev_ops v4l2_subdev_call_wrappers;
> > > > > > >    void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
> > > > > > >    			      const struct v4l2_event *ev);
> > > > > > >
> > > > > > > +/**
> > > > > > > + * v4l2_subdev_init_finalize() - Finalize the initialization of the subdevice
> > > > > > > + * @sd: The subdev
> > > > > > > + *
> > > > > > > + * This finalizes the initialization of the subdev, including allocation of
> > > > > > > + * the active state for the subdev.
> > > > > > > + *
> > > > > > > + * This must be called by the subdev drivers that use the centralized active
> > > > > > > + * state, after the subdev struct has been initialized and
> > > > > > > + * media_entity_pads_init() has been called.
> > > > > >
> > > > > > Or just "It must be called by subdev drivers that support active state
> > > > > > as the last operation before registering the subdev."
> > > > >
> > > > > That's not quite true either. The driver may do setup operations requiring
> > > > > the active state before registering the subdev.
> > > >
> > > > Are there other setup operations on the subdev besides the usual
> > > > v4l2_.._subdev_init(), media_pads_init() etc that have to be called
> > > > after a function that by its name 'finalizes' the initialization ?
> > > > Anyway, just a nit, whatever is fine
> > >
> > > I was only thinking about the finalization of the state. But as it's now
> > > "init finalize", I think it makes sense that it would be just before subdev
> > > registration. But I'm actually setting up the notifiers between the
> > > v4l2_subdev_init_finalize and v4l2_async_register_subdev... I need to see if
> > > there's any reason for that, or can the v4l2_subdev_init_finalize be moved
> > > to be just before subdev registration.
> > 
> > I see, of course there might be other operations going on between
> > init_finalize() and subdev_register(), what I meant was to only
> > discuss about helper function provided by the core, not driver
> > specific functions. Intrpreted this way I understand why my suggestion
> > is confusing, feel free to ignore it!
> > 
> > > > > > > + *
> > > > > > > + * Must call v4l2_subdev_cleanup() when the subdev is being removed.
> > > > > > > + */
> > > > > > > +int v4l2_subdev_init_finalize(struct v4l2_subdev *sd);
> > > > > > > +
> > > > > > > +/**
> > > > > > > + * v4l2_subdev_cleanup() - Release the resources needed by the subdevice
> > > > > > > + * @sd: The subdevice
> > > > > > > + *
> > > > > > > + * This will release the resources allocated in v4l2_subdev_init_finalize.
> > > > > > > + */
> > > > > > > +void v4l2_subdev_cleanup(struct v4l2_subdev *sd);
> > > > > > > +
> > > > > > > +/**
> > > > > > > + * v4l2_subdev_get_active_state() - Return the active subdev state for subdevice
> > > > > > > + * @sd: The subdevice
> > > > > > > + *
> > > > > > > + * Return the active state for the subdevice, or NULL if the subdev does not
> > > > > >
> > > > > > nit:
> > > > > > The other functions in the file are documented with "Returns" (and
> > > > > > also "This function" seems to be more commonly used that just "This").
> 
> kerneldoc actually documents usage of "Return:", but seems to support
> other syntaxes as well. That's something that should be fixed through
> the file first if needed, for now let's just be consistent.
> 
> > > > > Ok, I'll try to get these consistent.
> > > > >
> > > > > > > + * support active state.
> > > > > > > + */
> > > > > > > +static inline struct v4l2_subdev_state *
> > > > > > > +v4l2_subdev_get_active_state(struct v4l2_subdev *sd)
> > > > > > > +{
> > > > > > > +	return sd->active_state;
> > > > > > > +}
> > > > > > > +
> > > > > >
> > > > > > This is only used by the core, and by top drivers when doing:
> > > > > >
> > > > > >         ret = v4l2_subdev_call(sd, pad, enum_frame_size,
> > > > > >                                v4l2_subdev_get_active_state(sd), &fse);
> > > > > >
> > > > > > To pass to the subdev its own state.
> > > > > >
> > > > > > I wonder if a macro like
> > > > > >
> > > > > > #define v4l2_subdev_call_state(sd, o, f, args)					\
> > > > > > 	({									\
> > > > > > 	 struct v4l2_subdev_state *st = v4l2_subdev_get_active_state(sd);	\
> > > > > > 	 v4l2_subdev_call(sd, o, f, st, args);					\
> > > > > > 	 })
> > > > > >
> > > > > > Could help simplify the state API.
> > > > >
> > > > > Possibly, but we need to support the TRY case also. So I would call it
> > > > > something more specific, v4l2_subdev_call_active(), or such.
> > > >
> > > > If the caller intends to pass is a try state (which it has to allocate
> > > > explicitly) shouldn't it just use v4l2_subdev_call() ?
> > >
> > > Yes. But I think this is pretty confusing already, and in my opinion adding
> > > v4l2_subdev_call_state() would confuse this even more =). I think at least
> > > it has to make it clear that the subdev is being called using its active
> > > state.
> > >
> > > But maybe just using v4l2_subdev_call() for now and seeing how this evolves
> > > is best. It's trivial to add the helper later.
> 
> As mentioned separately, I think we should have wrappers for individual
> operations. We'll see about that later.
> 
> > Make sense
> > 
> > > > > > Please be patient if I digress a bit more on the API now..
> > > > > >
> > > > > > Looking forward in the series we'll end up having 5 functions to
> > > > > > manage state, if I've not missed any
> > > > > >
> > > > > > 1) struct v4l2_subdev_state *v4l2_subdev_get_active_state(struct v4l2_subdev *sd);
> > > > > >
> > > > > > 2) struct v4l2_subdev_state *v4l2_subdev_lock_active_state(struct v4l2_subdev *sd);
> > > > > >
> > > > > > 3) void v4l2_subdev_lock_state(struct v4l2_subdev_state *state);
> > > > > >
> > > > > > 4) void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state);
> > > > > >
> > > > > > 5) struct v4l2_subdev_state *v4l2_subdev_lock_and_return_state(struct v4l2_subdev *sd,
> > > > > > 				                               struct v4l2_subdev_state *state)
> > > > > >
> > > > > > 1) and 2) are helpers to get (locked/unlocked) v4l2_subdevice.state,
> > > > > > used in operations which implicitly operates on the active state
> > > > > > (s_stream etc)
> > > > > >
> > > > > > I wonder if we shouldn't just have a locked v4l2_subdev_get_state()
> > > > > > and an optional unlocked version which can be used by the core in
> > > > > > place of v4l2_subdev_get_active_state() and by drivers which know what
> > > > > > they're doing by using the unlocked version
> > > > >
> > > > > I think we need to always make it clear if we're dealing with the active or
> > > > > try state (when it's relevant). And I also want to be clear if a lock is
> > > > > taken. v4l2_subdev_get_state() would hide both of those.
> > > >
> > > > It is clear you're dealing with active state if the only argument is
> > > > a struct v4l2_subdevice *.
> > >
> > > Perhaps, if you understand the internals here. I'd rather be more explicit
> > > and clear in the function names. Again, we can easily change these later if
> > > they don't feel necessary.
> > >
> > > > If you do not receive a state as an operation parameter you can only
> > > > get the one is the subdev, which is by definition the active one.
> > > >
> > > > When it comes to locked, if the usage pattern is that the state is to
> > > > be locked before being accessed, and the unlocked case is the
> > > > exception (as it is only used by the core and by the top drivers) then
> > > > I don't feel so strongly as you about having _locked() in the name.
> > >
> > > My reasoning here is that we can have function pairs like get_xxx, put_xxx,
> > > and then it's clear that those match and they need to be paired.
> > 
> > Like:
> > 
> > v4l2_subdev_lock_state(state)
> > v4l2_subdev_unlock_state(state)
> > 
> > > But if we have just a get_xxx, at least my thinking is that it just returns
> > > something, and I'll use it and forget about it when I don't need it anymore.
> > >
> > > But lock_xxx makes it clear I have to unlock it afterwards.
> > 
> > so it gets
> > 
> > state = v4l2_subdev_lock_active_state(sd)
> > v4l2_subdev_unlock_state(state)
> > 
> > > That said, maybe instead of v4l2_subdev_lock_active_state we should have
> > > v4l2_subdev_lock_and_get_active_state. But it gets a bit long...
> > 
> > No no please :)
> > 
> > You bought me with:
> > v4l2_subdev_lock_active_state() + v4l2_subdev_unlock_state()
> > 
> > > > Anyway, your code, your names..
> > > >
> > > > > > 3) and 4) lock the state at hand (to be used in operations with a
> > > > > > 'state' parameter)
> > > > > >
> > > > > > 5) is currently unused and it still feels a bit ill-defined. If the state
> > > > > > passed in as parameter is NULL then deflect it to sd->state, so I
> > > > > > guess it assumes that the user is a state-aware subdev driver which
> > > > > > needs to protect against in-kernel callers that do no pass in a valid state to
> > > > >
> > > > > Correct.
> > > > >
> > > > > > the operation call. This implies that if the caller passes in a NULL
> > > > > > state for the TRY format, this gets deflected to the subdev's active
> > > > > > state, something which should not happen, right ? I would be tempted
> > > > >
> > > > > Yes. That's an illegal call, isn't it? Or does it happen and we need to
> > > > > protect against it?
> > > > >
> > > > > > to just fail loud if !state and assume if you're moving your subdev to
> > > > > > use state you should be able to fix the caller as well.
> > > > >
> > > > > That would be nice, but I think it's not realistic. If you convert a sensor
> > > > > driver to multiplexed streams, are you going to fix (convert to multiplexed
> > > > > streams) all the bridges and SoC drivers that may use that sensor? How do
> > > > > you even find all those bridges and SoCs...
> > > >
> > > > Of course no. You fix the one you're using. You're converting a sensor
> > > > driver, you can fix the top driver too. Other users of the sensor
> > > > driver will probably scream when moving to the next driver release
> > > > that has been made state aware, so they'll hopefully fix their top driver
> > > > too. After all, this applies equally to downstrems as well and
> > >
> > > Well, I'm not a maintainer in linux-media, but I would nack that approach
> > > =). We can't just go breaking other platforms, and hoping other devs will
> > > fix them.
> 
> I'd love to agree with Jacopo here and break things all the time to get
> other people to fix them, but I doubt that I'd make many friends while
> doing so :-)
> 
> This being said, the more we can push for conversions to happen quickly,
> the better. For instance, new sensor drivers should rely on the active
> state being passed to all subdev operations, so that any existing host
> driver that wants to use them will need to be converted.
> v4l2_subdev_lock_and_return_state() in subdev drivers should only be
> used as a transition tool.
> 
> Similarly, a way to enforce that new host drivers only work with sensor
> drivers that have been converted would be good.
> 
> We should aim for the removal of v4l2_subdev_lock_and_return_state(). A
> WARN_ON() there will be useful at some point, but it's of course too
> early. A bit of yak shaving may also help, by asking maintainers and
> contributors to sensor and host drivers to migrate to the new API.
> 
> > > > providing an helper to work around issues is not the best idea in my
> > > > opinion. Also the helper should be used in the subdev driver in place
> > > > of the regular v4l2_subdev_lock_state() to protect just in case
> > > > against legacy callers. When will they be moved to use the regular
> > > > v4l2_subdev_lock_state() ?
> > >
> > > Note that this is needed only when porting an existing and presumably in-use
> > > subdev driver. You don't need this if you write a new driver.
> > >
> > > The users of v4l2_subdev_lock_and_return_state are easy to find and easy to
> > > change to v4l2_subdev_lock_state when we decide a particular driver doesn't
> > > need to support legacy subdevs.
> > 
> > I'm just concerned that as long as we offer an helper to work this
> > around (and the helper might introduce subtle issues like mixing
> > try/active context) once we
> > s/v4l2_subdev_lock_and_return_state/v4l2_subdev_lock_state we'll be
> > anyway breaking users.
> > 
> > > I don't like this at all but, afaics, we can't break the existing platforms.
> > > This function is a very simple work-around for the time being, and easy to
> > > drop later.
> > >
> > > > Once a subdev driver has been moved to be state aware callers that
> > > > passes in a NULL state should be fixed. As we can't fix them all,
> > > > screaming loud and clearly say what has to be done to move forward is
> > > > in my opinion better than introducing a temporary function that
> > > > replaces the regular API and that subdev should decide to use just in
> > > > case (and which can lead to subtle errors like using the active state
> > > > in a TRY context).
> > > >
> > > > If you want to protect against broken callers then what about
> > > > doing the "state = state ? : sd->active_state;" dance in
> > > > v4l2_subdev_lock_state() with a WARN_ONCE(!state) so that
> > > > subdev drivers can use the regular API from day 0 ?
> > >
> > > Hmm, I think that is an option. I didn't implement "state = state ? :
> > > sd->active_state;" in the v4l2_subdev_lock_state() as I didn't want
> > > v4l2_subdev_lock_state() to hide an invalid NULL parameter. But adding
> > > WARN_ONCE() would warn about it.
> 
> As we can't WARN() unconditionally yet when encountering a driver that
> hasn't been converted, we need to keep v4l2_subdev_lock_and_return_state()
> as an alternative that won't WARN. Do I understand that adding
> "state = state ? : sd->active_state;" + WARN_ON in
> v4l2_subdev_lock_state() would then be used only to catch invalid
> combinations with a warning instead of a crash ? What would it help with
> ?
> 
> > > I'm still undecided, though. The WARN_ONCE would come only once for the
> > > whole kernel, per boot, wouldn't it? We could have a macro for
> > > v4l2_subdev_lock_state, but then we'd get lots of warnings. And a full WARN
> > > just because a driver hasn't been updated is a bit harsh. Maybe we can start
> > > with just a normal warning print.
> > 
> > There is a precendet I can think of: async matching on device or
> > endpoints. Initially all async subdevs where matched on the remote end
> > device node. For more complex setups this didn't scale, as the same
> > remote can have multiple endpoints, and matching on the device node
> > would have created false positives. So v4l2-async was moved to match on
> > endpoints with some legacy compatibility code
> > 
> > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/media/v4l2-core/v4l2-async.c#n69
> > 
> > 	if (dev && dev->driver) {
> > 		if (sd_fwnode_is_ep)
> > 			dev_warn(dev, "Driver %s uses device fwnode, incorrect match may occur\n",
> > 				 dev->driver->name);
> > 		dev_notice(dev, "Consider updating driver %s to match on endpoints\n",
> > 			   dev->driver->name);
> > 	}
> > 
> > Can't we have something like this in v4l2_subdev_lock_state() ?
> 
> I think we need to start by being silent first, then move to dev_warn(),
> then to WARN(), and finally drop the option. This can be done with
> v4l2_subdev_lock_and_return_state() I believe, which would be used by
> driver that have been converted but still need to support legacy
> combinations. Drivers that don't want to support legacy combinations
> would use v4l2_subdev_lock_state(). Usage of
> v4l2_subdev_lock_and_return_state() in a driver would indicate more work
> is needed, and would be a useful indication of how far we are in our
> conversion effort.

I've discussed this a bit further with Jacopo on IRC. I understand his
concern that silently using the active state when the state is NULL may
cause issues that could be hard to debug. However, I believe we need a
way to handle the transition, without being too harsh (otherwise the
mob will call for a revert, with pitchforks) or too lenient (otherwise
nothing will happen).

One option would be to keep both v4l2_subdev_lock_state() and
v4l2_subdev_lock_and_return_state(), with both functions replacing a
NULL state with subdev->active_state. The former would WARN(), and the
latter would dev_warn(). This would allow a gradual transition from
v4l2_subdev_lock_and_return_state() to v4l2_subdev_lock_state() when
we're confident enough that it should be OK for a particular driver. In
the occasional case where the WARN() would trigger, we could either
revert and fix, or just fix directly (the latter would be my preferred
option, as the incentive for the reporter of the issue to write a fix
would be higher).

I however don't oppose returning NULL from v4l2_subdev_lock_state() when
state is NULL (with a WARN_ON()). The caller would likely crash, but
that's an issue in the caller :-)

-- 
Regards,

Laurent Pinchart

Re: [PATCH v10 06/38] media: Documentation: add documentation about subdev state

[Laurent Pinchart]

Hello,

On Tue, Dec 14, 2021 at 09:31:51AM +0100, Jacopo Mondi wrote:
> On Tue, Nov 30, 2021 at 04:15:04PM +0200, Tomi Valkeinen wrote:
> > Add documentation about centrally managed subdev state.
> >
> > Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > ---
> >  .../driver-api/media/v4l2-subdev.rst          | 28 +++++++++++++++++++
> >  1 file changed, 28 insertions(+)
> >
> > diff --git a/Documentation/driver-api/media/v4l2-subdev.rst b/Documentation/driver-api/media/v4l2-subdev.rst
> > index 7736da077fb8..db012729f8e3 100644
> > --- a/Documentation/driver-api/media/v4l2-subdev.rst
> > +++ b/Documentation/driver-api/media/v4l2-subdev.rst
> > @@ -518,6 +518,34 @@ The :c:func:`v4l2_i2c_new_subdev` function will call
> >  :c:type:`i2c_board_info` structure using the ``client_type`` and the
> >  ``addr`` to fill it.
> >
> > +Centrally managed subdev active state
> > +-------------------------------------
> > +
> > +Traditionally V4L2 subdev drivers maintained internal state for the active
> > +configuration for the subdev. This is often implemented as an array of struct
> > +v4l2_mbus_framefmt, one entry for each pad.
> 
> Not only v4l2_mbus_framefmt, but the 'active' configuration also
> contains cropping rectangles. I would drop the last statement maybe ?
> 
> > +
> > +In addition to the active configuration, each subdev filehandle has an array of
> > +struct v4l2_subdev_pad_config, managed by V4L2 core, which contains the TRY
> > +configuration.
> > +
> > +To simplify the subdev drivers the V4L2 subdev API now optionally supports a
> > +centrally managed active configuration. A subdev driver must use
> 
> I would here explain what active and try states are and where they
> live.
> 
> "a centrally managed active configuration represented by
> :c:type:`v4l2_subdev_state`. One instance of state, which contains
> the active device configuration, is associated with the sub-device itself as
> part of the :c:type:`v4l2_subdev` structure, while the core associates to
> each open file handle a try state, which contains the configuration
> valid in the file-handle context only."
> 
> > +v4l2_subdev_init_finalize() to initialize the active state between calls to
> > +media_entity_pads_init() and v4l2_*_register_subdev(), and must call
> > +v4l2_subdev_cleanup() to free the state.
> 
> "Sub-device drivers can opt-in and use state to manage their active
> configuration by initializing the subdevice state with a call to
> v4l2_subdev_init_finalize() as the last operation before registering
> the sub-device. They must also call v4l2_subdev_cleanup() to release
> all the acquired resources before unregistering the sub-device. The
> core automatically initializes a state for each open file handle where
> to store the try configurations and releases them at file handle
> closing time."
> 
> > +
> > +The active state must be locked before access, and that can be done with
> > +v4l2_subdev_lock_state() or v4l2_subdev_lock_active_state().
> 
> I would also explain which state to use and when.
> 
> "V4L2 sub-device operations that operates on both the :ref:`ACTIVE and

s/operates/operate/

> TRY formats <v4l2-subdev-format-whence>` receive by the core the

s/by the/from the/ (or just s/by the core //)

> correct state to operate on an operation parameter. The sub-device driver
> can access and modify the configuration stored in the provided state
> after having exclusively acquired it by calling

s/exclusively acquired/locked/

> :c:func:`v4l2_subdev_lock_state()` and released it with
> :c:func:`v4l2_subdev_unlock_state()` when done.

"The driver must then call :c:func:`v4l2_subdev_unlock_state()` to
unlock the state when done."

> Operations that do not receive a state parameter implicitly operates
> on the subdevice active state, which drivers can exclusively access by
> calling :c:func:`v4l2_subdev_lock_active_state()`. The sub-device active
> state should equally be released by calling
> :c:func:`v4l2_subdev_unlock_state()`.
> 
> In no occasions driver should try to manually access the state stored
> in the :c:type`v4l2_subdev` or in the file handle without going
> through the designated helpers".
> 
> > +
> > +The V4L2 core will pass either the TRY or ACTIVE state to various subdev ops.
> > +Unfortunately not all the subdev drivers comply with this yet, and may pass
> 
> If I'm not mistaken it's not the subdev drivers but the top drivers
> which might not comply.
> 
> > +NULL for the ACTIVE case. This is only a problem for subdev drivers which use
> > +the centrally managed active state and are used in media pipelines with older
> > +subdev drivers. In these cases the called subdev ops must also handle the NULL
> > +case. This can be easily managed by the use of
> > +v4l2_subdev_validate_and_lock_state() helper.
> 
> depending on the previous suggestion to fail hard if !state, this
> might be removed ?

Or expanded with a note that this option is only possible when
transitioning existing subdev drivers, and that the function will
eventually be removed. And maybe that it prints a warning, and that host
drivers must be converted as a result.

Otherwise this looks good to me.

On a side note, I think we should prepare to allow subdev drivers to
subclass the state structure at some point :-)

> > +
> >  V4L2 sub-device functions and data structures
> >  ---------------------------------------------
> >

-- 
Regards,

Laurent Pinchart

Re: [PATCH v10 28/38] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Laurent Pinchart]

Hi Tomi,

Thank you for the patch.

On Tue, Nov 30, 2021 at 04:15:26PM +0200, Tomi Valkeinen wrote:
> From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> 
> Add support for subdev internal routing. A route is defined as a single
> stream from a sink pad to a source pad.
> 
> The userspace can configure the routing via two new ioctls,
> VIDIOC_SUBDEV_G_ROUTING and VIDIOC_SUBDEV_S_ROUTING, and subdevs can
> implement the functionality with v4l2_subdev_pad_ops.set_routing().
> 
> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Signed-off-by: Michal Simek <michal.simek@xilinx.com>
> 
> - Add sink and source streams for multiplexed links
> - Copy the argument back in case of an error. This is needed to let the
>   caller know the number of routes.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> 
> - Expand and refine documentation.
> - Make the 'routes' pointer a __u64 __user pointer so that a compat32
>   version of the ioctl is not required.
> - Add struct v4l2_subdev_krouting to be used for subdevice operations.
> 
> Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
> 
> - Fix typecasing warnings
> - Check sink & source pad types
> - Add 'which' field
> - Add V4L2_SUBDEV_ROUTE_FL_SOURCE
> - Routing to subdev state
> - Dropped get_routing subdev op
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/v4l2-core/v4l2-ioctl.c  | 25 ++++++++-
>  drivers/media/v4l2-core/v4l2-subdev.c | 75 +++++++++++++++++++++++++++
>  include/media/v4l2-subdev.h           | 22 ++++++++
>  include/uapi/linux/v4l2-subdev.h      | 57 ++++++++++++++++++++
>  4 files changed, 178 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index 05d5db3d85e5..8e9315ffcb99 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -16,6 +16,7 @@
>  #include <linux/kernel.h>
>  #include <linux/version.h>
>  
> +#include <linux/v4l2-subdev.h>
>  #include <linux/videodev2.h>
>  
>  #include <media/v4l2-common.h>
> @@ -3065,6 +3066,21 @@ static int check_array_args(unsigned int cmd, void *parg, size_t *array_size,
>  		ret = 1;
>  		break;
>  	}
> +
> +	case VIDIOC_SUBDEV_G_ROUTING:
> +	case VIDIOC_SUBDEV_S_ROUTING: {
> +		struct v4l2_subdev_routing *routing = parg;
> +
> +		if (routing->num_routes > 256)
> +			return -EINVAL;
> +
> +		*user_ptr = u64_to_user_ptr(routing->routes);
> +		*kernel_ptr = (void **)&routing->routes;
> +		*array_size = sizeof(struct v4l2_subdev_route)
> +			    * routing->num_routes;
> +		ret = 1;
> +		break;
> +	}
>  	}
>  
>  	return ret;
> @@ -3328,8 +3344,15 @@ video_usercopy(struct file *file, unsigned int orig_cmd, unsigned long arg,
>  	/*
>  	 * Some ioctls can return an error, but still have valid
>  	 * results that must be returned.
> +	 *
> +	 * FIXME: subdev IOCTLS are partially handled here and partially in
> +	 * v4l2-subdev.c and the 'always_copy' flag can only be set for IOCTLS
> +	 * defined here as part of the 'v4l2_ioctls' array. As
> +	 * VIDIOC_SUBDEV_G_ROUTING needs to return results to applications even
> +	 * in case of failure, but it is not defined here as part of the
> +	 * 'v4l2_ioctls' array, insert an ad-hoc check to address that.
>  	 */
> -	if (err < 0 && !always_copy)
> +	if (err < 0 && !always_copy && cmd != VIDIOC_SUBDEV_G_ROUTING)
>  		goto out;
>  
>  out_array_args:
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 721148e35624..06e234e0aa98 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -394,6 +394,12 @@ subdev_ioctl_get_state(struct v4l2_subdev *sd, struct v4l2_subdev_fh *subdev_fh,
>  		which = ((struct v4l2_subdev_selection *)arg)->which;
>  		break;
>  	}
> +
> +	case VIDIOC_SUBDEV_G_ROUTING:
> +	case VIDIOC_SUBDEV_S_ROUTING: {
> +		which = ((struct v4l2_subdev_routing *)arg)->which;
> +		break;
> +	}
>  	}
>  
>  	return which == V4L2_SUBDEV_FORMAT_TRY ?
> @@ -712,6 +718,74 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  	case VIDIOC_SUBDEV_QUERYSTD:
>  		return v4l2_subdev_call(sd, video, querystd, arg);
>  
> +	case VIDIOC_SUBDEV_G_ROUTING: {
> +		struct v4l2_subdev_routing *routing = arg;
> +		struct v4l2_subdev_krouting *krouting;
> +
> +		if (!(sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED))
> +			return -ENOIOCTLCMD;
> +
> +		memset(routing->reserved, 0, sizeof(routing->reserved));
> +
> +		krouting = &state->routing;
> +
> +		if (routing->num_routes < krouting->num_routes) {
> +			routing->num_routes = krouting->num_routes;
> +			return -ENOSPC;
> +		}
> +
> +		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
> +		       krouting->routes,
> +		       krouting->num_routes * sizeof(*krouting->routes));
> +		routing->num_routes = krouting->num_routes;
> +
> +		return 0;
> +	}
> +
> +	case VIDIOC_SUBDEV_S_ROUTING: {
> +		struct v4l2_subdev_routing *routing = arg;
> +		struct v4l2_subdev_route *routes =
> +			(struct v4l2_subdev_route *)(uintptr_t)routing->routes;
> +		struct v4l2_subdev_krouting krouting = {};
> +		unsigned int i;
> +
> +		if (!(sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED))
> +			return -ENOIOCTLCMD;
> +
> +		if (routing->which != V4L2_SUBDEV_FORMAT_TRY && ro_subdev)
> +			return -EPERM;
> +
> +		memset(routing->reserved, 0, sizeof(routing->reserved));
> +
> +		for (i = 0; i < routing->num_routes; ++i) {
> +			const struct v4l2_subdev_route *route = &routes[i];
> +			const struct media_pad *pads = sd->entity.pads;
> +
> +			/* Do not check sink pad for source routes */
> +			if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE)) {
> +				if (route->sink_pad >= sd->entity.num_pads)

Could you add dev_dbg() messages in error paths, to help debugging
application code ?

> +					return -EINVAL;
> +
> +				if (!(pads[route->sink_pad].flags &
> +				      MEDIA_PAD_FL_SINK))
> +					return -EINVAL;
> +			}
> +
> +			if (route->source_pad >= sd->entity.num_pads)
> +				return -EINVAL;
> +
> +			if (!(pads[route->source_pad].flags &
> +			      MEDIA_PAD_FL_SOURCE))
> +				return -EINVAL;
> +		}
> +
> +		krouting.num_routes = routing->num_routes;
> +		krouting.routes = routes;
> +
> +		return v4l2_subdev_call(sd, pad, set_routing, state,
> +					routing->which, &krouting);
> +	}
> +
>  	default:
>  		return v4l2_subdev_call(sd, core, ioctl, cmd, arg);
>  	}
> @@ -972,6 +1046,7 @@ void __v4l2_subdev_state_free(struct v4l2_subdev_state *state)
>  
>  	mutex_destroy(&state->lock);
>  
> +	kfree(state->routing.routes);
>  	kvfree(state->pads);
>  	kfree(state);
>  }
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 45861bcdccf5..946455cc1f01 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -697,11 +697,25 @@ struct v4l2_subdev_pad_config {
>  	struct v4l2_rect try_compose;
>  };
>  
> +/**
> + * struct v4l2_subdev_krouting - subdev routing table
> + *
> + * @num_routes: number of routes
> + * @routes: &struct v4l2_subdev_route
> + *
> + * This structure contains the routing table for a subdev.
> + */
> +struct v4l2_subdev_krouting {
> +	unsigned int num_routes;
> +	struct v4l2_subdev_route *routes;
> +};
> +
>  /**
>   * struct v4l2_subdev_state - Used for storing subdev state information.
>   *
>   * @lock: mutex for the state
>   * @pads: &struct v4l2_subdev_pad_config array
> + * @routing: routing table for the subdev
>   *
>   * This structure only needs to be passed to the pad op if the 'which' field
>   * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
> @@ -711,6 +725,7 @@ struct v4l2_subdev_state {
>  	/* lock for the struct v4l2_subdev_state fields */
>  	struct mutex lock;
>  	struct v4l2_subdev_pad_config *pads;
> +	struct v4l2_subdev_krouting routing;
>  };
>  
>  /**
> @@ -774,6 +789,9 @@ struct v4l2_subdev_state {
>   *		     applied to the hardware. The operation shall fail if the
>   *		     pad index it has been called on is not valid or in case of
>   *		     unrecoverable failures.
> + *
> + * @set_routing: enable or disable data connection routes described in the
> + *		 subdevice routing table.
>   */
>  struct v4l2_subdev_pad_ops {
>  	int (*init_cfg)(struct v4l2_subdev *sd,
> @@ -818,6 +836,10 @@ struct v4l2_subdev_pad_ops {
>  			       struct v4l2_mbus_config *config);
>  	int (*set_mbus_config)(struct v4l2_subdev *sd, unsigned int pad,
>  			       struct v4l2_mbus_config *config);
> +	int (*set_routing)(struct v4l2_subdev *sd,
> +			   struct v4l2_subdev_state *state,
> +			   enum v4l2_subdev_format_whence which,
> +			   struct v4l2_subdev_krouting *route);
>  };
>  
>  /**
> diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
> index d91ab6f22fa7..1ec3141bf860 100644
> --- a/include/uapi/linux/v4l2-subdev.h
> +++ b/include/uapi/linux/v4l2-subdev.h
> @@ -191,6 +191,61 @@ struct v4l2_subdev_capability {
>  /* The v4l2 sub-device supports multiplexed streams. */
>  #define V4L2_SUBDEV_CAP_MPLEXED			0x00000002
>  
> +/*
> + * Is the route active? An active route will start when streaming is enabled
> + * on a video node.
> + */
> +#define V4L2_SUBDEV_ROUTE_FL_ACTIVE		_BITUL(0)
> +
> +/*
> + * Is the route immutable, i.e. can it be activated and inactivated?
> + * Set by the driver.
> + */
> +#define V4L2_SUBDEV_ROUTE_FL_IMMUTABLE		_BITUL(1)
> +
> +/*
> + * Is the route a source endpoint? A source endpoint route refers to a stream
> + * generated internally by the subdevice (usually a sensor), and thus there
> + * is no sink-side endpoint for the route. The sink_pad and sink_stream
> + * fields are unused.
> + * Set by the driver.
> + */
> +#define V4L2_SUBDEV_ROUTE_FL_SOURCE		_BITUL(2)
> +
> +/**
> + * struct v4l2_subdev_route - A route inside a subdev
> + *
> + * @sink_pad: the sink pad index
> + * @sink_stream: the sink stream identifier
> + * @source_pad: the source pad index
> + * @source_stream: the source stream identifier
> + * @flags: route flags V4L2_SUBDEV_ROUTE_FL_*
> + * @reserved: drivers and applications must zero this array
> + */
> +struct v4l2_subdev_route {
> +	__u32 sink_pad;
> +	__u32 sink_stream;
> +	__u32 source_pad;
> +	__u32 source_stream;
> +	__u32 flags;
> +	__u32 reserved[5];
> +};
> +
> +/**
> + * struct v4l2_subdev_routing - Subdev routing information
> + *
> + * @which: configuration type (from enum v4l2_subdev_format_whence)
> + * @num_routes: the total number of routes in the routes array
> + * @routes: pointer to the routes array
> + * @reserved: drivers and applications must zero this array
> + */
> +struct v4l2_subdev_routing {
> +	__u32 which;
> +	__u32 num_routes;
> +	__u64 routes;
> +	__u32 reserved[6];
> +};
> +
>  /* Backwards compatibility define --- to be removed */
>  #define v4l2_subdev_edid v4l2_edid
>  
> @@ -206,6 +261,8 @@ struct v4l2_subdev_capability {
>  #define VIDIOC_SUBDEV_S_CROP			_IOWR('V', 60, struct v4l2_subdev_crop)
>  #define VIDIOC_SUBDEV_G_SELECTION		_IOWR('V', 61, struct v4l2_subdev_selection)
>  #define VIDIOC_SUBDEV_S_SELECTION		_IOWR('V', 62, struct v4l2_subdev_selection)
> +#define VIDIOC_SUBDEV_G_ROUTING			_IOWR('V', 38, struct v4l2_subdev_routing)
> +#define VIDIOC_SUBDEV_S_ROUTING			_IOWR('V', 39, struct v4l2_subdev_routing)
>  /* The following ioctls are identical to the ioctls in videodev2.h */
>  #define VIDIOC_SUBDEV_G_STD			_IOR('V', 23, v4l2_std_id)
>  #define VIDIOC_SUBDEV_S_STD			_IOW('V', 24, v4l2_std_id)

-- 
Regards,

Laurent Pinchart

Re: [PATCH v10 02/38] media: subdev: add active state to struct v4l2_subdev

[Tomi Valkeinen]

On 16/12/2021 16:41, Laurent Pinchart wrote:

>>>>>>> 5) is currently unused and it still feels a bit ill-defined. If the state
>>>>>>> passed in as parameter is NULL then deflect it to sd->state, so I
>>>>>>> guess it assumes that the user is a state-aware subdev driver which
>>>>>>> needs to protect against in-kernel callers that do no pass in a valid state to
>>>>>>
>>>>>> Correct.
>>>>>>
>>>>>>> the operation call. This implies that if the caller passes in a NULL
>>>>>>> state for the TRY format, this gets deflected to the subdev's active
>>>>>>> state, something which should not happen, right ? I would be tempted
>>>>>>
>>>>>> Yes. That's an illegal call, isn't it? Or does it happen and we need to
>>>>>> protect against it?
>>>>>>
>>>>>>> to just fail loud if !state and assume if you're moving your subdev to
>>>>>>> use state you should be able to fix the caller as well.
>>>>>>
>>>>>> That would be nice, but I think it's not realistic. If you convert a sensor
>>>>>> driver to multiplexed streams, are you going to fix (convert to multiplexed
>>>>>> streams) all the bridges and SoC drivers that may use that sensor? How do
>>>>>> you even find all those bridges and SoCs...
>>>>>
>>>>> Of course no. You fix the one you're using. You're converting a sensor
>>>>> driver, you can fix the top driver too. Other users of the sensor
>>>>> driver will probably scream when moving to the next driver release
>>>>> that has been made state aware, so they'll hopefully fix their top driver
>>>>> too. After all, this applies equally to downstrems as well and
>>>>
>>>> Well, I'm not a maintainer in linux-media, but I would nack that approach
>>>> =). We can't just go breaking other platforms, and hoping other devs will
>>>> fix them.
>>
>> I'd love to agree with Jacopo here and break things all the time to get
>> other people to fix them, but I doubt that I'd make many friends while
>> doing so :-)
>>
>> This being said, the more we can push for conversions to happen quickly,
>> the better. For instance, new sensor drivers should rely on the active
>> state being passed to all subdev operations, so that any existing host
>> driver that wants to use them will need to be converted.
>> v4l2_subdev_lock_and_return_state() in subdev drivers should only be
>> used as a transition tool.
>>
>> Similarly, a way to enforce that new host drivers only work with sensor
>> drivers that have been converted would be good.
>>
>> We should aim for the removal of v4l2_subdev_lock_and_return_state(). A
>> WARN_ON() there will be useful at some point, but it's of course too
>> early. A bit of yak shaving may also help, by asking maintainers and
>> contributors to sensor and host drivers to migrate to the new API.
>>
>>>>> providing an helper to work around issues is not the best idea in my
>>>>> opinion. Also the helper should be used in the subdev driver in place
>>>>> of the regular v4l2_subdev_lock_state() to protect just in case
>>>>> against legacy callers. When will they be moved to use the regular
>>>>> v4l2_subdev_lock_state() ?
>>>>
>>>> Note that this is needed only when porting an existing and presumably in-use
>>>> subdev driver. You don't need this if you write a new driver.
>>>>
>>>> The users of v4l2_subdev_lock_and_return_state are easy to find and easy to
>>>> change to v4l2_subdev_lock_state when we decide a particular driver doesn't
>>>> need to support legacy subdevs.
>>>
>>> I'm just concerned that as long as we offer an helper to work this
>>> around (and the helper might introduce subtle issues like mixing
>>> try/active context) once we
>>> s/v4l2_subdev_lock_and_return_state/v4l2_subdev_lock_state we'll be
>>> anyway breaking users.
>>>
>>>> I don't like this at all but, afaics, we can't break the existing platforms.
>>>> This function is a very simple work-around for the time being, and easy to
>>>> drop later.
>>>>
>>>>> Once a subdev driver has been moved to be state aware callers that
>>>>> passes in a NULL state should be fixed. As we can't fix them all,
>>>>> screaming loud and clearly say what has to be done to move forward is
>>>>> in my opinion better than introducing a temporary function that
>>>>> replaces the regular API and that subdev should decide to use just in
>>>>> case (and which can lead to subtle errors like using the active state
>>>>> in a TRY context).
>>>>>
>>>>> If you want to protect against broken callers then what about
>>>>> doing the "state = state ? : sd->active_state;" dance in
>>>>> v4l2_subdev_lock_state() with a WARN_ONCE(!state) so that
>>>>> subdev drivers can use the regular API from day 0 ?
>>>>
>>>> Hmm, I think that is an option. I didn't implement "state = state ? :
>>>> sd->active_state;" in the v4l2_subdev_lock_state() as I didn't want
>>>> v4l2_subdev_lock_state() to hide an invalid NULL parameter. But adding
>>>> WARN_ONCE() would warn about it.
>>
>> As we can't WARN() unconditionally yet when encountering a driver that
>> hasn't been converted, we need to keep v4l2_subdev_lock_and_return_state()
>> as an alternative that won't WARN. Do I understand that adding
>> "state = state ? : sd->active_state;" + WARN_ON in
>> v4l2_subdev_lock_state() would then be used only to catch invalid
>> combinations with a warning instead of a crash ? What would it help with
>> ?
>>
>>>> I'm still undecided, though. The WARN_ONCE would come only once for the
>>>> whole kernel, per boot, wouldn't it? We could have a macro for
>>>> v4l2_subdev_lock_state, but then we'd get lots of warnings. And a full WARN
>>>> just because a driver hasn't been updated is a bit harsh. Maybe we can start
>>>> with just a normal warning print.
>>>
>>> There is a precendet I can think of: async matching on device or
>>> endpoints. Initially all async subdevs where matched on the remote end
>>> device node. For more complex setups this didn't scale, as the same
>>> remote can have multiple endpoints, and matching on the device node
>>> would have created false positives. So v4l2-async was moved to match on
>>> endpoints with some legacy compatibility code
>>>
>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/media/v4l2-core/v4l2-async.c#n69
>>>
>>> 	if (dev && dev->driver) {
>>> 		if (sd_fwnode_is_ep)
>>> 			dev_warn(dev, "Driver %s uses device fwnode, incorrect match may occur\n",
>>> 				 dev->driver->name);
>>> 		dev_notice(dev, "Consider updating driver %s to match on endpoints\n",
>>> 			   dev->driver->name);
>>> 	}
>>>
>>> Can't we have something like this in v4l2_subdev_lock_state() ?
>>
>> I think we need to start by being silent first, then move to dev_warn(),
>> then to WARN(), and finally drop the option. This can be done with
>> v4l2_subdev_lock_and_return_state() I believe, which would be used by
>> driver that have been converted but still need to support legacy
>> combinations. Drivers that don't want to support legacy combinations
>> would use v4l2_subdev_lock_state(). Usage of
>> v4l2_subdev_lock_and_return_state() in a driver would indicate more work
>> is needed, and would be a useful indication of how far we are in our
>> conversion effort.
> 
> I've discussed this a bit further with Jacopo on IRC. I understand his
> concern that silently using the active state when the state is NULL may
> cause issues that could be hard to debug. However, I believe we need a
> way to handle the transition, without being too harsh (otherwise the
> mob will call for a revert, with pitchforks) or too lenient (otherwise
> nothing will happen).
> 
> One option would be to keep both v4l2_subdev_lock_state() and
> v4l2_subdev_lock_and_return_state(), with both functions replacing a
> NULL state with subdev->active_state. The former would WARN(), and the
> latter would dev_warn(). This would allow a gradual transition from
> v4l2_subdev_lock_and_return_state() to v4l2_subdev_lock_state() when
> we're confident enough that it should be OK for a particular driver. In
> the occasional case where the WARN() would trigger, we could either
> revert and fix, or just fix directly (the latter would be my preferred
> option, as the incentive for the reporter of the issue to write a fix
> would be higher).
> 
> I however don't oppose returning NULL from v4l2_subdev_lock_state() when
> state is NULL (with a WARN_ON()). The caller would likely crash, but
> that's an issue in the caller :-)

Note that v4l2_subdev_lock_state() doesn't return a value. We can't use 
that function to pick up the active state for the caller if the passed 
state is NULL.

We could change v4l2_subdev_lock_state() to return a value, or we could 
change v4l2_subdev_lock_state() to a macro and do magics there, but I 
think both options are very bad.

I think v4l2_subdev_lock_and_return_state() is the best way forward.

I can add a dev_warn there, though.

  Tomi

Re: [PATCH v10 06/38] media: Documentation: add documentation about subdev state

[Tomi Valkeinen]

On 14/12/2021 10:31, Jacopo Mondi wrote:
> Hi Tomi,
> 
> On Tue, Nov 30, 2021 at 04:15:04PM +0200, Tomi Valkeinen wrote:
>> Add documentation about centrally managed subdev state.
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>>   .../driver-api/media/v4l2-subdev.rst          | 28 +++++++++++++++++++
>>   1 file changed, 28 insertions(+)
>>
>> diff --git a/Documentation/driver-api/media/v4l2-subdev.rst b/Documentation/driver-api/media/v4l2-subdev.rst
>> index 7736da077fb8..db012729f8e3 100644
>> --- a/Documentation/driver-api/media/v4l2-subdev.rst
>> +++ b/Documentation/driver-api/media/v4l2-subdev.rst
>> @@ -518,6 +518,34 @@ The :c:func:`v4l2_i2c_new_subdev` function will call
>>   :c:type:`i2c_board_info` structure using the ``client_type`` and the
>>   ``addr`` to fill it.
>>
>> +Centrally managed subdev active state
>> +-------------------------------------
>> +
>> +Traditionally V4L2 subdev drivers maintained internal state for the active
>> +configuration for the subdev. This is often implemented as an array of struct
>> +v4l2_mbus_framefmt, one entry for each pad.
> 
> Not only v4l2_mbus_framefmt, but the 'active' configuration also
> contains cropping rectangles. I would drop the last statement maybe ?
> 
>> +
>> +In addition to the active configuration, each subdev filehandle has an array of
>> +struct v4l2_subdev_pad_config, managed by V4L2 core, which contains the TRY
>> +configuration.
>> +
>> +To simplify the subdev drivers the V4L2 subdev API now optionally supports a
>> +centrally managed active configuration. A subdev driver must use
> 
> I would here explain what active and try states are and where they
> live.
> 
> "a centrally managed active configuration represented by
> :c:type:`v4l2_subdev_state`. One instance of state, which contains
> the active device configuration, is associated with the sub-device itself as
> part of the :c:type:`v4l2_subdev` structure, while the core associates to
> each open file handle a try state, which contains the configuration
> valid in the file-handle context only."
> 
>> +v4l2_subdev_init_finalize() to initialize the active state between calls to
>> +media_entity_pads_init() and v4l2_*_register_subdev(), and must call
>> +v4l2_subdev_cleanup() to free the state.
> 
> "Sub-device drivers can opt-in and use state to manage their active
> configuration by initializing the subdevice state with a call to
> v4l2_subdev_init_finalize() as the last operation before registering
> the sub-device. They must also call v4l2_subdev_cleanup() to release
> all the acquired resources before unregistering the sub-device. The
> core automatically initializes a state for each open file handle where
> to store the try configurations and releases them at file handle
> closing time."
> 
>> +
>> +The active state must be locked before access, and that can be done with
>> +v4l2_subdev_lock_state() or v4l2_subdev_lock_active_state().
> 
> I would also explain which state to use and when.
> 
> "V4L2 sub-device operations that operates on both the :ref:`ACTIVE and
> TRY formats <v4l2-subdev-format-whence>` receive by the core the
> correct state to operate on an operation parameter. The sub-device driver
> can access and modify the configuration stored in the provided state
> after having exclusively acquired it by calling
> :c:func:`v4l2_subdev_lock_state()` and released it with
> :c:func:`v4l2_subdev_unlock_state()` when done.
> 
> Operations that do not receive a state parameter implicitly operates
> on the subdevice active state, which drivers can exclusively access by
> calling :c:func:`v4l2_subdev_lock_active_state()`. The sub-device active
> state should equally be released by calling
> :c:func:`v4l2_subdev_unlock_state()`.
> 
> In no occasions driver should try to manually access the state stored
> in the :c:type`v4l2_subdev` or in the file handle without going
> through the designated helpers".
> 
>> +
>> +The V4L2 core will pass either the TRY or ACTIVE state to various subdev ops.
>> +Unfortunately not all the subdev drivers comply with this yet, and may pass
> 
> If I'm not mistaken it's not the subdev drivers but the top drivers
> which might not comply.

No, it's for any callers of subdev ops (core, top-drivers, subdev 
drivers). I'll change it to "not all callers comply".

Thanks Jacopo and Laurent. I like this method of writing docs!

  Tomi

Re: [PATCH v10 02/38] media: subdev: add active state to struct v4l2_subdev

[Hans Verkuil]

On 30/11/2021 15:15, Tomi Valkeinen wrote:
> Add a new 'active_state' field to struct v4l2_subdev to which we can
> store the active state of a subdev. This will place the subdev
> configuration into a known place, allowing us to use the state directly
> from the v4l2 framework, thus simplifying the drivers.
> 
> Also add functions v4l2_subdev_init_finalize() and
> v4l2_subdev_cleanup(), which will allocate and free the active state.
> The functions are named in a generic way so that they can be also used
> for other subdev initialization work.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>

Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>

Regards,

	Hans

> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 21 +++++++++++
>  include/media/v4l2-subdev.h           | 54 +++++++++++++++++++++++++++
>  2 files changed, 75 insertions(+)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index fe49c86a9b02..de160140d63b 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -943,3 +943,24 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
>  	v4l2_subdev_notify(sd, V4L2_DEVICE_NOTIFY_EVENT, (void *)ev);
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
> +
> +int v4l2_subdev_init_finalize(struct v4l2_subdev *sd)
> +{
> +	struct v4l2_subdev_state *state;
> +
> +	state = __v4l2_subdev_state_alloc(sd);
> +	if (IS_ERR(state))
> +		return PTR_ERR(state);
> +
> +	sd->active_state = state;
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_init_finalize);
> +
> +void v4l2_subdev_cleanup(struct v4l2_subdev *sd)
> +{
> +	__v4l2_subdev_state_free(sd->active_state);
> +	sd->active_state = NULL;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_cleanup);
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index e52bf508c75b..87c3126b2754 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -645,6 +645,9 @@ struct v4l2_subdev_ir_ops {
>   * This structure only needs to be passed to the pad op if the 'which' field
>   * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
>   * %V4L2_SUBDEV_FORMAT_ACTIVE it is safe to pass %NULL.
> + *
> + * Note: This struct is also used in active state, and the try_ prefix is
> + * historical and to be removed.
>   */
>  struct v4l2_subdev_pad_config {
>  	struct v4l2_mbus_framefmt try_fmt;
> @@ -898,6 +901,8 @@ struct v4l2_subdev_platform_data {
>   * @subdev_notifier: A sub-device notifier implicitly registered for the sub-
>   *		     device using v4l2_async_register_subdev_sensor().
>   * @pdata: common part of subdevice platform data
> + * @active_state: active state for the subdev (NULL for subdevs tracking the
> + *                state internally)
>   *
>   * Each instance of a subdev driver should create this struct, either
>   * stand-alone or embedded in a larger struct.
> @@ -929,6 +934,19 @@ struct v4l2_subdev {
>  	struct v4l2_async_notifier *notifier;
>  	struct v4l2_async_notifier *subdev_notifier;
>  	struct v4l2_subdev_platform_data *pdata;
> +
> +	/*
> +	 * The fields below are private, and should only be accessed via
> +	 * appropriate functions.
> +	 */
> +
> +	/*
> +	 * TODO: state should most likely be changed from a pointer to an
> +	 * embedded field. For the time being it's kept as a pointer to more
> +	 * easily catch uses of state in the cases where the driver doesn't
> +	 * support it.
> +	 */
> +	struct v4l2_subdev_state *active_state;
>  };
>  
>  
> @@ -1217,4 +1235,40 @@ extern const struct v4l2_subdev_ops v4l2_subdev_call_wrappers;
>  void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
>  			      const struct v4l2_event *ev);
>  
> +/**
> + * v4l2_subdev_init_finalize() - Finalize the initialization of the subdevice
> + * @sd: The subdev
> + *
> + * This finalizes the initialization of the subdev, including allocation of
> + * the active state for the subdev.
> + *
> + * This must be called by the subdev drivers that use the centralized active
> + * state, after the subdev struct has been initialized and
> + * media_entity_pads_init() has been called.
> + *
> + * Must call v4l2_subdev_cleanup() when the subdev is being removed.
> + */
> +int v4l2_subdev_init_finalize(struct v4l2_subdev *sd);
> +
> +/**
> + * v4l2_subdev_cleanup() - Release the resources needed by the subdevice
> + * @sd: The subdevice
> + *
> + * This will release the resources allocated in v4l2_subdev_init_finalize.
> + */
> +void v4l2_subdev_cleanup(struct v4l2_subdev *sd);
> +
> +/**
> + * v4l2_subdev_get_active_state() - Return the active subdev state for subdevice
> + * @sd: The subdevice
> + *
> + * Return the active state for the subdevice, or NULL if the subdev does not
> + * support active state.
> + */
> +static inline struct v4l2_subdev_state *
> +v4l2_subdev_get_active_state(struct v4l2_subdev *sd)
> +{
> +	return sd->active_state;
> +}
> +
>  #endif
> 

Re: [PATCH v10 04/38] media: subdev: add subdev state locking

[Hans Verkuil]

On 30/11/2021 15:15, Tomi Valkeinen wrote:
> The V4L2 subdevs have managed without centralized locking for the state
> (previously pad_config), as the TRY state is supposedly safe (although I
> believe two TRY ioctls for the same fd would race), and the ACTIVE
> state, and its locking, is managed by the drivers internally.
> 
> We now have ACTIVE state in a centralized position, and need locking.
> Strictly speaking the locking is only needed for new drivers that use
> the new state, as the current drivers continue behaving as they used to.
> 
> Add a mutex to the struct v4l2_subdev_state, along with a few helper
> functions for locking/unlocking.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>

Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>

Regards,

	Hans

> ---
>  drivers/media/platform/rcar-vin/rcar-v4l2.c |  3 +-
>  drivers/media/platform/vsp1/vsp1_entity.c   |  4 +-
>  drivers/media/v4l2-core/v4l2-subdev.c       | 38 +++++++++++++---
>  drivers/staging/media/tegra-video/vi.c      |  4 +-
>  include/media/v4l2-subdev.h                 | 50 ++++++++++++++++++++-
>  5 files changed, 89 insertions(+), 10 deletions(-)
> 
> diff --git a/drivers/media/platform/rcar-vin/rcar-v4l2.c b/drivers/media/platform/rcar-vin/rcar-v4l2.c
> index ba1d16ab1651..e6bd94d63e4f 100644
> --- a/drivers/media/platform/rcar-vin/rcar-v4l2.c
> +++ b/drivers/media/platform/rcar-vin/rcar-v4l2.c
> @@ -244,6 +244,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
>  {
>  	struct v4l2_subdev *sd = vin_to_source(vin);
>  	struct v4l2_subdev_state *sd_state;
> +	static struct lock_class_key key;
>  	struct v4l2_subdev_format format = {
>  		.which = which,
>  		.pad = vin->parallel.source_pad,
> @@ -252,7 +253,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
>  	u32 width, height;
>  	int ret;
>  
> -	sd_state = __v4l2_subdev_state_alloc(sd);
> +	sd_state = __v4l2_subdev_state_alloc(sd, "rvin:state->lock", &key);
>  	if (IS_ERR(sd_state))
>  		return PTR_ERR(sd_state);
>  
> diff --git a/drivers/media/platform/vsp1/vsp1_entity.c b/drivers/media/platform/vsp1/vsp1_entity.c
> index 869cadc1468d..e607c3ae2520 100644
> --- a/drivers/media/platform/vsp1/vsp1_entity.c
> +++ b/drivers/media/platform/vsp1/vsp1_entity.c
> @@ -613,6 +613,7 @@ int vsp1_entity_init(struct vsp1_device *vsp1, struct vsp1_entity *entity,
>  		     const char *name, unsigned int num_pads,
>  		     const struct v4l2_subdev_ops *ops, u32 function)
>  {
> +	static struct lock_class_key key;
>  	struct v4l2_subdev *subdev;
>  	unsigned int i;
>  	int ret;
> @@ -675,7 +676,8 @@ int vsp1_entity_init(struct vsp1_device *vsp1, struct vsp1_entity *entity,
>  	 * Allocate the pad configuration to store formats and selection
>  	 * rectangles.
>  	 */
> -	entity->config = __v4l2_subdev_state_alloc(&entity->subdev);
> +	entity->config = __v4l2_subdev_state_alloc(&entity->subdev,
> +						   "vsp1:config->lock", &key);
>  	if (IS_ERR(entity->config)) {
>  		media_entity_cleanup(&entity->subdev.entity);
>  		return PTR_ERR(entity->config);
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 3289875d9ec1..2053fe1cd67d 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -27,8 +27,9 @@
>  static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
>  {
>  	struct v4l2_subdev_state *state;
> +	static struct lock_class_key key;
>  
> -	state = __v4l2_subdev_state_alloc(sd);
> +	state = __v4l2_subdev_state_alloc(sd, "fh->state->lock", &key);
>  	if (IS_ERR(state))
>  		return PTR_ERR(state);
>  
> @@ -923,7 +924,9 @@ int v4l2_subdev_link_validate(struct media_link *link)
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
>  
> -struct v4l2_subdev_state *__v4l2_subdev_state_alloc(struct v4l2_subdev *sd)
> +struct v4l2_subdev_state *
> +__v4l2_subdev_state_alloc(struct v4l2_subdev *sd, const char *lock_name,
> +			  struct lock_class_key *lock_key)
>  {
>  	struct v4l2_subdev_state *state;
>  	int ret;
> @@ -932,6 +935,8 @@ struct v4l2_subdev_state *__v4l2_subdev_state_alloc(struct v4l2_subdev *sd)
>  	if (!state)
>  		return ERR_PTR(-ENOMEM);
>  
> +	__mutex_init(&state->lock, lock_name, lock_key);
> +
>  	if (sd->entity.num_pads) {
>  		state->pads = kvmalloc_array(sd->entity.num_pads,
>  					     sizeof(*state->pads),
> @@ -963,6 +968,8 @@ void __v4l2_subdev_state_free(struct v4l2_subdev_state *state)
>  	if (!state)
>  		return;
>  
> +	mutex_destroy(&state->lock);
> +
>  	kvfree(state->pads);
>  	kfree(state);
>  }
> @@ -997,11 +1004,12 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
>  
> -int v4l2_subdev_init_finalize(struct v4l2_subdev *sd)
> +int __v4l2_subdev_init_finalize(struct v4l2_subdev *sd, const char *name,
> +				struct lock_class_key *key)
>  {
>  	struct v4l2_subdev_state *state;
>  
> -	state = __v4l2_subdev_state_alloc(sd);
> +	state = __v4l2_subdev_state_alloc(sd, name, key);
>  	if (IS_ERR(state))
>  		return PTR_ERR(state);
>  
> @@ -1009,7 +1017,7 @@ int v4l2_subdev_init_finalize(struct v4l2_subdev *sd)
>  
>  	return 0;
>  }
> -EXPORT_SYMBOL_GPL(v4l2_subdev_init_finalize);
> +EXPORT_SYMBOL_GPL(__v4l2_subdev_init_finalize);
>  
>  void v4l2_subdev_cleanup(struct v4l2_subdev *sd)
>  {
> @@ -1017,3 +1025,23 @@ void v4l2_subdev_cleanup(struct v4l2_subdev *sd)
>  	sd->active_state = NULL;
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_cleanup);
> +
> +struct v4l2_subdev_state *v4l2_subdev_lock_active_state(struct v4l2_subdev *sd)
> +{
> +	mutex_lock(&sd->active_state->lock);
> +
> +	return sd->active_state;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_lock_active_state);
> +
> +void v4l2_subdev_lock_state(struct v4l2_subdev_state *state)
> +{
> +	mutex_lock(&state->lock);
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_lock_state);
> +
> +void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state)
> +{
> +	mutex_unlock(&state->lock);
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_unlock_state);
> diff --git a/drivers/staging/media/tegra-video/vi.c b/drivers/staging/media/tegra-video/vi.c
> index 66b9ce160472..3516735f0cad 100644
> --- a/drivers/staging/media/tegra-video/vi.c
> +++ b/drivers/staging/media/tegra-video/vi.c
> @@ -491,6 +491,7 @@ static int __tegra_channel_try_format(struct tegra_vi_channel *chan,
>  				      struct v4l2_pix_format *pix)
>  {
>  	const struct tegra_video_format *fmtinfo;
> +	static struct lock_class_key key;
>  	struct v4l2_subdev *subdev;
>  	struct v4l2_subdev_format fmt;
>  	struct v4l2_subdev_state *sd_state;
> @@ -507,7 +508,8 @@ static int __tegra_channel_try_format(struct tegra_vi_channel *chan,
>  	if (!subdev)
>  		return -ENODEV;
>  
> -	sd_state = __v4l2_subdev_state_alloc(subdev);
> +	sd_state = __v4l2_subdev_state_alloc(subdev, "tegra:state->lock",
> +					     &key);
>  	if (IS_ERR(sd_state))
>  		return PTR_ERR(sd_state);
>  	/*
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 87c3126b2754..1810dde9c7fc 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -658,6 +658,7 @@ struct v4l2_subdev_pad_config {
>  /**
>   * struct v4l2_subdev_state - Used for storing subdev state information.
>   *
> + * @lock: mutex for the state
>   * @pads: &struct v4l2_subdev_pad_config array
>   *
>   * This structure only needs to be passed to the pad op if the 'which' field
> @@ -665,6 +666,8 @@ struct v4l2_subdev_pad_config {
>   * %V4L2_SUBDEV_FORMAT_ACTIVE it is safe to pass %NULL.
>   */
>  struct v4l2_subdev_state {
> +	/* lock for the struct v4l2_subdev_state fields */
> +	struct mutex lock;
>  	struct v4l2_subdev_pad_config *pads;
>  };
>  
> @@ -1156,10 +1159,14 @@ int v4l2_subdev_link_validate(struct media_link *link);
>   * __v4l2_subdev_state_alloc - allocate v4l2_subdev_state
>   *
>   * @sd: pointer to &struct v4l2_subdev for which the state is being allocated.
> + * @lock_name: name of the state lock
> + * @key: lock_class_key for the lock
>   *
>   * Must call __v4l2_subdev_state_free() when state is no longer needed.
>   */
> -struct v4l2_subdev_state *__v4l2_subdev_state_alloc(struct v4l2_subdev *sd);
> +struct v4l2_subdev_state *__v4l2_subdev_state_alloc(struct v4l2_subdev *sd,
> +						    const char *lock_name,
> +						    struct lock_class_key *key);
>  
>  /**
>   * __v4l2_subdev_state_free - free a v4l2_subdev_state
> @@ -1248,7 +1255,16 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
>   *
>   * Must call v4l2_subdev_cleanup() when the subdev is being removed.
>   */
> -int v4l2_subdev_init_finalize(struct v4l2_subdev *sd);
> +#define v4l2_subdev_init_finalize(sd)                                          \
> +	({                                                                     \
> +		static struct lock_class_key __key;                            \
> +		const char *name = KBUILD_BASENAME                             \
> +			":" __stringify(__LINE__) ":subdev->state->lock";      \
> +		__v4l2_subdev_init_finalize(sd, name, &__key);                 \
> +	})
> +
> +int __v4l2_subdev_init_finalize(struct v4l2_subdev *sd, const char *name,
> +				struct lock_class_key *key);
>  
>  /**
>   * v4l2_subdev_cleanup() - Release the resources needed by the subdevice
> @@ -1271,4 +1287,34 @@ v4l2_subdev_get_active_state(struct v4l2_subdev *sd)
>  	return sd->active_state;
>  }
>  
> +/**
> + * v4l2_subdev_lock_active_state() - Lock and return the active subdev state for
> + *				     subdevice
> + * @sd: The subdevice
> + *
> + * Return the locked active state for the subdevice, or NULL if the subdev
> + * does not support active state.
> + *
> + * Must be unlocked with v4l2_subdev_unlock_state() after use.
> + */
> +struct v4l2_subdev_state *v4l2_subdev_lock_active_state(struct v4l2_subdev *sd);
> +
> +/**
> + * v4l2_subdev_lock_state() - Lock the subdev state
> + * @state: The subdevice state
> + *
> + * Lock the given subdev state.
> + *
> + * Must be unlocked with v4l2_subdev_unlock_state() after use.
> + */
> +void v4l2_subdev_lock_state(struct v4l2_subdev_state *state);
> +
> +/**
> + * v4l2_subdev_unlock_state() - Unlock the subdev state
> + * @state: The subdevice state
> + *
> + * Unlock the given subdev state.
> + */
> +void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state);
> +
>  #endif
> 

Re: [PATCH v10 05/38] media: subdev: Add v4l2_subdev_lock_and_return_state()

[Hans Verkuil]

On 30/11/2021 15:15, Tomi Valkeinen wrote:
> All suitable subdev ops are now passed either the TRY or the ACTIVE
> state by the v4l2 core. However, other subdev drivers can still call the
> ops passing NULL as the state, implying the active case.
> 
> For all current upstream drivers this doesn't matter, as they do not
> expect to get a valid state for ACTIVE case. But future drivers which
> support multiplexed streaming and routing will depend on getting a state
> for both active and try cases.
> 
> For new drivers we can mandate that the pipelines where the drivers are
> used need to pass the state properly, or preferably, not call such
> subdev ops at all.
> 
> However, if an existing subdev driver is changed to support multiplexed
> streams, the driver has to consider cases where its ops will be called
> with NULL state. The problem can easily be solved by using the
> v4l2_subdev_lock_and_return_state() helper, introduced here.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  include/media/v4l2-subdev.h | 31 +++++++++++++++++++++++++++++++
>  1 file changed, 31 insertions(+)
> 
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 1810dde9c7fc..873bbe0686e3 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -1317,4 +1317,35 @@ void v4l2_subdev_lock_state(struct v4l2_subdev_state *state);
>   */
>  void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state);
>  
> +/**
> + * v4l2_subdev_lock_and_return_state() - Gets locked TRY or ACTIVE subdev state
> + * @sd: subdevice
> + * @state: subdevice state as passed to the subdev op
> + *
> + * Due to legacy reasons, when subdev drivers call ops in other subdevs they use
> + * NULL as the state parameter, as subdevs always used to have their active
> + * state stored privately.
> + *
> + * However, newer state-aware subdev drivers, which store their active state in
> + * a common place, subdev->active_state, expect to always get a proper state as
> + * a parameter.
> + *
> + * These state-aware drivers can use v4l2_subdev_lock_and_return_state() instead
> + * of v4l2_subdev_lock_state(). v4l2_subdev_lock_and_return_state() solves the
> + * issue by using subdev->state in case the passed state is NULL.

Should be: by using subdev->active_state

With that change:

Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>

Regards,

	hans

> + *
> + * This is a temporary helper function, and should be removed when we can ensure
> + * that all drivers pass proper state when calling other subdevs.
> + */
> +static inline struct v4l2_subdev_state *
> +v4l2_subdev_lock_and_return_state(struct v4l2_subdev *sd,
> +				  struct v4l2_subdev_state *state)
> +{
> +	state = state ? state : sd->active_state;
> +
> +	v4l2_subdev_lock_state(state);
> +
> +	return state;
> +}
> +
>  #endif
> 

Re: [PATCH v10 05/38] media: subdev: Add v4l2_subdev_lock_and_return_state()

[Sakari Ailus]

Hi Laurent,

On Thu, Dec 16, 2021 at 04:34:22PM +0200, Laurent Pinchart wrote:
> Hi Tomi,
> 
> Thank you for the patch.
> 
> On Tue, Nov 30, 2021 at 04:15:03PM +0200, Tomi Valkeinen wrote:
> > All suitable subdev ops are now passed either the TRY or the ACTIVE
> > state by the v4l2 core. However, other subdev drivers can still call the
> > ops passing NULL as the state, implying the active case.
> > 
> > For all current upstream drivers this doesn't matter, as they do not
> > expect to get a valid state for ACTIVE case. But future drivers which
> > support multiplexed streaming and routing will depend on getting a state
> > for both active and try cases.
> > 
> > For new drivers we can mandate that the pipelines where the drivers are
> > used need to pass the state properly, or preferably, not call such
> > subdev ops at all.
> > 
> > However, if an existing subdev driver is changed to support multiplexed
> > streams, the driver has to consider cases where its ops will be called
> > with NULL state. The problem can easily be solved by using the
> > v4l2_subdev_lock_and_return_state() helper, introduced here.
> > 
> > Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > ---
> >  include/media/v4l2-subdev.h | 31 +++++++++++++++++++++++++++++++
> >  1 file changed, 31 insertions(+)
> > 
> > diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> > index 1810dde9c7fc..873bbe0686e3 100644
> > --- a/include/media/v4l2-subdev.h
> > +++ b/include/media/v4l2-subdev.h
> > @@ -1317,4 +1317,35 @@ void v4l2_subdev_lock_state(struct v4l2_subdev_state *state);
> >   */
> >  void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state);
> >  
> > +/**
> > + * v4l2_subdev_lock_and_return_state() - Gets locked TRY or ACTIVE subdev state
> > + * @sd: subdevice
> > + * @state: subdevice state as passed to the subdev op
> > + *
> > + * Due to legacy reasons, when subdev drivers call ops in other subdevs they use
> > + * NULL as the state parameter, as subdevs always used to have their active
> > + * state stored privately.
> > + *
> > + * However, newer state-aware subdev drivers, which store their active state in
> > + * a common place, subdev->active_state, expect to always get a proper state as
> > + * a parameter.
> > + *
> > + * These state-aware drivers can use v4l2_subdev_lock_and_return_state() instead
> > + * of v4l2_subdev_lock_state(). v4l2_subdev_lock_and_return_state() solves the
> > + * issue by using subdev->state in case the passed state is NULL.
> > + *
> > + * This is a temporary helper function, and should be removed when we can ensure
> > + * that all drivers pass proper state when calling other subdevs.
> > + */
> > +static inline struct v4l2_subdev_state *
> > +v4l2_subdev_lock_and_return_state(struct v4l2_subdev *sd,
> > +				  struct v4l2_subdev_state *state)
> > +{
> > +	state = state ? state : sd->active_state;
> 
> Can we add a dev_warn() when state is NULL ? This will help speeding up
> the transition.

Wouldn't this produce lots of warnings? I'd rather use dev_warn_once() to
avoid flooding logs.

> 
> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> 
> > +
> > +	v4l2_subdev_lock_state(state);
> > +
> > +	return state;
> > +}
> > +
> >  #endif
> 

-- 
Sakari Ailus

Re: [PATCH v10 05/38] media: subdev: Add v4l2_subdev_lock_and_return_state()

[Laurent Pinchart]

Hi Sakari,

On Fri, Dec 17, 2021 at 07:43:55PM +0200, Sakari Ailus wrote:
> On Thu, Dec 16, 2021 at 04:34:22PM +0200, Laurent Pinchart wrote:
> > On Tue, Nov 30, 2021 at 04:15:03PM +0200, Tomi Valkeinen wrote:
> > > All suitable subdev ops are now passed either the TRY or the ACTIVE
> > > state by the v4l2 core. However, other subdev drivers can still call the
> > > ops passing NULL as the state, implying the active case.
> > > 
> > > For all current upstream drivers this doesn't matter, as they do not
> > > expect to get a valid state for ACTIVE case. But future drivers which
> > > support multiplexed streaming and routing will depend on getting a state
> > > for both active and try cases.
> > > 
> > > For new drivers we can mandate that the pipelines where the drivers are
> > > used need to pass the state properly, or preferably, not call such
> > > subdev ops at all.
> > > 
> > > However, if an existing subdev driver is changed to support multiplexed
> > > streams, the driver has to consider cases where its ops will be called
> > > with NULL state. The problem can easily be solved by using the
> > > v4l2_subdev_lock_and_return_state() helper, introduced here.
> > > 
> > > Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > > ---
> > >  include/media/v4l2-subdev.h | 31 +++++++++++++++++++++++++++++++
> > >  1 file changed, 31 insertions(+)
> > > 
> > > diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> > > index 1810dde9c7fc..873bbe0686e3 100644
> > > --- a/include/media/v4l2-subdev.h
> > > +++ b/include/media/v4l2-subdev.h
> > > @@ -1317,4 +1317,35 @@ void v4l2_subdev_lock_state(struct v4l2_subdev_state *state);
> > >   */
> > >  void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state);
> > >  
> > > +/**
> > > + * v4l2_subdev_lock_and_return_state() - Gets locked TRY or ACTIVE subdev state
> > > + * @sd: subdevice
> > > + * @state: subdevice state as passed to the subdev op
> > > + *
> > > + * Due to legacy reasons, when subdev drivers call ops in other subdevs they use
> > > + * NULL as the state parameter, as subdevs always used to have their active
> > > + * state stored privately.
> > > + *
> > > + * However, newer state-aware subdev drivers, which store their active state in
> > > + * a common place, subdev->active_state, expect to always get a proper state as
> > > + * a parameter.
> > > + *
> > > + * These state-aware drivers can use v4l2_subdev_lock_and_return_state() instead
> > > + * of v4l2_subdev_lock_state(). v4l2_subdev_lock_and_return_state() solves the
> > > + * issue by using subdev->state in case the passed state is NULL.
> > > + *
> > > + * This is a temporary helper function, and should be removed when we can ensure
> > > + * that all drivers pass proper state when calling other subdevs.
> > > + */
> > > +static inline struct v4l2_subdev_state *
> > > +v4l2_subdev_lock_and_return_state(struct v4l2_subdev *sd,
> > > +				  struct v4l2_subdev_state *state)
> > > +{
> > > +	state = state ? state : sd->active_state;
> > 
> > Can we add a dev_warn() when state is NULL ? This will help speeding up
> > the transition.
> 
> Wouldn't this produce lots of warnings? I'd rather use dev_warn_once() to
> avoid flooding logs.

The goal is to notice the issue, to get it fixed, so I'd prefer a few
warnings instead of a dev_warn_once().

Please note that the first 6 patches from this series have been posted
in a new version as "[PATCH v2 0/6] v4l: subdev active state"
(20211217135022.364954-1-tomi.valkeinen@ideasonboard.com).

> > Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > 
> > > +
> > > +	v4l2_subdev_lock_state(state);
> > > +
> > > +	return state;
> > > +}
> > > +
> > >  #endif

-- 
Regards,

Laurent Pinchart

Re: [PATCH v10 05/38] media: subdev: Add v4l2_subdev_lock_and_return_state()

[Laurent Pinchart]

Hi Hans,

On Fri, Dec 17, 2021 at 12:48:56PM +0100, Hans Verkuil wrote:
> On 30/11/2021 15:15, Tomi Valkeinen wrote:
> > All suitable subdev ops are now passed either the TRY or the ACTIVE
> > state by the v4l2 core. However, other subdev drivers can still call the
> > ops passing NULL as the state, implying the active case.
> > 
> > For all current upstream drivers this doesn't matter, as they do not
> > expect to get a valid state for ACTIVE case. But future drivers which
> > support multiplexed streaming and routing will depend on getting a state
> > for both active and try cases.
> > 
> > For new drivers we can mandate that the pipelines where the drivers are
> > used need to pass the state properly, or preferably, not call such
> > subdev ops at all.
> > 
> > However, if an existing subdev driver is changed to support multiplexed
> > streams, the driver has to consider cases where its ops will be called
> > with NULL state. The problem can easily be solved by using the
> > v4l2_subdev_lock_and_return_state() helper, introduced here.
> > 
> > Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > ---
> >  include/media/v4l2-subdev.h | 31 +++++++++++++++++++++++++++++++
> >  1 file changed, 31 insertions(+)
> > 
> > diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> > index 1810dde9c7fc..873bbe0686e3 100644
> > --- a/include/media/v4l2-subdev.h
> > +++ b/include/media/v4l2-subdev.h
> > @@ -1317,4 +1317,35 @@ void v4l2_subdev_lock_state(struct v4l2_subdev_state *state);
> >   */
> >  void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state);
> >  
> > +/**
> > + * v4l2_subdev_lock_and_return_state() - Gets locked TRY or ACTIVE subdev state
> > + * @sd: subdevice
> > + * @state: subdevice state as passed to the subdev op
> > + *
> > + * Due to legacy reasons, when subdev drivers call ops in other subdevs they use
> > + * NULL as the state parameter, as subdevs always used to have their active
> > + * state stored privately.
> > + *
> > + * However, newer state-aware subdev drivers, which store their active state in
> > + * a common place, subdev->active_state, expect to always get a proper state as
> > + * a parameter.
> > + *
> > + * These state-aware drivers can use v4l2_subdev_lock_and_return_state() instead
> > + * of v4l2_subdev_lock_state(). v4l2_subdev_lock_and_return_state() solves the
> > + * issue by using subdev->state in case the passed state is NULL.
> 
> Should be: by using subdev->active_state
> 
> With that change:
> 
> Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>

Please note that the first 6 patches from this series have been posted
in a new version as "[PATCH v2 0/6] v4l: subdev active state"
(20211217135022.364954-1-tomi.valkeinen@ideasonboard.com).

> > + *
> > + * This is a temporary helper function, and should be removed when we can ensure
> > + * that all drivers pass proper state when calling other subdevs.
> > + */
> > +static inline struct v4l2_subdev_state *
> > +v4l2_subdev_lock_and_return_state(struct v4l2_subdev *sd,
> > +				  struct v4l2_subdev_state *state)
> > +{
> > +	state = state ? state : sd->active_state;
> > +
> > +	v4l2_subdev_lock_state(state);
> > +
> > +	return state;
> > +}
> > +
> >  #endif
> > 

-- 
Regards,

Laurent Pinchart

Re: [PATCH v10 05/38] media: subdev: Add v4l2_subdev_lock_and_return_state()

[Sakari Ailus]

Hi Laurent,

On Fri, Dec 17, 2021 at 07:54:38PM +0200, Laurent Pinchart wrote:
> Hi Sakari,
> 
> On Fri, Dec 17, 2021 at 07:43:55PM +0200, Sakari Ailus wrote:
> > On Thu, Dec 16, 2021 at 04:34:22PM +0200, Laurent Pinchart wrote:
> > > On Tue, Nov 30, 2021 at 04:15:03PM +0200, Tomi Valkeinen wrote:
> > > > All suitable subdev ops are now passed either the TRY or the ACTIVE
> > > > state by the v4l2 core. However, other subdev drivers can still call the
> > > > ops passing NULL as the state, implying the active case.
> > > > 
> > > > For all current upstream drivers this doesn't matter, as they do not
> > > > expect to get a valid state for ACTIVE case. But future drivers which
> > > > support multiplexed streaming and routing will depend on getting a state
> > > > for both active and try cases.
> > > > 
> > > > For new drivers we can mandate that the pipelines where the drivers are
> > > > used need to pass the state properly, or preferably, not call such
> > > > subdev ops at all.
> > > > 
> > > > However, if an existing subdev driver is changed to support multiplexed
> > > > streams, the driver has to consider cases where its ops will be called
> > > > with NULL state. The problem can easily be solved by using the
> > > > v4l2_subdev_lock_and_return_state() helper, introduced here.
> > > > 
> > > > Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > > > ---
> > > >  include/media/v4l2-subdev.h | 31 +++++++++++++++++++++++++++++++
> > > >  1 file changed, 31 insertions(+)
> > > > 
> > > > diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> > > > index 1810dde9c7fc..873bbe0686e3 100644
> > > > --- a/include/media/v4l2-subdev.h
> > > > +++ b/include/media/v4l2-subdev.h
> > > > @@ -1317,4 +1317,35 @@ void v4l2_subdev_lock_state(struct v4l2_subdev_state *state);
> > > >   */
> > > >  void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state);
> > > >  
> > > > +/**
> > > > + * v4l2_subdev_lock_and_return_state() - Gets locked TRY or ACTIVE subdev state
> > > > + * @sd: subdevice
> > > > + * @state: subdevice state as passed to the subdev op
> > > > + *
> > > > + * Due to legacy reasons, when subdev drivers call ops in other subdevs they use
> > > > + * NULL as the state parameter, as subdevs always used to have their active
> > > > + * state stored privately.
> > > > + *
> > > > + * However, newer state-aware subdev drivers, which store their active state in
> > > > + * a common place, subdev->active_state, expect to always get a proper state as
> > > > + * a parameter.
> > > > + *
> > > > + * These state-aware drivers can use v4l2_subdev_lock_and_return_state() instead
> > > > + * of v4l2_subdev_lock_state(). v4l2_subdev_lock_and_return_state() solves the
> > > > + * issue by using subdev->state in case the passed state is NULL.
> > > > + *
> > > > + * This is a temporary helper function, and should be removed when we can ensure
> > > > + * that all drivers pass proper state when calling other subdevs.
> > > > + */
> > > > +static inline struct v4l2_subdev_state *
> > > > +v4l2_subdev_lock_and_return_state(struct v4l2_subdev *sd,
> > > > +				  struct v4l2_subdev_state *state)
> > > > +{
> > > > +	state = state ? state : sd->active_state;
> > > 
> > > Can we add a dev_warn() when state is NULL ? This will help speeding up
> > > the transition.
> > 
> > Wouldn't this produce lots of warnings? I'd rather use dev_warn_once() to
> > avoid flooding logs.
> 
> The goal is to notice the issue, to get it fixed, so I'd prefer a few
> warnings instead of a dev_warn_once().

This wouldn't be a few, but quite possibly system logs full of such lines
unless some kind of limiting factor is in place.

> 
> Please note that the first 6 patches from this series have been posted
> in a new version as "[PATCH v2 0/6] v4l: subdev active state"
> (20211217135022.364954-1-tomi.valkeinen@ideasonboard.com).

Yes, I see this is addressed in v2.

-- 
Sakari Ailus

Re: [PATCH v10 28/38] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Dave Stevenson]

Hi Tomi

On Tue, 30 Nov 2021 at 14:17, Tomi Valkeinen
<tomi.valkeinen@ideasonboard.com> wrote:
>
> From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
>
> Add support for subdev internal routing. A route is defined as a single
> stream from a sink pad to a source pad.
>
> The userspace can configure the routing via two new ioctls,
> VIDIOC_SUBDEV_G_ROUTING and VIDIOC_SUBDEV_S_ROUTING, and subdevs can
> implement the functionality with v4l2_subdev_pad_ops.set_routing().

Minor point on naming. We already have an s_routing function in and
struct v4l2_subdev_audio_ops and struct v4l2_subdev_video_ops.
Having a new IOCTL that actually doesn't map onto the function of the
same name caused me a few minutes of head scratching, and I suspect it
will for others too.

  Dave

> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Signed-off-by: Michal Simek <michal.simek@xilinx.com>
>
> - Add sink and source streams for multiplexed links
> - Copy the argument back in case of an error. This is needed to let the
>   caller know the number of routes.
>
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
>
> - Expand and refine documentation.
> - Make the 'routes' pointer a __u64 __user pointer so that a compat32
>   version of the ioctl is not required.
> - Add struct v4l2_subdev_krouting to be used for subdevice operations.
>
> Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
>
> - Fix typecasing warnings
> - Check sink & source pad types
> - Add 'which' field
> - Add V4L2_SUBDEV_ROUTE_FL_SOURCE
> - Routing to subdev state
> - Dropped get_routing subdev op
>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/v4l2-core/v4l2-ioctl.c  | 25 ++++++++-
>  drivers/media/v4l2-core/v4l2-subdev.c | 75 +++++++++++++++++++++++++++
>  include/media/v4l2-subdev.h           | 22 ++++++++
>  include/uapi/linux/v4l2-subdev.h      | 57 ++++++++++++++++++++
>  4 files changed, 178 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index 05d5db3d85e5..8e9315ffcb99 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -16,6 +16,7 @@
>  #include <linux/kernel.h>
>  #include <linux/version.h>
>
> +#include <linux/v4l2-subdev.h>
>  #include <linux/videodev2.h>
>
>  #include <media/v4l2-common.h>
> @@ -3065,6 +3066,21 @@ static int check_array_args(unsigned int cmd, void *parg, size_t *array_size,
>                 ret = 1;
>                 break;
>         }
> +
> +       case VIDIOC_SUBDEV_G_ROUTING:
> +       case VIDIOC_SUBDEV_S_ROUTING: {
> +               struct v4l2_subdev_routing *routing = parg;
> +
> +               if (routing->num_routes > 256)
> +                       return -EINVAL;
> +
> +               *user_ptr = u64_to_user_ptr(routing->routes);
> +               *kernel_ptr = (void **)&routing->routes;
> +               *array_size = sizeof(struct v4l2_subdev_route)
> +                           * routing->num_routes;
> +               ret = 1;
> +               break;
> +       }
>         }
>
>         return ret;
> @@ -3328,8 +3344,15 @@ video_usercopy(struct file *file, unsigned int orig_cmd, unsigned long arg,
>         /*
>          * Some ioctls can return an error, but still have valid
>          * results that must be returned.
> +        *
> +        * FIXME: subdev IOCTLS are partially handled here and partially in
> +        * v4l2-subdev.c and the 'always_copy' flag can only be set for IOCTLS
> +        * defined here as part of the 'v4l2_ioctls' array. As
> +        * VIDIOC_SUBDEV_G_ROUTING needs to return results to applications even
> +        * in case of failure, but it is not defined here as part of the
> +        * 'v4l2_ioctls' array, insert an ad-hoc check to address that.
>          */
> -       if (err < 0 && !always_copy)
> +       if (err < 0 && !always_copy && cmd != VIDIOC_SUBDEV_G_ROUTING)
>                 goto out;
>
>  out_array_args:
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 721148e35624..06e234e0aa98 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -394,6 +394,12 @@ subdev_ioctl_get_state(struct v4l2_subdev *sd, struct v4l2_subdev_fh *subdev_fh,
>                 which = ((struct v4l2_subdev_selection *)arg)->which;
>                 break;
>         }
> +
> +       case VIDIOC_SUBDEV_G_ROUTING:
> +       case VIDIOC_SUBDEV_S_ROUTING: {
> +               which = ((struct v4l2_subdev_routing *)arg)->which;
> +               break;
> +       }
>         }
>
>         return which == V4L2_SUBDEV_FORMAT_TRY ?
> @@ -712,6 +718,74 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>         case VIDIOC_SUBDEV_QUERYSTD:
>                 return v4l2_subdev_call(sd, video, querystd, arg);
>
> +       case VIDIOC_SUBDEV_G_ROUTING: {
> +               struct v4l2_subdev_routing *routing = arg;
> +               struct v4l2_subdev_krouting *krouting;
> +
> +               if (!(sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED))
> +                       return -ENOIOCTLCMD;
> +
> +               memset(routing->reserved, 0, sizeof(routing->reserved));
> +
> +               krouting = &state->routing;
> +
> +               if (routing->num_routes < krouting->num_routes) {
> +                       routing->num_routes = krouting->num_routes;
> +                       return -ENOSPC;
> +               }
> +
> +               memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
> +                      krouting->routes,
> +                      krouting->num_routes * sizeof(*krouting->routes));
> +               routing->num_routes = krouting->num_routes;
> +
> +               return 0;
> +       }
> +
> +       case VIDIOC_SUBDEV_S_ROUTING: {
> +               struct v4l2_subdev_routing *routing = arg;
> +               struct v4l2_subdev_route *routes =
> +                       (struct v4l2_subdev_route *)(uintptr_t)routing->routes;
> +               struct v4l2_subdev_krouting krouting = {};
> +               unsigned int i;
> +
> +               if (!(sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED))
> +                       return -ENOIOCTLCMD;
> +
> +               if (routing->which != V4L2_SUBDEV_FORMAT_TRY && ro_subdev)
> +                       return -EPERM;
> +
> +               memset(routing->reserved, 0, sizeof(routing->reserved));
> +
> +               for (i = 0; i < routing->num_routes; ++i) {
> +                       const struct v4l2_subdev_route *route = &routes[i];
> +                       const struct media_pad *pads = sd->entity.pads;
> +
> +                       /* Do not check sink pad for source routes */
> +                       if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE)) {
> +                               if (route->sink_pad >= sd->entity.num_pads)
> +                                       return -EINVAL;
> +
> +                               if (!(pads[route->sink_pad].flags &
> +                                     MEDIA_PAD_FL_SINK))
> +                                       return -EINVAL;
> +                       }
> +
> +                       if (route->source_pad >= sd->entity.num_pads)
> +                               return -EINVAL;
> +
> +                       if (!(pads[route->source_pad].flags &
> +                             MEDIA_PAD_FL_SOURCE))
> +                               return -EINVAL;
> +               }
> +
> +               krouting.num_routes = routing->num_routes;
> +               krouting.routes = routes;
> +
> +               return v4l2_subdev_call(sd, pad, set_routing, state,
> +                                       routing->which, &krouting);
> +       }
> +
>         default:
>                 return v4l2_subdev_call(sd, core, ioctl, cmd, arg);
>         }
> @@ -972,6 +1046,7 @@ void __v4l2_subdev_state_free(struct v4l2_subdev_state *state)
>
>         mutex_destroy(&state->lock);
>
> +       kfree(state->routing.routes);
>         kvfree(state->pads);
>         kfree(state);
>  }
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 45861bcdccf5..946455cc1f01 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -697,11 +697,25 @@ struct v4l2_subdev_pad_config {
>         struct v4l2_rect try_compose;
>  };
>
> +/**
> + * struct v4l2_subdev_krouting - subdev routing table
> + *
> + * @num_routes: number of routes
> + * @routes: &struct v4l2_subdev_route
> + *
> + * This structure contains the routing table for a subdev.
> + */
> +struct v4l2_subdev_krouting {
> +       unsigned int num_routes;
> +       struct v4l2_subdev_route *routes;
> +};
> +
>  /**
>   * struct v4l2_subdev_state - Used for storing subdev state information.
>   *
>   * @lock: mutex for the state
>   * @pads: &struct v4l2_subdev_pad_config array
> + * @routing: routing table for the subdev
>   *
>   * This structure only needs to be passed to the pad op if the 'which' field
>   * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
> @@ -711,6 +725,7 @@ struct v4l2_subdev_state {
>         /* lock for the struct v4l2_subdev_state fields */
>         struct mutex lock;
>         struct v4l2_subdev_pad_config *pads;
> +       struct v4l2_subdev_krouting routing;
>  };
>
>  /**
> @@ -774,6 +789,9 @@ struct v4l2_subdev_state {
>   *                  applied to the hardware. The operation shall fail if the
>   *                  pad index it has been called on is not valid or in case of
>   *                  unrecoverable failures.
> + *
> + * @set_routing: enable or disable data connection routes described in the
> + *              subdevice routing table.
>   */
>  struct v4l2_subdev_pad_ops {
>         int (*init_cfg)(struct v4l2_subdev *sd,
> @@ -818,6 +836,10 @@ struct v4l2_subdev_pad_ops {
>                                struct v4l2_mbus_config *config);
>         int (*set_mbus_config)(struct v4l2_subdev *sd, unsigned int pad,
>                                struct v4l2_mbus_config *config);
> +       int (*set_routing)(struct v4l2_subdev *sd,
> +                          struct v4l2_subdev_state *state,
> +                          enum v4l2_subdev_format_whence which,
> +                          struct v4l2_subdev_krouting *route);
>  };
>
>  /**
> diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
> index d91ab6f22fa7..1ec3141bf860 100644
> --- a/include/uapi/linux/v4l2-subdev.h
> +++ b/include/uapi/linux/v4l2-subdev.h
> @@ -191,6 +191,61 @@ struct v4l2_subdev_capability {
>  /* The v4l2 sub-device supports multiplexed streams. */
>  #define V4L2_SUBDEV_CAP_MPLEXED                        0x00000002
>
> +/*
> + * Is the route active? An active route will start when streaming is enabled
> + * on a video node.
> + */
> +#define V4L2_SUBDEV_ROUTE_FL_ACTIVE            _BITUL(0)
> +
> +/*
> + * Is the route immutable, i.e. can it be activated and inactivated?
> + * Set by the driver.
> + */
> +#define V4L2_SUBDEV_ROUTE_FL_IMMUTABLE         _BITUL(1)
> +
> +/*
> + * Is the route a source endpoint? A source endpoint route refers to a stream
> + * generated internally by the subdevice (usually a sensor), and thus there
> + * is no sink-side endpoint for the route. The sink_pad and sink_stream
> + * fields are unused.
> + * Set by the driver.
> + */
> +#define V4L2_SUBDEV_ROUTE_FL_SOURCE            _BITUL(2)
> +
> +/**
> + * struct v4l2_subdev_route - A route inside a subdev
> + *
> + * @sink_pad: the sink pad index
> + * @sink_stream: the sink stream identifier
> + * @source_pad: the source pad index
> + * @source_stream: the source stream identifier
> + * @flags: route flags V4L2_SUBDEV_ROUTE_FL_*
> + * @reserved: drivers and applications must zero this array
> + */
> +struct v4l2_subdev_route {
> +       __u32 sink_pad;
> +       __u32 sink_stream;
> +       __u32 source_pad;
> +       __u32 source_stream;
> +       __u32 flags;
> +       __u32 reserved[5];
> +};
> +
> +/**
> + * struct v4l2_subdev_routing - Subdev routing information
> + *
> + * @which: configuration type (from enum v4l2_subdev_format_whence)
> + * @num_routes: the total number of routes in the routes array
> + * @routes: pointer to the routes array
> + * @reserved: drivers and applications must zero this array
> + */
> +struct v4l2_subdev_routing {
> +       __u32 which;
> +       __u32 num_routes;
> +       __u64 routes;
> +       __u32 reserved[6];
> +};
> +
>  /* Backwards compatibility define --- to be removed */
>  #define v4l2_subdev_edid v4l2_edid
>
> @@ -206,6 +261,8 @@ struct v4l2_subdev_capability {
>  #define VIDIOC_SUBDEV_S_CROP                   _IOWR('V', 60, struct v4l2_subdev_crop)
>  #define VIDIOC_SUBDEV_G_SELECTION              _IOWR('V', 61, struct v4l2_subdev_selection)
>  #define VIDIOC_SUBDEV_S_SELECTION              _IOWR('V', 62, struct v4l2_subdev_selection)
> +#define VIDIOC_SUBDEV_G_ROUTING                        _IOWR('V', 38, struct v4l2_subdev_routing)
> +#define VIDIOC_SUBDEV_S_ROUTING                        _IOWR('V', 39, struct v4l2_subdev_routing)
>  /* The following ioctls are identical to the ioctls in videodev2.h */
>  #define VIDIOC_SUBDEV_G_STD                    _IOR('V', 23, v4l2_std_id)
>  #define VIDIOC_SUBDEV_S_STD                    _IOW('V', 24, v4l2_std_id)
> --
> 2.25.1
>

Re: [PATCH v10 26/38] media: add V4L2_SUBDEV_CAP_MPLEXED

[Laurent Pinchart]

On Tue, Dec 14, 2021 at 09:39:50AM +0100, Jacopo Mondi wrote:
> On Tue, Nov 30, 2021 at 04:15:24PM +0200, Tomi Valkeinen wrote:
> > Add a subdev capability flag to expose to userspace if a subdev supports
> > multiplexed streams.
> >
> > Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > ---
> >  drivers/media/v4l2-core/v4l2-subdev.c | 4 +++-
> >  include/uapi/linux/v4l2-subdev.h      | 3 +++
> >  2 files changed, 6 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> > index 2053fe1cd67d..721148e35624 100644
> > --- a/drivers/media/v4l2-core/v4l2-subdev.c
> > +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> > @@ -419,7 +419,9 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
> >
> >  		memset(cap->reserved, 0, sizeof(cap->reserved));
> >  		cap->version = LINUX_VERSION_CODE;
> > -		cap->capabilities = ro_subdev ? V4L2_SUBDEV_CAP_RO_SUBDEV : 0;
> > +		cap->capabilities =
> > +			(ro_subdev ? V4L2_SUBDEV_CAP_RO_SUBDEV : 0) |
> > +			((sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED) ? V4L2_SUBDEV_CAP_MPLEXED : 0);
> 
> I had been suggested to go for:
> 
> 	bool ro_subdev = test_bit(V4L2_FL_SUBDEV_RO_DEVNODE, &vdev->flags);
> 
> when introducing V4L2_SUBDEV_CAP_RO_SUBDEV.
> 
> To me it doesn't make much difference
> 
> Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
> 
> >
> >  		return 0;
> >  	}
> > diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
> > index 658106f5b5dc..d91ab6f22fa7 100644
> > --- a/include/uapi/linux/v4l2-subdev.h
> > +++ b/include/uapi/linux/v4l2-subdev.h
> > @@ -188,6 +188,9 @@ struct v4l2_subdev_capability {
> >  /* The v4l2 sub-device video device node is registered in read-only mode. */
> >  #define V4L2_SUBDEV_CAP_RO_SUBDEV		0x00000001
> >
> > +/* The v4l2 sub-device supports multiplexed streams. */
> > +#define V4L2_SUBDEV_CAP_MPLEXED			0x00000002

We have V4L2_SUBDEV_FL_MULTIPLEXED vs. V4L2_SUBDEV_CAP_MPLEXED, can we
try to standardize naming ? V4L2 is notoriously bad in this area
(VIDIOC_ENUM_FMT vs. VIDIOC_ENUMSTD, v4l2_fmtdesc vs. v4l2_format, ...).
It would be nice to avoid repeating the same mistakes all the time, my
OCD is suffering :-( (and this also really hinders development, when one
has to always remember which abbreviation has been used in a given
context).

> > +
> >  /* Backwards compatibility define --- to be removed */
> >  #define v4l2_subdev_edid v4l2_edid
> >

-- 
Regards,

Laurent Pinchart

Re: [PATCH v10 12/38] media: entity: Move the pipeline from entity to pads

[Laurent Pinchart]

Hi Tomi and Sakari,

Thank you for the patch.

On Tue, Nov 30, 2021 at 04:15:10PM +0200, Tomi Valkeinen wrote:
> From: Sakari Ailus <sakari.ailus@linux.intel.com>
> 
> This moves the pipe and stream_count fields from struct media_entity to
> struct media_pad. Effectively streams become pad-specific rather than
> being entity specific, allowing several independent streams to traverse a
> single entity and an entity to be part of several streams.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
> 
> - Update documentation to use 'pads'
> - Use the media pad iterator in media_entity.c
> - Update rcar-dma.c to use the new per-pad stream count
> Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
> 
> - Fix cleanup in the error path of __media_pipeline_start()
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> ---
>  drivers/media/mc/mc-entity.c                  | 68 +++++++++++--------
>  drivers/media/platform/exynos4-is/fimc-isp.c  |  2 +-
>  drivers/media/platform/exynos4-is/fimc-lite.c |  2 +-
>  drivers/media/platform/omap3isp/isp.c         |  2 +-
>  drivers/media/platform/omap3isp/ispvideo.c    |  2 +-
>  drivers/media/platform/omap3isp/ispvideo.h    |  2 +-
>  drivers/media/platform/rcar-vin/rcar-core.c   | 16 +++--
>  drivers/media/platform/rcar-vin/rcar-dma.c    |  2 +-
>  drivers/media/platform/xilinx/xilinx-dma.c    |  2 +-
>  drivers/media/platform/xilinx/xilinx-dma.h    |  2 +-
>  drivers/staging/media/imx/imx-media-utils.c   |  2 +-
>  drivers/staging/media/omap4iss/iss.c          |  2 +-
>  drivers/staging/media/omap4iss/iss_video.c    |  2 +-
>  drivers/staging/media/omap4iss/iss_video.h    |  2 +-
>  include/media/media-entity.h                  | 21 +++---
>  15 files changed, 74 insertions(+), 55 deletions(-)
> 
> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> index 2b438c481812..8ad4cb845f4a 100644
> --- a/drivers/media/mc/mc-entity.c
> +++ b/drivers/media/mc/mc-entity.c
> @@ -424,24 +424,30 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
>  
>  	while ((pad = media_graph_walk_next(graph))) {
>  		struct media_entity *entity = pad->entity;
> +		bool skip_validation = pad->pipe != NULL;
> +		struct media_pad *iter;
>  
>  		DECLARE_BITMAP(active, MEDIA_ENTITY_MAX_PADS);
>  		DECLARE_BITMAP(has_no_links, MEDIA_ENTITY_MAX_PADS);
>  
> -		entity->stream_count++;
> +		ret = 0;
>  
> -		if (entity->pipe && entity->pipe != pipe) {
> -			pr_err("Pipe active for %s. Can't start for %s\n",
> -				entity->name,
> -				pad_err->entity->name);
> -			ret = -EBUSY;
> -			goto error;
> +		media_entity_for_each_pad(entity, iter) {
> +			if (iter->pipe && iter->pipe != pipe) {
> +				pr_err("Pipe active for %s. Can't start for %s\n",
> +				       entity->name, iter->entity->name);

'iter' is a pad of 'entity'. As 'entity' is set to 'pad->entity',
'entity' and 'iter->entity' are the same. The message should be
reworked.

> +				ret = -EBUSY;
> +			} else {
> +				iter->pipe = pipe;
> +			}
> +			iter->stream_count++;
>  		}
>  
> -		entity->pipe = pipe;
> +		if (ret)
> +			goto error;
>  
> -		/* Already streaming --- no need to check. */
> -		if (entity->stream_count > 1)
> +		/* Already part of the pipeline, skip validation. */
> +		if (skip_validation)
>  			continue;
>  
>  		if (!entity->ops || !entity->ops->link_validate)
> @@ -510,20 +516,23 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
>  	media_graph_walk_start(graph, pad_err);
>  
>  	while ((pad_err = media_graph_walk_next(graph))) {
> -		struct media_entity *entity_err = pad_err->entity;
> -
> -		/* Sanity check for negative stream_count */
> -		if (!WARN_ON_ONCE(entity_err->stream_count <= 0)) {
> -			entity_err->stream_count--;
> -			if (entity_err->stream_count == 0)
> -				entity_err->pipe = NULL;
> +		struct media_entity *entity = pad_err->entity;
> +		struct media_pad *iter;
> +
> +		media_entity_for_each_pad(entity, iter) {
> +			/* Sanity check for negative stream_count */
> +			if (!WARN_ON_ONCE(iter->stream_count <= 0)) {
> +				--iter->stream_count;
> +				if (iter->stream_count == 0)
> +					iter->pipe = NULL;
> +			}
>  		}
>  
>  		/*
>  		 * We haven't increased stream_count further than this
>  		 * so we quit here.
>  		 */
> -		if (pad_err == pad)
> +		if (pad_err->entity == pad->entity)
>  			break;
>  	}
>  
> @@ -550,7 +559,7 @@ EXPORT_SYMBOL_GPL(media_pipeline_start);
>  
>  void __media_pipeline_stop(struct media_entity *entity)
>  {
> -	struct media_pipeline *pipe = entity->pipe;
> +	struct media_pipeline *pipe = entity->pads->pipe;
>  	struct media_graph *graph = &pipe->graph;
>  	struct media_pad *pad;
>  
> @@ -565,12 +574,15 @@ void __media_pipeline_stop(struct media_entity *entity)
>  
>  	while ((pad = media_graph_walk_next(graph))) {
>  		struct media_entity *entity = pad->entity;
> -
> -		/* Sanity check for negative stream_count */
> -		if (!WARN_ON_ONCE(entity->stream_count <= 0)) {
> -			entity->stream_count--;
> -			if (entity->stream_count == 0)
> -				entity->pipe = NULL;
> +		struct media_pad *iter;
> +
> +		media_entity_for_each_pad(entity, iter) {
> +			/* Sanity check for negative stream_count */
> +			if (!WARN_ON_ONCE(iter->stream_count <= 0)) {
> +				iter->stream_count--;
> +				if (iter->stream_count == 0)
> +					iter->pipe = NULL;
> +			}
>  		}
>  	}
>  
> @@ -840,7 +852,7 @@ int __media_entity_setup_link(struct media_link *link, u32 flags)
>  {
>  	const u32 mask = MEDIA_LNK_FL_ENABLED;
>  	struct media_device *mdev;
> -	struct media_entity *source, *sink;
> +	struct media_pad *source, *sink;
>  	int ret = -EBUSY;
>  
>  	if (link == NULL)
> @@ -856,8 +868,8 @@ int __media_entity_setup_link(struct media_link *link, u32 flags)
>  	if (link->flags == flags)
>  		return 0;
>  
> -	source = link->source->entity;
> -	sink = link->sink->entity;
> +	source = link->source;
> +	sink = link->sink;
>  
>  	if (!(link->flags & MEDIA_LNK_FL_DYNAMIC) &&
>  	    (source->stream_count || sink->stream_count))
> diff --git a/drivers/media/platform/exynos4-is/fimc-isp.c b/drivers/media/platform/exynos4-is/fimc-isp.c
> index 855235bea46d..80274e29ccc5 100644
> --- a/drivers/media/platform/exynos4-is/fimc-isp.c
> +++ b/drivers/media/platform/exynos4-is/fimc-isp.c
> @@ -226,7 +226,7 @@ static int fimc_isp_subdev_set_fmt(struct v4l2_subdev *sd,
>  			}
>  		}
>  	} else {
> -		if (sd->entity.stream_count == 0) {
> +		if (sd->entity.pads->stream_count == 0) {
>  			if (fmt->pad == FIMC_ISP_SD_PAD_SINK) {
>  				struct v4l2_subdev_format format = *fmt;
>  
> diff --git a/drivers/media/platform/exynos4-is/fimc-lite.c b/drivers/media/platform/exynos4-is/fimc-lite.c
> index aaa3af0493ce..67bfb1ad2ba2 100644
> --- a/drivers/media/platform/exynos4-is/fimc-lite.c
> +++ b/drivers/media/platform/exynos4-is/fimc-lite.c
> @@ -1073,7 +1073,7 @@ static int fimc_lite_subdev_set_fmt(struct v4l2_subdev *sd,
>  	mutex_lock(&fimc->lock);
>  
>  	if ((atomic_read(&fimc->out_path) == FIMC_IO_ISP &&
> -	    sd->entity.stream_count > 0) ||
> +	    sd->entity.pads->stream_count > 0) ||
>  	    (atomic_read(&fimc->out_path) == FIMC_IO_DMA &&
>  	    vb2_is_busy(&fimc->vb_queue))) {
>  		mutex_unlock(&fimc->lock);
> diff --git a/drivers/media/platform/omap3isp/isp.c b/drivers/media/platform/omap3isp/isp.c
> index 20f59c59ff8a..28aab16d7662 100644
> --- a/drivers/media/platform/omap3isp/isp.c
> +++ b/drivers/media/platform/omap3isp/isp.c
> @@ -936,7 +936,7 @@ static int isp_pipeline_is_last(struct media_entity *me)
>  	struct isp_pipeline *pipe;
>  	struct media_pad *pad;
>  
> -	if (!me->pipe)
> +	if (!me->pads->pipe)
>  		return 0;
>  	pipe = to_isp_pipeline(me);
>  	if (pipe->stream_state == ISP_PIPELINE_STREAM_STOPPED)
> diff --git a/drivers/media/platform/omap3isp/ispvideo.c b/drivers/media/platform/omap3isp/ispvideo.c
> index 5c1cbb1a9003..a8438040c4aa 100644
> --- a/drivers/media/platform/omap3isp/ispvideo.c
> +++ b/drivers/media/platform/omap3isp/ispvideo.c
> @@ -1094,7 +1094,7 @@ isp_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
>  	/* Start streaming on the pipeline. No link touching an entity in the
>  	 * pipeline can be activated or deactivated once streaming is started.
>  	 */
> -	pipe = video->video.entity.pipe
> +	pipe = video->video.entity.pads->pipe
>  	     ? to_isp_pipeline(&video->video.entity) : &video->pipe;
>  
>  	ret = media_entity_enum_init(&pipe->ent_enum, &video->isp->media_dev);
> diff --git a/drivers/media/platform/omap3isp/ispvideo.h b/drivers/media/platform/omap3isp/ispvideo.h
> index a0908670c0cf..4c9c5b719ec5 100644
> --- a/drivers/media/platform/omap3isp/ispvideo.h
> +++ b/drivers/media/platform/omap3isp/ispvideo.h
> @@ -100,7 +100,7 @@ struct isp_pipeline {
>  };
>  
>  #define to_isp_pipeline(__e) \
> -	container_of((__e)->pipe, struct isp_pipeline, pipe)
> +	container_of((__e)->pads->pipe, struct isp_pipeline, pipe)
>  
>  static inline int isp_pipeline_ready(struct isp_pipeline *pipe)
>  {
> diff --git a/drivers/media/platform/rcar-vin/rcar-core.c b/drivers/media/platform/rcar-vin/rcar-core.c
> index 33957cc9118c..e59453d6b7c3 100644
> --- a/drivers/media/platform/rcar-vin/rcar-core.c
> +++ b/drivers/media/platform/rcar-vin/rcar-core.c
> @@ -132,13 +132,17 @@ static int rvin_group_link_notify(struct media_link *link, u32 flags,
>  		return 0;
>  
>  	/*
> -	 * Don't allow link changes if any entity in the graph is
> -	 * streaming, modifying the CHSEL register fields can disrupt
> -	 * running streams.
> +	 * Don't allow link changes if any stream in the graph is active as
> +	 * modifying the CHSEL register fields can disrupt running streams.
>  	 */
> -	media_device_for_each_entity(entity, &group->mdev)
> -		if (entity->stream_count)
> -			return -EBUSY;
> +	media_device_for_each_entity(entity, &group->mdev) {
> +		struct media_pad *iter;
> +
> +		media_entity_for_each_pad(entity, iter) {
> +			if (iter->stream_count)
> +				return -EBUSY;
> +		}
> +	}
>  
>  	mutex_lock(&group->lock);
>  
> diff --git a/drivers/media/platform/rcar-vin/rcar-dma.c b/drivers/media/platform/rcar-vin/rcar-dma.c
> index f5f722ab1d4e..80b7ae47d165 100644
> --- a/drivers/media/platform/rcar-vin/rcar-dma.c
> +++ b/drivers/media/platform/rcar-vin/rcar-dma.c
> @@ -1231,7 +1231,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
>  	 */
>  	mdev = vin->vdev.entity.graph_obj.mdev;
>  	mutex_lock(&mdev->graph_mutex);
> -	pipe = sd->entity.pipe ? sd->entity.pipe : &vin->vdev.pipe;
> +	pipe = sd->entity.pads->pipe ? sd->entity.pads->pipe : &vin->vdev.pipe;
>  	ret = __media_pipeline_start(&vin->vdev.entity, pipe);
>  	mutex_unlock(&mdev->graph_mutex);
>  	if (ret)
> diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
> index d33f99c6ffa4..03ee19d00041 100644
> --- a/drivers/media/platform/xilinx/xilinx-dma.c
> +++ b/drivers/media/platform/xilinx/xilinx-dma.c
> @@ -402,7 +402,7 @@ static int xvip_dma_start_streaming(struct vb2_queue *vq, unsigned int count)
>  	 * Use the pipeline object embedded in the first DMA object that starts
>  	 * streaming.
>  	 */
> -	pipe = dma->video.entity.pipe
> +	pipe = dma->video.entity.pads->pipe
>  	     ? to_xvip_pipeline(&dma->video.entity) : &dma->pipe;
>  
>  	ret = media_pipeline_start(&dma->video.entity, &pipe->pipe);
> diff --git a/drivers/media/platform/xilinx/xilinx-dma.h b/drivers/media/platform/xilinx/xilinx-dma.h
> index 2378bdae57ae..69ced71a5696 100644
> --- a/drivers/media/platform/xilinx/xilinx-dma.h
> +++ b/drivers/media/platform/xilinx/xilinx-dma.h
> @@ -47,7 +47,7 @@ struct xvip_pipeline {
>  
>  static inline struct xvip_pipeline *to_xvip_pipeline(struct media_entity *e)
>  {
> -	return container_of(e->pipe, struct xvip_pipeline, pipe);
> +	return container_of(e->pads->pipe, struct xvip_pipeline, pipe);
>  }
>  
>  /**
> diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c
> index 6f90acf9c725..535da4dda3c6 100644
> --- a/drivers/staging/media/imx/imx-media-utils.c
> +++ b/drivers/staging/media/imx/imx-media-utils.c
> @@ -913,7 +913,7 @@ int imx_media_pipeline_set_stream(struct imx_media_dev *imxmd,
>  			__media_pipeline_stop(entity);
>  	} else {
>  		v4l2_subdev_call(sd, video, s_stream, 0);
> -		if (entity->pipe)
> +		if (entity->pads->pipe)
>  			__media_pipeline_stop(entity);
>  	}
>  
> diff --git a/drivers/staging/media/omap4iss/iss.c b/drivers/staging/media/omap4iss/iss.c
> index 68588e9dab0b..4c6f25aa8b57 100644
> --- a/drivers/staging/media/omap4iss/iss.c
> +++ b/drivers/staging/media/omap4iss/iss.c
> @@ -548,7 +548,7 @@ static int iss_pipeline_is_last(struct media_entity *me)
>  	struct iss_pipeline *pipe;
>  	struct media_pad *pad;
>  
> -	if (!me->pipe)
> +	if (!me->pads->pipe)
>  		return 0;
>  	pipe = to_iss_pipeline(me);
>  	if (pipe->stream_state == ISS_PIPELINE_STREAM_STOPPED)
> diff --git a/drivers/staging/media/omap4iss/iss_video.c b/drivers/staging/media/omap4iss/iss_video.c
> index 8c25ad73a81e..b74f7891711d 100644
> --- a/drivers/staging/media/omap4iss/iss_video.c
> +++ b/drivers/staging/media/omap4iss/iss_video.c
> @@ -871,7 +871,7 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
>  	 * Start streaming on the pipeline. No link touching an entity in the
>  	 * pipeline can be activated or deactivated once streaming is started.
>  	 */
> -	pipe = pad->entity->pipe
> +	pipe = pad->pipe
>  	     ? to_iss_pipeline(pad->entity) : &video->pipe;
>  	pipe->external = NULL;
>  	pipe->external_rate = 0;
> diff --git a/drivers/staging/media/omap4iss/iss_video.h b/drivers/staging/media/omap4iss/iss_video.h
> index 526281bf0051..9b8ec27bf87d 100644
> --- a/drivers/staging/media/omap4iss/iss_video.h
> +++ b/drivers/staging/media/omap4iss/iss_video.h
> @@ -91,7 +91,7 @@ struct iss_pipeline {
>  };
>  
>  #define to_iss_pipeline(__e) \
> -	container_of((__e)->pipe, struct iss_pipeline, pipe)
> +	container_of((__e)->pads->pipe, struct iss_pipeline, pipe)
>  
>  static inline int iss_pipeline_ready(struct iss_pipeline *pipe)
>  {
> diff --git a/include/media/media-entity.h b/include/media/media-entity.h
> index 5f6eed24e63f..c9d97c902d05 100644
> --- a/include/media/media-entity.h
> +++ b/include/media/media-entity.h
> @@ -180,15 +180,24 @@ enum media_pad_signal_type {
>   *
>   * @graph_obj:	Embedded structure containing the media object common data
>   * @entity:	Entity this pad belongs to
> + * @pipe:	Pipeline this pad belongs to
> + * @stream_count: Stream count for the pad
>   * @index:	Pad index in the entity pads array, numbered from 0 to n
>   * @sig_type:	Type of the signal inside a media pad
>   * @flags:	Pad flags, as defined in
>   *		:ref:`include/uapi/linux/media.h <media_header>`
>   *		(seek for ``MEDIA_PAD_FL_*``)
> + * .. note::
> + *
> + *    @stream_count reference count must never be negative, but is a signed
> + *    integer on purpose: a simple ``WARN_ON(<0)`` check can be used to
> + *    detect reference count bugs that would make it negative.
>   */
>  struct media_pad {
>  	struct media_gobj graph_obj;	/* must be first field in struct */
>  	struct media_entity *entity;
> +	struct media_pipeline *pipe;
> +	int stream_count;
>  	u16 index;
>  	enum media_pad_signal_type sig_type;
>  	unsigned long flags;
> @@ -267,9 +276,7 @@ enum media_entity_type {
>   * @pads:	Pads array with the size defined by @num_pads.
>   * @links:	List of data links.
>   * @ops:	Entity operations.
> - * @stream_count: Stream count for the entity.
>   * @use_count:	Use count for the entity.
> - * @pipe:	Pipeline this entity belongs to.
>   * @info:	Union with devnode information.  Kept just for backward
>   *		compatibility.
>   * @info.dev:	Contains device major and minor info.
> @@ -282,10 +289,9 @@ enum media_entity_type {
>   *
>   * .. note::
>   *
> - *    @stream_count and @use_count reference counts must never be
> - *    negative, but are signed integers on purpose: a simple ``WARN_ON(<0)``
> - *    check can be used to detect reference count bugs that would make them
> - *    negative.
> + *    @use_count reference count must never be negative, but is a signed
> + *    integer on purpose: a simple ``WARN_ON(<0)`` check can be used to
> + *    detect reference count bugs that would make it negative.
>   */
>  struct media_entity {
>  	struct media_gobj graph_obj;	/* must be first field in struct */
> @@ -304,11 +310,8 @@ struct media_entity {
>  
>  	const struct media_entity_operations *ops;
>  
> -	int stream_count;
>  	int use_count;
>  
> -	struct media_pipeline *pipe;
> -
>  	union {
>  		struct {
>  			u32 major;

-- 
Regards,

Laurent Pinchart

Re: [PATCH v10 19/38] media: entity: Add only connected pads to the pipeline

[Laurent Pinchart]

Hi Tomi and Sakari,

Thank you for the patch.

On Tue, Nov 30, 2021 at 04:15:17PM +0200, Tomi Valkeinen wrote:
> From: Sakari Ailus <sakari.ailus@linux.intel.com>
> 
> A single entity may contain multiple pipelines. Only add pads that were
> connected to the pad through which the entity was reached to the pipeline.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/mc/mc-entity.c | 8 +++-----
>  1 file changed, 3 insertions(+), 5 deletions(-)
> 
> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> index 072f017b399a..4eb4b94c09e2 100644
> --- a/drivers/media/mc/mc-entity.c
> +++ b/drivers/media/mc/mc-entity.c
> @@ -474,7 +474,7 @@ __must_check int __media_pipeline_start(struct media_pad *pad,

Adding a bit more context:

	while ((pad = media_graph_walk_next(graph))) {
		struct media_entity *entity = pad->entity;
		bool skip_validation = pad->pipe != NULL;
		struct media_pad *iter;

		DECLARE_BITMAP(active, MEDIA_ENTITY_MAX_PADS);
		DECLARE_BITMAP(has_no_links, MEDIA_ENTITY_MAX_PADS);
>  
>  		ret = 0;
>  
> -		media_entity_for_each_pad(entity, iter) {
> +		media_entity_for_each_routed_pad(pad, iter) {
>  			if (iter->pipe && iter->pipe != pipe) {
>  				pr_err("Pipe active for %s. Can't start for %s\n",
>  				       entity->name, iter->entity->name);

So here 'iter' is a pad that is reached using an internal route from the
pad 'pad'. 'pad' and 'iter' thus belong to the same entity. 'entity' is
pad->entity, so, unless I'm mistaken, entity->name and
iter->entity->name are the same. I think the message should be reworked.

> @@ -563,10 +563,9 @@ __must_check int __media_pipeline_start(struct media_pad *pad,
>  	media_graph_walk_start(graph, pad_err);
>  
>  	while ((pad_err = media_graph_walk_next(graph))) {
> -		struct media_entity *entity = pad_err->entity;
>  		struct media_pad *iter;
>  
> -		media_entity_for_each_pad(entity, iter) {
> +		media_entity_for_each_routed_pad(pad_err, iter) {
>  			/* Sanity check for negative stream_count */
>  			if (!WARN_ON_ONCE(iter->stream_count <= 0)) {
>  				--iter->stream_count;
> @@ -619,10 +618,9 @@ void __media_pipeline_stop(struct media_pad *pad)
>  	media_graph_walk_start(graph, pad);
>  
>  	while ((pad = media_graph_walk_next(graph))) {
> -		struct media_entity *entity = pad->entity;
>  		struct media_pad *iter;
>  
> -		media_entity_for_each_pad(entity, iter) {
> +		media_entity_for_each_routed_pad(pad, iter) {
>  			/* Sanity check for negative stream_count */
>  			if (!WARN_ON_ONCE(iter->stream_count <= 0)) {
>  				iter->stream_count--;

-- 
Regards,

Laurent Pinchart

Re: [PATCH v10 15/38] media: entity: Add media_entity_has_route() function

[Laurent Pinchart]

Hello Tomi,

Thank you for the patch.

On Tue, Nov 30, 2021 at 04:15:13PM +0200, Tomi Valkeinen wrote:
> From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> 
> This is a wrapper around the media entity has_route operation.
> 
> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Signed-off-by: Michal Simek <michal.simek@xilinx.com>
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/mc/mc-entity.c | 19 +++++++++++++++++++
>  include/media/media-entity.h | 17 +++++++++++++++++
>  2 files changed, 36 insertions(+)
> 
> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> index b44ab423b49b..a83f004efd37 100644
> --- a/drivers/media/mc/mc-entity.c
> +++ b/drivers/media/mc/mc-entity.c
> @@ -229,6 +229,25 @@ EXPORT_SYMBOL_GPL(media_entity_pads_init);
>   * Graph traversal
>   */
>  
> +bool media_entity_has_route(struct media_entity *entity, unsigned int pad0,
> +			    unsigned int pad1)
> +{
> +	if (pad0 >= entity->num_pads || pad1 >= entity->num_pads)
> +		return false;
> +
> +	if (pad0 == pad1)
> +		return true;
> +
> +	if (!entity->ops || !entity->ops->has_route)
> +		return true;
> +
> +	if (entity->pads[pad1].index < entity->pads[pad0].index)
> +		swap(pad0, pad1);
> +
> +	return entity->ops->has_route(entity, pad0, pad1);
> +}
> +EXPORT_SYMBOL_GPL(media_entity_has_route);

As this function is only used in mc-entity.c, can we avoid exposing it
for now ? It's a new API, and I'd rather be notified if drivers start
using it directly by requiring a patch to expose the function at that
point.

> +
>  static struct media_pad *
>  media_pad_other(struct media_pad *pad, struct media_link *link)
>  {
> diff --git a/include/media/media-entity.h b/include/media/media-entity.h
> index ad4020b2df65..b3069eef7fdb 100644
> --- a/include/media/media-entity.h
> +++ b/include/media/media-entity.h
> @@ -904,6 +904,23 @@ int media_entity_get_fwnode_pad(struct media_entity *entity,
>  __must_check int media_graph_walk_init(
>  	struct media_graph *graph, struct media_device *mdev);
>  
> +/**
> + * media_entity_has_route - Check if two entity pads are connected internally
> + *
> + * @entity: The entity
> + * @pad0: The first pad index
> + * @pad1: The second pad index
> + *
> + * This function can be used to check whether two pads of an entity are
> + * connected internally in the entity.
> + *
> + * The caller must hold entity->graph_obj.mdev->mutex.
> + *
> + * Return: true if the pads are connected internally and false otherwise.
> + */
> +bool media_entity_has_route(struct media_entity *entity, unsigned int pad0,
> +			    unsigned int pad1);
> +
>  /**
>   * media_graph_walk_cleanup - Release resources used by graph walk.
>   *

-- 
Regards,

Laurent Pinchart

Re: [PATCH v10 12/38] media: entity: Move the pipeline from entity to pads

[Laurent Pinchart]

Hi Tomi and Sakari,

Another comment (probably more annoying).

On Fri, Jan 07, 2022 at 09:52:06PM +0200, Laurent Pinchart wrote:
> On Tue, Nov 30, 2021 at 04:15:10PM +0200, Tomi Valkeinen wrote:
> > From: Sakari Ailus <sakari.ailus@linux.intel.com>
> > 
> > This moves the pipe and stream_count fields from struct media_entity to
> > struct media_pad. Effectively streams become pad-specific rather than
> > being entity specific, allowing several independent streams to traverse a
> > single entity and an entity to be part of several streams.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
> > 
> > - Update documentation to use 'pads'
> > - Use the media pad iterator in media_entity.c
> > - Update rcar-dma.c to use the new per-pad stream count
> > Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
> > 
> > - Fix cleanup in the error path of __media_pipeline_start()
> > Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > ---
> >  drivers/media/mc/mc-entity.c                  | 68 +++++++++++--------
> >  drivers/media/platform/exynos4-is/fimc-isp.c  |  2 +-
> >  drivers/media/platform/exynos4-is/fimc-lite.c |  2 +-
> >  drivers/media/platform/omap3isp/isp.c         |  2 +-
> >  drivers/media/platform/omap3isp/ispvideo.c    |  2 +-
> >  drivers/media/platform/omap3isp/ispvideo.h    |  2 +-
> >  drivers/media/platform/rcar-vin/rcar-core.c   | 16 +++--
> >  drivers/media/platform/rcar-vin/rcar-dma.c    |  2 +-
> >  drivers/media/platform/xilinx/xilinx-dma.c    |  2 +-
> >  drivers/media/platform/xilinx/xilinx-dma.h    |  2 +-
> >  drivers/staging/media/imx/imx-media-utils.c   |  2 +-
> >  drivers/staging/media/omap4iss/iss.c          |  2 +-
> >  drivers/staging/media/omap4iss/iss_video.c    |  2 +-
> >  drivers/staging/media/omap4iss/iss_video.h    |  2 +-
> >  include/media/media-entity.h                  | 21 +++---
> >  15 files changed, 74 insertions(+), 55 deletions(-)
> > 
> > diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> > index 2b438c481812..8ad4cb845f4a 100644
> > --- a/drivers/media/mc/mc-entity.c
> > +++ b/drivers/media/mc/mc-entity.c
> > @@ -424,24 +424,30 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
> >  
> >  	while ((pad = media_graph_walk_next(graph))) {
> >  		struct media_entity *entity = pad->entity;
> > +		bool skip_validation = pad->pipe != NULL;
> > +		struct media_pad *iter;
> >  
> >  		DECLARE_BITMAP(active, MEDIA_ENTITY_MAX_PADS);
> >  		DECLARE_BITMAP(has_no_links, MEDIA_ENTITY_MAX_PADS);
> >  
> > -		entity->stream_count++;
> > +		ret = 0;
> >  
> > -		if (entity->pipe && entity->pipe != pipe) {
> > -			pr_err("Pipe active for %s. Can't start for %s\n",
> > -				entity->name,
> > -				pad_err->entity->name);
> > -			ret = -EBUSY;
> > -			goto error;
> > +		media_entity_for_each_pad(entity, iter) {
> > +			if (iter->pipe && iter->pipe != pipe) {
> > +				pr_err("Pipe active for %s. Can't start for %s\n",
> > +				       entity->name, iter->entity->name);
> 
> 'iter' is a pad of 'entity'. As 'entity' is set to 'pad->entity',
> 'entity' and 'iter->entity' are the same. The message should be
> reworked.
> 
> > +				ret = -EBUSY;
> > +			} else {
> > +				iter->pipe = pipe;
> > +			}
> > +			iter->stream_count++;
> >  		}
> >  
> > -		entity->pipe = pipe;
> > +		if (ret)
> > +			goto error;
> >  
> > -		/* Already streaming --- no need to check. */
> > -		if (entity->stream_count > 1)
> > +		/* Already part of the pipeline, skip validation. */
> > +		if (skip_validation)
> >  			continue;
> >  
> >  		if (!entity->ops || !entity->ops->link_validate)
> > @@ -510,20 +516,23 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
> >  	media_graph_walk_start(graph, pad_err);
> >  
> >  	while ((pad_err = media_graph_walk_next(graph))) {
> > -		struct media_entity *entity_err = pad_err->entity;
> > -
> > -		/* Sanity check for negative stream_count */
> > -		if (!WARN_ON_ONCE(entity_err->stream_count <= 0)) {
> > -			entity_err->stream_count--;
> > -			if (entity_err->stream_count == 0)
> > -				entity_err->pipe = NULL;
> > +		struct media_entity *entity = pad_err->entity;
> > +		struct media_pad *iter;
> > +
> > +		media_entity_for_each_pad(entity, iter) {
> > +			/* Sanity check for negative stream_count */
> > +			if (!WARN_ON_ONCE(iter->stream_count <= 0)) {
> > +				--iter->stream_count;
> > +				if (iter->stream_count == 0)
> > +					iter->pipe = NULL;
> > +			}
> >  		}
> >  
> >  		/*
> >  		 * We haven't increased stream_count further than this
> >  		 * so we quit here.
> >  		 */
> > -		if (pad_err == pad)
> > +		if (pad_err->entity == pad->entity)
> >  			break;
> >  	}
> >  
> > @@ -550,7 +559,7 @@ EXPORT_SYMBOL_GPL(media_pipeline_start);
> >  
> >  void __media_pipeline_stop(struct media_entity *entity)
> >  {
> > -	struct media_pipeline *pipe = entity->pipe;
> > +	struct media_pipeline *pipe = entity->pads->pipe;
> >  	struct media_graph *graph = &pipe->graph;
> >  	struct media_pad *pad;
> >  
> > @@ -565,12 +574,15 @@ void __media_pipeline_stop(struct media_entity *entity)
> >  
> >  	while ((pad = media_graph_walk_next(graph))) {
> >  		struct media_entity *entity = pad->entity;
> > -
> > -		/* Sanity check for negative stream_count */
> > -		if (!WARN_ON_ONCE(entity->stream_count <= 0)) {
> > -			entity->stream_count--;
> > -			if (entity->stream_count == 0)
> > -				entity->pipe = NULL;
> > +		struct media_pad *iter;
> > +
> > +		media_entity_for_each_pad(entity, iter) {
> > +			/* Sanity check for negative stream_count */
> > +			if (!WARN_ON_ONCE(iter->stream_count <= 0)) {
> > +				iter->stream_count--;
> > +				if (iter->stream_count == 0)
> > +					iter->pipe = NULL;
> > +			}
> >  		}
> >  	}
> >  
> > @@ -840,7 +852,7 @@ int __media_entity_setup_link(struct media_link *link, u32 flags)
> >  {
> >  	const u32 mask = MEDIA_LNK_FL_ENABLED;
> >  	struct media_device *mdev;
> > -	struct media_entity *source, *sink;
> > +	struct media_pad *source, *sink;
> >  	int ret = -EBUSY;
> >  
> >  	if (link == NULL)
> > @@ -856,8 +868,8 @@ int __media_entity_setup_link(struct media_link *link, u32 flags)
> >  	if (link->flags == flags)
> >  		return 0;
> >  
> > -	source = link->source->entity;
> > -	sink = link->sink->entity;
> > +	source = link->source;
> > +	sink = link->sink;
> >  
> >  	if (!(link->flags & MEDIA_LNK_FL_DYNAMIC) &&
> >  	    (source->stream_count || sink->stream_count))
> > diff --git a/drivers/media/platform/exynos4-is/fimc-isp.c b/drivers/media/platform/exynos4-is/fimc-isp.c
> > index 855235bea46d..80274e29ccc5 100644
> > --- a/drivers/media/platform/exynos4-is/fimc-isp.c
> > +++ b/drivers/media/platform/exynos4-is/fimc-isp.c
> > @@ -226,7 +226,7 @@ static int fimc_isp_subdev_set_fmt(struct v4l2_subdev *sd,
> >  			}
> >  		}
> >  	} else {
> > -		if (sd->entity.stream_count == 0) {
> > +		if (sd->entity.pads->stream_count == 0) {
> >  			if (fmt->pad == FIMC_ISP_SD_PAD_SINK) {
> >  				struct v4l2_subdev_format format = *fmt;
> >  
> > diff --git a/drivers/media/platform/exynos4-is/fimc-lite.c b/drivers/media/platform/exynos4-is/fimc-lite.c
> > index aaa3af0493ce..67bfb1ad2ba2 100644
> > --- a/drivers/media/platform/exynos4-is/fimc-lite.c
> > +++ b/drivers/media/platform/exynos4-is/fimc-lite.c
> > @@ -1073,7 +1073,7 @@ static int fimc_lite_subdev_set_fmt(struct v4l2_subdev *sd,
> >  	mutex_lock(&fimc->lock);
> >  
> >  	if ((atomic_read(&fimc->out_path) == FIMC_IO_ISP &&
> > -	    sd->entity.stream_count > 0) ||
> > +	    sd->entity.pads->stream_count > 0) ||
> >  	    (atomic_read(&fimc->out_path) == FIMC_IO_DMA &&
> >  	    vb2_is_busy(&fimc->vb_queue))) {
> >  		mutex_unlock(&fimc->lock);
> > diff --git a/drivers/media/platform/omap3isp/isp.c b/drivers/media/platform/omap3isp/isp.c
> > index 20f59c59ff8a..28aab16d7662 100644
> > --- a/drivers/media/platform/omap3isp/isp.c
> > +++ b/drivers/media/platform/omap3isp/isp.c
> > @@ -936,7 +936,7 @@ static int isp_pipeline_is_last(struct media_entity *me)
> >  	struct isp_pipeline *pipe;
> >  	struct media_pad *pad;
> >  
> > -	if (!me->pipe)
> > +	if (!me->pads->pipe)
> >  		return 0;
> >  	pipe = to_isp_pipeline(me);
> >  	if (pipe->stream_state == ISP_PIPELINE_STREAM_STOPPED)
> > diff --git a/drivers/media/platform/omap3isp/ispvideo.c b/drivers/media/platform/omap3isp/ispvideo.c
> > index 5c1cbb1a9003..a8438040c4aa 100644
> > --- a/drivers/media/platform/omap3isp/ispvideo.c
> > +++ b/drivers/media/platform/omap3isp/ispvideo.c
> > @@ -1094,7 +1094,7 @@ isp_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
> >  	/* Start streaming on the pipeline. No link touching an entity in the
> >  	 * pipeline can be activated or deactivated once streaming is started.
> >  	 */
> > -	pipe = video->video.entity.pipe
> > +	pipe = video->video.entity.pads->pipe
> >  	     ? to_isp_pipeline(&video->video.entity) : &video->pipe;
> >  
> >  	ret = media_entity_enum_init(&pipe->ent_enum, &video->isp->media_dev);
> > diff --git a/drivers/media/platform/omap3isp/ispvideo.h b/drivers/media/platform/omap3isp/ispvideo.h
> > index a0908670c0cf..4c9c5b719ec5 100644
> > --- a/drivers/media/platform/omap3isp/ispvideo.h
> > +++ b/drivers/media/platform/omap3isp/ispvideo.h
> > @@ -100,7 +100,7 @@ struct isp_pipeline {
> >  };
> >  
> >  #define to_isp_pipeline(__e) \
> > -	container_of((__e)->pipe, struct isp_pipeline, pipe)
> > +	container_of((__e)->pads->pipe, struct isp_pipeline, pipe)
> >  
> >  static inline int isp_pipeline_ready(struct isp_pipeline *pipe)
> >  {
> > diff --git a/drivers/media/platform/rcar-vin/rcar-core.c b/drivers/media/platform/rcar-vin/rcar-core.c
> > index 33957cc9118c..e59453d6b7c3 100644
> > --- a/drivers/media/platform/rcar-vin/rcar-core.c
> > +++ b/drivers/media/platform/rcar-vin/rcar-core.c
> > @@ -132,13 +132,17 @@ static int rvin_group_link_notify(struct media_link *link, u32 flags,
> >  		return 0;
> >  
> >  	/*
> > -	 * Don't allow link changes if any entity in the graph is
> > -	 * streaming, modifying the CHSEL register fields can disrupt
> > -	 * running streams.
> > +	 * Don't allow link changes if any stream in the graph is active as
> > +	 * modifying the CHSEL register fields can disrupt running streams.
> >  	 */
> > -	media_device_for_each_entity(entity, &group->mdev)
> > -		if (entity->stream_count)
> > -			return -EBUSY;
> > +	media_device_for_each_entity(entity, &group->mdev) {
> > +		struct media_pad *iter;
> > +
> > +		media_entity_for_each_pad(entity, iter) {
> > +			if (iter->stream_count)
> > +				return -EBUSY;
> > +		}
> > +	}
> >  
> >  	mutex_lock(&group->lock);
> >  
> > diff --git a/drivers/media/platform/rcar-vin/rcar-dma.c b/drivers/media/platform/rcar-vin/rcar-dma.c
> > index f5f722ab1d4e..80b7ae47d165 100644
> > --- a/drivers/media/platform/rcar-vin/rcar-dma.c
> > +++ b/drivers/media/platform/rcar-vin/rcar-dma.c
> > @@ -1231,7 +1231,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
> >  	 */
> >  	mdev = vin->vdev.entity.graph_obj.mdev;
> >  	mutex_lock(&mdev->graph_mutex);
> > -	pipe = sd->entity.pipe ? sd->entity.pipe : &vin->vdev.pipe;
> > +	pipe = sd->entity.pads->pipe ? sd->entity.pads->pipe : &vin->vdev.pipe;
> >  	ret = __media_pipeline_start(&vin->vdev.entity, pipe);
> >  	mutex_unlock(&mdev->graph_mutex);
> >  	if (ret)
> > diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
> > index d33f99c6ffa4..03ee19d00041 100644
> > --- a/drivers/media/platform/xilinx/xilinx-dma.c
> > +++ b/drivers/media/platform/xilinx/xilinx-dma.c
> > @@ -402,7 +402,7 @@ static int xvip_dma_start_streaming(struct vb2_queue *vq, unsigned int count)
> >  	 * Use the pipeline object embedded in the first DMA object that starts
> >  	 * streaming.
> >  	 */
> > -	pipe = dma->video.entity.pipe
> > +	pipe = dma->video.entity.pads->pipe
> >  	     ? to_xvip_pipeline(&dma->video.entity) : &dma->pipe;
> >  
> >  	ret = media_pipeline_start(&dma->video.entity, &pipe->pipe);
> > diff --git a/drivers/media/platform/xilinx/xilinx-dma.h b/drivers/media/platform/xilinx/xilinx-dma.h
> > index 2378bdae57ae..69ced71a5696 100644
> > --- a/drivers/media/platform/xilinx/xilinx-dma.h
> > +++ b/drivers/media/platform/xilinx/xilinx-dma.h
> > @@ -47,7 +47,7 @@ struct xvip_pipeline {
> >  
> >  static inline struct xvip_pipeline *to_xvip_pipeline(struct media_entity *e)
> >  {
> > -	return container_of(e->pipe, struct xvip_pipeline, pipe);
> > +	return container_of(e->pads->pipe, struct xvip_pipeline, pipe);
> >  }
> >  
> >  /**
> > diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c
> > index 6f90acf9c725..535da4dda3c6 100644
> > --- a/drivers/staging/media/imx/imx-media-utils.c
> > +++ b/drivers/staging/media/imx/imx-media-utils.c
> > @@ -913,7 +913,7 @@ int imx_media_pipeline_set_stream(struct imx_media_dev *imxmd,
> >  			__media_pipeline_stop(entity);
> >  	} else {
> >  		v4l2_subdev_call(sd, video, s_stream, 0);
> > -		if (entity->pipe)
> > +		if (entity->pads->pipe)
> >  			__media_pipeline_stop(entity);
> >  	}
> >  
> > diff --git a/drivers/staging/media/omap4iss/iss.c b/drivers/staging/media/omap4iss/iss.c
> > index 68588e9dab0b..4c6f25aa8b57 100644
> > --- a/drivers/staging/media/omap4iss/iss.c
> > +++ b/drivers/staging/media/omap4iss/iss.c
> > @@ -548,7 +548,7 @@ static int iss_pipeline_is_last(struct media_entity *me)
> >  	struct iss_pipeline *pipe;
> >  	struct media_pad *pad;
> >  
> > -	if (!me->pipe)
> > +	if (!me->pads->pipe)
> >  		return 0;
> >  	pipe = to_iss_pipeline(me);
> >  	if (pipe->stream_state == ISS_PIPELINE_STREAM_STOPPED)
> > diff --git a/drivers/staging/media/omap4iss/iss_video.c b/drivers/staging/media/omap4iss/iss_video.c
> > index 8c25ad73a81e..b74f7891711d 100644
> > --- a/drivers/staging/media/omap4iss/iss_video.c
> > +++ b/drivers/staging/media/omap4iss/iss_video.c
> > @@ -871,7 +871,7 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
> >  	 * Start streaming on the pipeline. No link touching an entity in the
> >  	 * pipeline can be activated or deactivated once streaming is started.
> >  	 */
> > -	pipe = pad->entity->pipe
> > +	pipe = pad->pipe
> >  	     ? to_iss_pipeline(pad->entity) : &video->pipe;
> >  	pipe->external = NULL;
> >  	pipe->external_rate = 0;
> > diff --git a/drivers/staging/media/omap4iss/iss_video.h b/drivers/staging/media/omap4iss/iss_video.h
> > index 526281bf0051..9b8ec27bf87d 100644
> > --- a/drivers/staging/media/omap4iss/iss_video.h
> > +++ b/drivers/staging/media/omap4iss/iss_video.h
> > @@ -91,7 +91,7 @@ struct iss_pipeline {
> >  };
> >  
> >  #define to_iss_pipeline(__e) \
> > -	container_of((__e)->pipe, struct iss_pipeline, pipe)
> > +	container_of((__e)->pads->pipe, struct iss_pipeline, pipe)
> >  
> >  static inline int iss_pipeline_ready(struct iss_pipeline *pipe)
> >  {
> > diff --git a/include/media/media-entity.h b/include/media/media-entity.h
> > index 5f6eed24e63f..c9d97c902d05 100644
> > --- a/include/media/media-entity.h
> > +++ b/include/media/media-entity.h
> > @@ -180,15 +180,24 @@ enum media_pad_signal_type {
> >   *
> >   * @graph_obj:	Embedded structure containing the media object common data
> >   * @entity:	Entity this pad belongs to
> > + * @pipe:	Pipeline this pad belongs to
> > + * @stream_count: Stream count for the pad
> >   * @index:	Pad index in the entity pads array, numbered from 0 to n
> >   * @sig_type:	Type of the signal inside a media pad
> >   * @flags:	Pad flags, as defined in
> >   *		:ref:`include/uapi/linux/media.h <media_header>`
> >   *		(seek for ``MEDIA_PAD_FL_*``)
> > + * .. note::
> > + *
> > + *    @stream_count reference count must never be negative, but is a signed
> > + *    integer on purpose: a simple ``WARN_ON(<0)`` check can be used to
> > + *    detect reference count bugs that would make it negative.
> >   */
> >  struct media_pad {
> >  	struct media_gobj graph_obj;	/* must be first field in struct */
> >  	struct media_entity *entity;
> > +	struct media_pipeline *pipe;
> > +	int stream_count;

This doesn't seem enough to me. When carrying multiple streams, it seems
to me that a pad can be part of multiple pipelines. This can be the case
for instance of the source pad of a CSI-2 transmitter or the sink pad of
a CSI-2 receiver.

What do you think ?

> >  	u16 index;
> >  	enum media_pad_signal_type sig_type;
> >  	unsigned long flags;
> > @@ -267,9 +276,7 @@ enum media_entity_type {
> >   * @pads:	Pads array with the size defined by @num_pads.
> >   * @links:	List of data links.
> >   * @ops:	Entity operations.
> > - * @stream_count: Stream count for the entity.
> >   * @use_count:	Use count for the entity.
> > - * @pipe:	Pipeline this entity belongs to.
> >   * @info:	Union with devnode information.  Kept just for backward
> >   *		compatibility.
> >   * @info.dev:	Contains device major and minor info.
> > @@ -282,10 +289,9 @@ enum media_entity_type {
> >   *
> >   * .. note::
> >   *
> > - *    @stream_count and @use_count reference counts must never be
> > - *    negative, but are signed integers on purpose: a simple ``WARN_ON(<0)``
> > - *    check can be used to detect reference count bugs that would make them
> > - *    negative.
> > + *    @use_count reference count must never be negative, but is a signed
> > + *    integer on purpose: a simple ``WARN_ON(<0)`` check can be used to
> > + *    detect reference count bugs that would make it negative.
> >   */
> >  struct media_entity {
> >  	struct media_gobj graph_obj;	/* must be first field in struct */
> > @@ -304,11 +310,8 @@ struct media_entity {
> >  
> >  	const struct media_entity_operations *ops;
> >  
> > -	int stream_count;
> >  	int use_count;
> >  
> > -	struct media_pipeline *pipe;
> > -
> >  	union {
> >  		struct {
> >  			u32 major;

-- 
Regards,

Laurent Pinchart

Re: [PATCH v10 15/38] media: entity: Add media_entity_has_route() function

[Tomi Valkeinen]

On 09/01/2022 01:31, Laurent Pinchart wrote:
> Hello Tomi,
> 
> Thank you for the patch.
> 
> On Tue, Nov 30, 2021 at 04:15:13PM +0200, Tomi Valkeinen wrote:
>> From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
>>
>> This is a wrapper around the media entity has_route operation.
>>
>> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
>> Signed-off-by: Michal Simek <michal.simek@xilinx.com>
>> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
>> Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>>   drivers/media/mc/mc-entity.c | 19 +++++++++++++++++++
>>   include/media/media-entity.h | 17 +++++++++++++++++
>>   2 files changed, 36 insertions(+)
>>
>> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
>> index b44ab423b49b..a83f004efd37 100644
>> --- a/drivers/media/mc/mc-entity.c
>> +++ b/drivers/media/mc/mc-entity.c
>> @@ -229,6 +229,25 @@ EXPORT_SYMBOL_GPL(media_entity_pads_init);
>>    * Graph traversal
>>    */
>>   
>> +bool media_entity_has_route(struct media_entity *entity, unsigned int pad0,
>> +			    unsigned int pad1)
>> +{
>> +	if (pad0 >= entity->num_pads || pad1 >= entity->num_pads)
>> +		return false;
>> +
>> +	if (pad0 == pad1)
>> +		return true;
>> +
>> +	if (!entity->ops || !entity->ops->has_route)
>> +		return true;
>> +
>> +	if (entity->pads[pad1].index < entity->pads[pad0].index)
>> +		swap(pad0, pad1);
>> +
>> +	return entity->ops->has_route(entity, pad0, pad1);
>> +}
>> +EXPORT_SYMBOL_GPL(media_entity_has_route);
> 
> As this function is only used in mc-entity.c, can we avoid exposing it
> for now ? It's a new API, and I'd rather be notified if drivers start
> using it directly by requiring a patch to expose the function at that
> point.
Sounds good to me.

  Tomi

Re: [PATCH v10 13/38] media: entity: Use pad as the starting point for a pipeline

[Laurent Pinchart]

Hi Tomi and Sakari,

Thank you for the patch.

On Tue, Nov 30, 2021 at 04:15:11PM +0200, Tomi Valkeinen wrote:
> From: Sakari Ailus <sakari.ailus@linux.intel.com>
> 
> The pipeline has been moved from the entity to the pads; reflect this in
> the media pipeline function API.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  Documentation/driver-api/media/mc-core.rst    |  9 +++--
>  drivers/media/mc/mc-entity.c                  | 24 ++++++-------
>  drivers/media/pci/intel/ipu3/ipu3-cio2-main.c |  6 ++--
>  .../media/platform/exynos4-is/fimc-capture.c  |  8 ++---
>  .../platform/exynos4-is/fimc-isp-video.c      |  8 ++---
>  drivers/media/platform/exynos4-is/fimc-lite.c |  8 ++---
>  drivers/media/platform/omap3isp/ispvideo.c    |  6 ++--
>  .../media/platform/qcom/camss/camss-video.c   |  6 ++--
>  drivers/media/platform/rcar-vin/rcar-dma.c    |  6 ++--
>  .../platform/rockchip/rkisp1/rkisp1-capture.c |  6 ++--
>  .../media/platform/s3c-camif/camif-capture.c  |  6 ++--
>  drivers/media/platform/stm32/stm32-dcmi.c     |  6 ++--
>  .../platform/sunxi/sun4i-csi/sun4i_dma.c      |  6 ++--
>  .../platform/sunxi/sun6i-csi/sun6i_video.c    |  6 ++--
>  drivers/media/platform/ti-vpe/cal-video.c     |  6 ++--
>  drivers/media/platform/vsp1/vsp1_video.c      |  6 ++--
>  drivers/media/platform/xilinx/xilinx-dma.c    |  6 ++--
>  .../media/test-drivers/vimc/vimc-capture.c    |  6 ++--
>  drivers/media/usb/au0828/au0828-core.c        |  8 ++---
>  drivers/staging/media/imx/imx-media-utils.c   |  6 ++--
>  drivers/staging/media/ipu3/ipu3-v4l2.c        |  6 ++--
>  drivers/staging/media/omap4iss/iss_video.c    |  6 ++--
>  drivers/staging/media/tegra-video/tegra210.c  |  6 ++--
>  include/media/media-entity.h                  | 34 +++++++++----------
>  24 files changed, 99 insertions(+), 102 deletions(-)

[snip]

> diff --git a/drivers/media/platform/s3c-camif/camif-capture.c b/drivers/media/platform/s3c-camif/camif-capture.c
> index 140854ab4dd8..0189b8a33032 100644
> --- a/drivers/media/platform/s3c-camif/camif-capture.c
> +++ b/drivers/media/platform/s3c-camif/camif-capture.c
> @@ -848,13 +848,13 @@ static int s3c_camif_streamon(struct file *file, void *priv,
>  	if (s3c_vp_active(vp))
>  		return 0;
>  
> -	ret = media_pipeline_start(sensor, camif->m_pipeline);
> +	ret = media_pipeline_start(sensor->pads, camif->m_pipeline);

This isn't an issue intruced by this patch, but shouldn't this use
camif->vdev.entity.pads instead of sensor->pads ? Most drivers start the
pipeline from the video node (with a couple of exceptions, see below).

>  	if (ret < 0)
>  		return ret;
>  
>  	ret = camif_pipeline_validate(camif);
>  	if (ret < 0) {
> -		media_pipeline_stop(sensor);
> +		media_pipeline_stop(sensor->pads);
>  		return ret;
>  	}
>  
> @@ -878,7 +878,7 @@ static int s3c_camif_streamoff(struct file *file, void *priv,
>  
>  	ret = vb2_streamoff(&vp->vb_queue, type);
>  	if (ret == 0)
> -		media_pipeline_stop(&camif->sensor.sd->entity);
> +		media_pipeline_stop(camif->sensor.sd->entity.pads);
>  	return ret;
>  }
>  

[snip]

> diff --git a/drivers/media/usb/au0828/au0828-core.c b/drivers/media/usb/au0828/au0828-core.c
> index caefac07af92..877e85a451cb 100644
> --- a/drivers/media/usb/au0828/au0828-core.c
> +++ b/drivers/media/usb/au0828/au0828-core.c
> @@ -410,7 +410,7 @@ static int au0828_enable_source(struct media_entity *entity,
>  		goto end;
>  	}
>  
> -	ret = __media_pipeline_start(entity, pipe);
> +	ret = __media_pipeline_start(entity->pads, pipe);

This I'd be happy to just ignore.

>  	if (ret) {
>  		pr_err("Start Pipeline: %s->%s Error %d\n",
>  			source->name, entity->name, ret);
> @@ -501,12 +501,12 @@ static void au0828_disable_source(struct media_entity *entity)
>  				return;
>  
>  			/* stop pipeline */
> -			__media_pipeline_stop(dev->active_link_owner);
> +			__media_pipeline_stop(dev->active_link_owner->pads);
>  			pr_debug("Pipeline stop for %s\n",
>  				dev->active_link_owner->name);
>  
>  			ret = __media_pipeline_start(
> -					dev->active_link_user,
> +					dev->active_link_user->pads,
>  					dev->active_link_user_pipe);
>  			if (ret) {
>  				pr_err("Start Pipeline: %s->%s %d\n",
> @@ -532,7 +532,7 @@ static void au0828_disable_source(struct media_entity *entity)
>  			return;
>  
>  		/* stop pipeline */
> -		__media_pipeline_stop(dev->active_link_owner);
> +		__media_pipeline_stop(dev->active_link_owner->pads);
>  		pr_debug("Pipeline stop for %s\n",
>  			dev->active_link_owner->name);
>  
> diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c
> index 535da4dda3c6..74218af45551 100644
> --- a/drivers/staging/media/imx/imx-media-utils.c
> +++ b/drivers/staging/media/imx/imx-media-utils.c
> @@ -905,16 +905,16 @@ int imx_media_pipeline_set_stream(struct imx_media_dev *imxmd,
>  	mutex_lock(&imxmd->md.graph_mutex);
>  
>  	if (on) {
> -		ret = __media_pipeline_start(entity, &imxmd->pipe);
> +		ret = __media_pipeline_start(entity->pads, &imxmd->pipe);

And this should be easy to switch to using the video device pad.

>  		if (ret)
>  			goto out;
>  		ret = v4l2_subdev_call(sd, video, s_stream, 1);
>  		if (ret)
> -			__media_pipeline_stop(entity);
> +			__media_pipeline_stop(entity->pads);
>  	} else {
>  		v4l2_subdev_call(sd, video, s_stream, 0);
>  		if (entity->pads->pipe)
> -			__media_pipeline_stop(entity);
> +			__media_pipeline_stop(entity->pads);
>  	}
>  
>  out:

[snip]

-- 
Regards,

Laurent Pinchart

Re: [PATCH v10 16/38] media: entity: Use routing information during graph traversal

[Laurent Pinchart]

Hello Tomi,

Thank you for the patch.

On Tue, Nov 30, 2021 at 04:15:14PM +0200, Tomi Valkeinen wrote:
> From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> 
> Take internal routing information as reported by the entity has_route
> operation into account during graph traversal to avoid following
> unrelated links.
> 
> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Signed-off-by: Michal Simek <michal.simek@xilinx.com>
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
> Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/mc/mc-entity.c | 46 ++++++++++++++++++++++--------------
>  1 file changed, 28 insertions(+), 18 deletions(-)
> 
> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> index a83f004efd37..58cdc9c6b342 100644
> --- a/drivers/media/mc/mc-entity.c
> +++ b/drivers/media/mc/mc-entity.c
> @@ -248,15 +248,6 @@ bool media_entity_has_route(struct media_entity *entity, unsigned int pad0,
>  }
>  EXPORT_SYMBOL_GPL(media_entity_has_route);
>  
> -static struct media_pad *
> -media_pad_other(struct media_pad *pad, struct media_link *link)
> -{
> -	if (link->source == pad)
> -		return link->sink;
> -	else
> -		return link->source;
> -}
> -
>  /* push an entity to traversal stack */
>  static void stack_push(struct media_graph *graph, struct media_pad *pad)
>  {
> @@ -327,7 +318,8 @@ static void media_graph_walk_iter(struct media_graph *graph)
>  {
>  	struct media_pad *pad = stack_top(graph);
>  	struct media_link *link;
> -	struct media_pad *next;
> +	struct media_pad *remote;
> +	struct media_pad *local;
>  
>  	link = list_entry(link_top(graph), typeof(*link), list);
>  
> @@ -341,24 +333,42 @@ static void media_graph_walk_iter(struct media_graph *graph)
>  		return;
>  	}
>  
> -	/* Get the entity at the other end of the link. */
> -	next = media_pad_other(pad, link);
> +	/*
> +	 * Get the local pad, the remote pad and the entity at the other
> +	 * end of the link.
> +	 */
> +	if (link->source->entity == pad->entity) {
> +		remote = link->sink;
> +		local = link->source;
> +	} else {
> +		remote = link->source;
> +		local = link->sink;
> +	}
> +
> +	/*
> +	 * Are the local pad and the pad we came from connected
> +	 * internally in the entity ?
> +	 */
> +	if (!media_entity_has_route(pad->entity, pad->index, local->index)) {

This will fail on the following graph:

+-------------+      +--------------+      +---------+
|             |      |            [1| ---> |0] DMA 0 |
|             |      |              |      +---------+
|   Source  [0| ---> |0] Broadcast  |
|             |      |              |      +---------+
|             |      |            [2| ---> |0] DMA 1 |
+-------------+      +--------------+      +---------+

The broadcast entity forwards the input unchanged to the two outputs.

If we walk the pipeline starting from the video node corresponding to
DMA 0, the pipeline will contain DMA 0, broadcast and source, but will
fail to find DMA 1, because pads 1 and 2 on the broadcast entity don't
have a route connecting them.

I have an alternative implementation of the pipeline walk algorithm that
fixes this, and takes streams into account. I'll send an RFC series.

> +		link_top(graph) = link_top(graph)->next;
> +		return;
> +	}
>  
>  	/* Has the entity already been visited? */
> -	if (media_entity_enum_test_and_set(&graph->ent_enum, next->entity)) {
> +	if (media_entity_enum_test_and_set(&graph->ent_enum, remote->entity)) {
>  		link_top(graph) = link_top(graph)->next;
>  		dev_dbg(pad->graph_obj.mdev->dev,
>  			"walk: skipping entity '%s' (already seen)\n",
> -			next->entity->name);
> +			remote->entity->name);
>  		return;
>  	}
>  
>  	/* Push the new entity to stack and start over. */
>  	link_top(graph) = link_top(graph)->next;
> -	stack_push(graph, next);
> -	dev_dbg(next->graph_obj.mdev->dev, "walk: pushing '%s':%u on stack\n",
> -		next->entity->name, next->index);
> -	lockdep_assert_held(&next->graph_obj.mdev->graph_mutex);
> +	stack_push(graph, remote);
> +	dev_dbg(remote->graph_obj.mdev->dev, "walk: pushing '%s':%u on stack\n",
> +		remote->entity->name, remote->index);
> +	lockdep_assert_held(&remote->graph_obj.mdev->graph_mutex);
>  }
>  
>  struct media_pad *media_graph_walk_next(struct media_graph *graph)

-- 
Regards,

Laurent Pinchart

Re: [PATCH v10 27/38] media: Documentation: Add GS_ROUTING documentation

[Kieran Bingham]

Quoting Tomi Valkeinen (2021-11-30 14:15:25)
> From: Jacopo Mondi <jacopo+renesas@jmondi.org>
> 
> Add documentation for VIDIOC_SUBDEV_G/S_ROUTING ioctl and add
> description of multiplexed media pads and internal routing to the
> V4L2-subdev documentation section.
> 
> Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
> ---
>  .../userspace-api/media/v4l/dev-subdev.rst    |   2 +
>  .../userspace-api/media/v4l/user-func.rst     |   1 +
>  .../media/v4l/vidioc-subdev-g-routing.rst     | 150 ++++++++++++++++++
>  3 files changed, 153 insertions(+)
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> 
> diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> index fd1de0a73a9f..a67c2749089a 100644
> --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> @@ -29,6 +29,8 @@ will feature a character device node on which ioctls can be called to
>  
>  -  negotiate image formats on individual pads
>  
> +-  inspect and modify internal data routing between pads of the same entity
> +
>  Sub-device character device nodes, conventionally named
>  ``/dev/v4l-subdev*``, use major number 81.
>  
> diff --git a/Documentation/userspace-api/media/v4l/user-func.rst b/Documentation/userspace-api/media/v4l/user-func.rst
> index 53e604bd7d60..228c1521f190 100644
> --- a/Documentation/userspace-api/media/v4l/user-func.rst
> +++ b/Documentation/userspace-api/media/v4l/user-func.rst
> @@ -70,6 +70,7 @@ Function Reference
>      vidioc-subdev-g-crop
>      vidioc-subdev-g-fmt
>      vidioc-subdev-g-frame-interval
> +    vidioc-subdev-g-routing
>      vidioc-subdev-g-selection
>      vidioc-subdev-querycap
>      vidioc-subscribe-event
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> new file mode 100644
> index 000000000000..a0d9c79e162f
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> @@ -0,0 +1,150 @@
> +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
> +.. c:namespace:: V4L
> +
> +.. _VIDIOC_SUBDEV_G_ROUTING:
> +
> +******************************************************
> +ioctl VIDIOC_SUBDEV_G_ROUTING, VIDIOC_SUBDEV_S_ROUTING
> +******************************************************
> +
> +Name
> +====
> +
> +VIDIOC_SUBDEV_G_ROUTING - VIDIOC_SUBDEV_S_ROUTING - Get or set routing between streams of media pads in a media entity.
> +
> +
> +Synopsis
> +========
> +
> +.. c:function:: int ioctl( int fd, VIDIOC_SUBDEV_G_ROUTING, struct v4l2_subdev_routing *argp )
> +    :name: VIDIOC_SUBDEV_G_ROUTING
> +
> +.. c:function:: int ioctl( int fd, VIDIOC_SUBDEV_S_ROUTING, struct v4l2_subdev_routing *argp )
> +    :name: VIDIOC_SUBDEV_S_ROUTING
> +
> +
> +Arguments
> +=========
> +
> +``fd``
> +    File descriptor returned by :ref:`open() <func-open>`.
> +
> +``argp``
> +    Pointer to struct :c:type:`v4l2_subdev_routing`.
> +
> +
> +Description
> +===========
> +
> +These ioctls are used to get and set the routing in a media entity.
> +The routing configuration determines the flows of data inside an entity.
> +
> +Drivers report their current routing tables using the
> +``VIDIOC_SUBDEV_G_ROUTING`` ioctl and application may enable or disable routes
> +with the ``VIDIOC_SUBDEV_S_ROUTING`` ioctl, by adding or removing routes and
> +setting or clearing flags of the  ``flags`` field of a
> +struct :c:type:`v4l2_subdev_route`.
> +
> +All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called. This
> +means that the userspace mut reconfigure all streams after calling the ioctl

s/mut/must/

> +with e.g. ``VIDIOC_SUBDEV_S_FMT``.
> +
> +A special case for routing are routes marked with
> +``V4L2_SUBDEV_ROUTE_FL_SOURCE`` flag. These routes are used to describe
> +source endpoints on sensors and the sink fields are unused.
> +
> +When inspecting routes through ``VIDIOC_SUBDEV_G_ROUTING`` and the application
> +provided ``num_routes`` is not big enough to contain all the available routes
> +the subdevice exposes, drivers return the ENOSPC error code and adjust the
> +value of the ``num_routes`` field. Application should then reserve enough memory
> +for all the route entries and call ``VIDIOC_SUBDEV_G_ROUTING`` again.
> +
> +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
> +
> +.. c:type:: v4l2_subdev_routing
> +
> +.. flat-table:: struct v4l2_subdev_routing
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths:       1 1 2
> +
> +    * - __u32
> +      - ``which``
> +      - Format to modified, from enum
> +        :ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
> +    * - struct :c:type:`v4l2_subdev_route`
> +      - ``routes[]``
> +      - Array of struct :c:type:`v4l2_subdev_route` entries
> +    * - __u32
> +      - ``num_routes``
> +      - Number of entries of the routes array
> +    * - __u32
> +      - ``reserved``\ [5]
> +      - Reserved for future extensions. Applications and drivers must set
> +       the array to zero.
> +
> +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
> +
> +.. c:type:: v4l2_subdev_route
> +
> +.. flat-table:: struct v4l2_subdev_route
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths:       1 1 2
> +
> +    * - __u32
> +      - ``sink_pad``
> +      - Sink pad number.
> +    * - __u32
> +      - ``sink_stream``
> +      - Sink pad stream number.
> +    * - __u32
> +      - ``source_pad``
> +      - Source pad number.
> +    * - __u32
> +      - ``source_stream``
> +      - Source pad stream number.
> +    * - __u32
> +      - ``flags``
> +      - Route enable/disable flags
> +       :ref:`v4l2_subdev_routing_flags <v4l2-subdev-routing-flags>`.
> +    * - __u32
> +      - ``reserved``\ [5]
> +      - Reserved for future extensions. Applications and drivers must set
> +       the array to zero.
> +
> +.. tabularcolumns:: |p{6.6cm}|p{2.2cm}|p{8.7cm}|
> +
> +.. _v4l2-subdev-routing-flags:
> +
> +.. flat-table:: enum v4l2_subdev_routing_flags
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths:       3 1 4
> +
> +    * - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +      - 0
> +      - The route is enabled. Set by applications.
> +    * - V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
> +      - 1
> +      - The route is immutable. Set by the driver.
> +    * - V4L2_SUBDEV_ROUTE_FL_SOURCE
> +      - 2
> +      - The route is a source route, and the ``sink_pad`` and ``sink_stream``
> +        fields are unused. Set by the driver.
> +
> +Return Value
> +============
> +
> +On success 0 is returned, on error -1 and the ``errno`` variable is set
> +appropriately. The generic error codes are described at the
> +:ref:`Generic Error Codes <gen-errors>` chapter.
> +
> +ENOSPC
> +   The number of provided route entries is less than the available ones.

This doesn't sound right. Maybe:

"The number of route entries provided is less than the available
routes on the device."

But presumably this would be opposites for G_ROUTING and S_ROUTING,
where on G_ROUTING the userspace must provide at least as many route
entries as the driver/device has, and on S_ROUTING, they must provide
not more than?



> +
> +EINVAL
> +   The sink or source pad identifiers reference a non-existing pad, or reference
> +   pads of different types (ie. the sink_pad identifiers refers to a source pad)
> +   or the sink or source stream identifiers reference a non-existing stream on
> +   the sink or source pad.
> -- 
> 2.25.1
>

Re: [PATCH v10 27/38] media: Documentation: Add GS_ROUTING documentation

[Kieran Bingham]

Quoting Kieran Bingham (2022-02-03 10:55:05)
> Quoting Tomi Valkeinen (2021-11-30 14:15:25)
> > From: Jacopo Mondi <jacopo+renesas@jmondi.org>
> > 
> > Add documentation for VIDIOC_SUBDEV_G/S_ROUTING ioctl and add
> > description of multiplexed media pads and internal routing to the
> > V4L2-subdev documentation section.
> > 
> > Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
> > Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
> > ---
> >  .../userspace-api/media/v4l/dev-subdev.rst    |   2 +
> >  .../userspace-api/media/v4l/user-func.rst     |   1 +
> >  .../media/v4l/vidioc-subdev-g-routing.rst     | 150 ++++++++++++++++++
> >  3 files changed, 153 insertions(+)
> >  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > index fd1de0a73a9f..a67c2749089a 100644
> > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > @@ -29,6 +29,8 @@ will feature a character device node on which ioctls can be called to
> >  
> >  -  negotiate image formats on individual pads
> >  
> > +-  inspect and modify internal data routing between pads of the same entity
> > +
> >  Sub-device character device nodes, conventionally named
> >  ``/dev/v4l-subdev*``, use major number 81.
> >  
> > diff --git a/Documentation/userspace-api/media/v4l/user-func.rst b/Documentation/userspace-api/media/v4l/user-func.rst
> > index 53e604bd7d60..228c1521f190 100644
> > --- a/Documentation/userspace-api/media/v4l/user-func.rst
> > +++ b/Documentation/userspace-api/media/v4l/user-func.rst
> > @@ -70,6 +70,7 @@ Function Reference
> >      vidioc-subdev-g-crop
> >      vidioc-subdev-g-fmt
> >      vidioc-subdev-g-frame-interval
> > +    vidioc-subdev-g-routing
> >      vidioc-subdev-g-selection
> >      vidioc-subdev-querycap
> >      vidioc-subscribe-event
> > diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > new file mode 100644
> > index 000000000000..a0d9c79e162f
> > --- /dev/null
> > +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > @@ -0,0 +1,150 @@
> > +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
> > +.. c:namespace:: V4L
> > +
> > +.. _VIDIOC_SUBDEV_G_ROUTING:
> > +
> > +******************************************************
> > +ioctl VIDIOC_SUBDEV_G_ROUTING, VIDIOC_SUBDEV_S_ROUTING
> > +******************************************************
> > +
> > +Name
> > +====
> > +
> > +VIDIOC_SUBDEV_G_ROUTING - VIDIOC_SUBDEV_S_ROUTING - Get or set routing between streams of media pads in a media entity.
> > +
> > +
> > +Synopsis
> > +========
> > +
> > +.. c:function:: int ioctl( int fd, VIDIOC_SUBDEV_G_ROUTING, struct v4l2_subdev_routing *argp )
> > +    :name: VIDIOC_SUBDEV_G_ROUTING
> > +
> > +.. c:function:: int ioctl( int fd, VIDIOC_SUBDEV_S_ROUTING, struct v4l2_subdev_routing *argp )
> > +    :name: VIDIOC_SUBDEV_S_ROUTING
> > +
> > +
> > +Arguments
> > +=========
> > +
> > +``fd``
> > +    File descriptor returned by :ref:`open() <func-open>`.
> > +
> > +``argp``
> > +    Pointer to struct :c:type:`v4l2_subdev_routing`.
> > +
> > +
> > +Description
> > +===========
> > +
> > +These ioctls are used to get and set the routing in a media entity.
> > +The routing configuration determines the flows of data inside an entity.
> > +
> > +Drivers report their current routing tables using the
> > +``VIDIOC_SUBDEV_G_ROUTING`` ioctl and application may enable or disable routes
> > +with the ``VIDIOC_SUBDEV_S_ROUTING`` ioctl, by adding or removing routes and
> > +setting or clearing flags of the  ``flags`` field of a
> > +struct :c:type:`v4l2_subdev_route`.
> > +
> > +All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called. This
> > +means that the userspace mut reconfigure all streams after calling the ioctl
> 
> s/mut/must/
> 
> > +with e.g. ``VIDIOC_SUBDEV_S_FMT``.
> > +
> > +A special case for routing are routes marked with
> > +``V4L2_SUBDEV_ROUTE_FL_SOURCE`` flag. These routes are used to describe
> > +source endpoints on sensors and the sink fields are unused.
> > +
> > +When inspecting routes through ``VIDIOC_SUBDEV_G_ROUTING`` and the application
> > +provided ``num_routes`` is not big enough to contain all the available routes
> > +the subdevice exposes, drivers return the ENOSPC error code and adjust the
> > +value of the ``num_routes`` field. Application should then reserve enough memory
> > +for all the route entries and call ``VIDIOC_SUBDEV_G_ROUTING`` again.
> > +
> > +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
> > +
> > +.. c:type:: v4l2_subdev_routing
> > +
> > +.. flat-table:: struct v4l2_subdev_routing
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths:       1 1 2
> > +
> > +    * - __u32
> > +      - ``which``
> > +      - Format to modified, from enum
> > +        :ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
> > +    * - struct :c:type:`v4l2_subdev_route`
> > +      - ``routes[]``
> > +      - Array of struct :c:type:`v4l2_subdev_route` entries
> > +    * - __u32
> > +      - ``num_routes``
> > +      - Number of entries of the routes array
> > +    * - __u32
> > +      - ``reserved``\ [5]
> > +      - Reserved for future extensions. Applications and drivers must set
> > +       the array to zero.
> > +
> > +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
> > +
> > +.. c:type:: v4l2_subdev_route
> > +
> > +.. flat-table:: struct v4l2_subdev_route
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths:       1 1 2
> > +
> > +    * - __u32
> > +      - ``sink_pad``
> > +      - Sink pad number.
> > +    * - __u32
> > +      - ``sink_stream``
> > +      - Sink pad stream number.
> > +    * - __u32
> > +      - ``source_pad``
> > +      - Source pad number.
> > +    * - __u32
> > +      - ``source_stream``
> > +      - Source pad stream number.
> > +    * - __u32
> > +      - ``flags``
> > +      - Route enable/disable flags
> > +       :ref:`v4l2_subdev_routing_flags <v4l2-subdev-routing-flags>`.
> > +    * - __u32
> > +      - ``reserved``\ [5]
> > +      - Reserved for future extensions. Applications and drivers must set
> > +       the array to zero.
> > +
> > +.. tabularcolumns:: |p{6.6cm}|p{2.2cm}|p{8.7cm}|
> > +
> > +.. _v4l2-subdev-routing-flags:
> > +
> > +.. flat-table:: enum v4l2_subdev_routing_flags
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths:       3 1 4
> > +
> > +    * - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > +      - 0
> > +      - The route is enabled. Set by applications.
> > +    * - V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
> > +      - 1
> > +      - The route is immutable. Set by the driver.
> > +    * - V4L2_SUBDEV_ROUTE_FL_SOURCE
> > +      - 2
> > +      - The route is a source route, and the ``sink_pad`` and ``sink_stream``
> > +        fields are unused. Set by the driver.
> > +
> > +Return Value
> > +============
> > +
> > +On success 0 is returned, on error -1 and the ``errno`` variable is set
> > +appropriately. The generic error codes are described at the
> > +:ref:`Generic Error Codes <gen-errors>` chapter.
> > +
> > +ENOSPC
> > +   The number of provided route entries is less than the available ones.
> 
> This doesn't sound right. Maybe:
> 
> "The number of route entries provided is less than the available
> routes on the device."
> 
> But presumably this would be opposites for G_ROUTING and S_ROUTING,
> where on G_ROUTING the userspace must provide at least as many route
> entries as the driver/device has, and on S_ROUTING, they must provide
> not more than?

Aha, now I see in the next patch that it only really applies to the
G_ROUTING.

But it might be worth noting that if -ENOSPC is returned, num_routes is
set to the number of routes required by the kernel.

--
Kieran


> 
> 
> 
> > +
> > +EINVAL
> > +   The sink or source pad identifiers reference a non-existing pad, or reference
> > +   pads of different types (ie. the sink_pad identifiers refers to a source pad)
> > +   or the sink or source stream identifiers reference a non-existing stream on
> > +   the sink or source pad.
> > -- 
> > 2.25.1
> >