[PATCH v8 00/36] v4l: subdev internal routing and streams

[PATCH v8 00/36] v4l: subdev internal routing and streams

[Tomi Valkeinen]

Hi,

This is v8 of the multiplexed streams series. v7 can be found from:

https://lore.kernel.org/linux-media/20210524104408.599645-1-tomi.valkeinen@ideasonboard.com/

The main change in this version is the implementation and use of
centralized active state for subdevs.

I have pushed my work branch to:

git://git.kernel.org/pub/scm/linux/kernel/git/tomba/linux.git multistream/work-v8

which contains the patches in this series, along with subdev drivers
using multiplexed streams.

Both this series and the branch above are based on top of today's
git://linuxtv.org/media_tree.git master.

The documentation still needs improving, but I hope the docs in this
series, and the drivers in the work branch, are enough to give the
reviewers enough information to do a review.

As can be guessed from the work branch, I have been testing this series
with TI's FPDLink setup. I have also done a "backwards compatibility"
test by dropping all multiplexed streams patches from the CAL driver
(the CSI-2 RX on the TI SoC), and using the FPDLink drivers with
single-stream configuration.

 Tomi

Jacopo Mondi (2):
  media: entity: Add iterator helper for entity pads
  media: Documentation: Add GS_ROUTING documentation

Laurent Pinchart (4):
  media: entity: Add has_route entity operation
  media: entity: Add media_entity_has_route() function
  media: entity: Use routing information during graph traversal
  media: subdev: Add [GS]_ROUTING subdev ioctls and operations

Sakari Ailus (13):
  media: entity: Use pad as a starting point for graph walk
  media: entity: Use pads instead of entities in the media graph walk
    stack
  media: entity: Walk the graph based on pads
  media: mc: Start walk from a specific pad in use count calculation
  media: entity: Move the pipeline from entity to pads
  media: entity: Use pad as the starting point for a pipeline
  media: entity: Skip link validation for pads to which there is no
    route
  media: entity: Add an iterator helper for connected pads
  media: entity: Add only connected pads to the pipeline
  media: entity: Add debug information in graph walk route check
  media: Add bus type to frame descriptors
  media: Add CSI-2 bus configuration to frame descriptors
  media: Add stream to frame descriptor

Tomi Valkeinen (17):
  media: subdev: rename subdev-state alloc & free
  media: subdev: add active state to struct v4l2_subdev
  media: subdev: add 'which' to subdev state
  media: subdev: pass also the active state to subdevs from ioctls
  media: subdev: add subdev state locking
  media: subdev: Add v4l2_subdev_validate(_and_lock)_state()
  media: Documentation: add documentation about subdev state
  media: subdev: increase V4L2_FRAME_DESC_ENTRY_MAX to 8
  media: add V4L2_SUBDEV_FL_MULTIPLEXED
  media: subdev: add v4l2_subdev_has_route()
  media: subdev: add v4l2_subdev_set_routing helper()
  media: subdev: add stream based configuration
  media: subdev: use streams in v4l2_subdev_link_validate()
  media: subdev: add "opposite" stream helper funcs
  media: subdev: add v4l2_subdev_get_fmt() helper function
  media: subdev: add v4l2_subdev_set_routing_with_fmt() helper
  media: subdev: add v4l2_routing_simple_verify() helper

 Documentation/driver-api/media/mc-core.rst    |  18 +-
 .../driver-api/media/v4l2-subdev.rst          |  25 +
 .../userspace-api/media/v4l/dev-subdev.rst    |   2 +
 .../userspace-api/media/v4l/user-func.rst     |   1 +
 .../v4l/vidioc-subdev-enum-frame-interval.rst |   5 +-
 .../v4l/vidioc-subdev-enum-frame-size.rst     |   5 +-
 .../v4l/vidioc-subdev-enum-mbus-code.rst      |   5 +-
 .../media/v4l/vidioc-subdev-g-crop.rst        |   5 +-
 .../media/v4l/vidioc-subdev-g-fmt.rst         |   5 +-
 .../v4l/vidioc-subdev-g-frame-interval.rst    |   5 +-
 .../media/v4l/vidioc-subdev-g-routing.rst     | 146 ++++
 .../media/v4l/vidioc-subdev-g-selection.rst   |   5 +-
 drivers/media/mc/mc-device.c                  |  13 +-
 drivers/media/mc/mc-entity.c                  | 257 +++---
 drivers/media/pci/intel/ipu3/ipu3-cio2-main.c |   6 +-
 .../media/platform/exynos4-is/fimc-capture.c  |   8 +-
 .../platform/exynos4-is/fimc-isp-video.c      |   8 +-
 drivers/media/platform/exynos4-is/fimc-isp.c  |   2 +-
 drivers/media/platform/exynos4-is/fimc-lite.c |  10 +-
 drivers/media/platform/exynos4-is/media-dev.c |  20 +-
 drivers/media/platform/omap3isp/isp.c         |   2 +-
 drivers/media/platform/omap3isp/ispvideo.c    |  25 +-
 drivers/media/platform/omap3isp/ispvideo.h    |   2 +-
 .../media/platform/qcom/camss/camss-video.c   |   6 +-
 drivers/media/platform/rcar-vin/rcar-core.c   |  16 +-
 drivers/media/platform/rcar-vin/rcar-dma.c    |   8 +-
 drivers/media/platform/rcar-vin/rcar-v4l2.c   |   4 +-
 .../platform/rockchip/rkisp1/rkisp1-capture.c |   6 +-
 .../media/platform/s3c-camif/camif-capture.c  |   6 +-
 drivers/media/platform/stm32/stm32-dcmi.c     |   6 +-
 .../platform/sunxi/sun4i-csi/sun4i_dma.c      |   6 +-
 .../platform/sunxi/sun6i-csi/sun6i_video.c    |   6 +-
 drivers/media/platform/ti-vpe/cal-video.c     |   6 +-
 drivers/media/platform/vsp1/vsp1_entity.c     |   4 +-
 drivers/media/platform/vsp1/vsp1_video.c      |  18 +-
 drivers/media/platform/xilinx/xilinx-dma.c    |  20 +-
 drivers/media/platform/xilinx/xilinx-dma.h    |   2 +-
 .../media/test-drivers/vimc/vimc-capture.c    |   6 +-
 drivers/media/usb/au0828/au0828-core.c        |   8 +-
 drivers/media/v4l2-core/v4l2-ioctl.c          |  25 +-
 drivers/media/v4l2-core/v4l2-mc.c             |  43 +-
 drivers/media/v4l2-core/v4l2-subdev.c         | 735 +++++++++++++++++-
 drivers/staging/media/imx/imx-media-utils.c   |   8 +-
 drivers/staging/media/ipu3/ipu3-v4l2.c        |   6 +-
 drivers/staging/media/omap4iss/iss.c          |   2 +-
 drivers/staging/media/omap4iss/iss_video.c    |  40 +-
 drivers/staging/media/omap4iss/iss_video.h    |   2 +-
 drivers/staging/media/tegra-video/tegra210.c  |   6 +-
 drivers/staging/media/tegra-video/vi.c        |   4 +-
 include/media/media-entity.h                  | 142 +++-
 include/media/v4l2-subdev.h                   | 371 ++++++++-
 include/uapi/linux/v4l2-subdev.h              |  85 +-
 52 files changed, 1808 insertions(+), 369 deletions(-)
 create mode 100644 Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst

-- 
2.25.1

[PATCH v8 01/36] media: subdev: rename subdev-state alloc & free

[Tomi Valkeinen]

The recently added v4l2_subdev_alloc_state() and
v4l2_subdev_free_state() were somewhat badly named. They allocate/free
the state that can be used for a subdev, but they don't change the
subdev itself in any way.

In the future we want to have the subdev state available easily for all
subdevs, and we want to store the state to subdev struct. Thus we will
be needing a new function which allocates the state and stores it into
the subdev struct.

This patch renames v4l2_subdev_alloc/free_state functions to
v4l2_alloc/free_subdev_state, so that we can later use
v4l2_subdev_alloc/free_state for the versions which work on the subdev
struct.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/platform/rcar-vin/rcar-v4l2.c |  4 ++--
 drivers/media/platform/vsp1/vsp1_entity.c   |  4 ++--
 drivers/media/v4l2-core/v4l2-subdev.c       | 12 ++++++------
 drivers/staging/media/tegra-video/vi.c      |  4 ++--
 include/media/v4l2-subdev.h                 | 10 +++++-----
 5 files changed, 17 insertions(+), 17 deletions(-)

diff --git a/drivers/media/platform/rcar-vin/rcar-v4l2.c b/drivers/media/platform/rcar-vin/rcar-v4l2.c
index 0d141155f0e3..5f4fa8c48f68 100644
--- a/drivers/media/platform/rcar-vin/rcar-v4l2.c
+++ b/drivers/media/platform/rcar-vin/rcar-v4l2.c
@@ -252,7 +252,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
 	u32 width, height;
 	int ret;
 
-	sd_state = v4l2_subdev_alloc_state(sd);
+	sd_state = v4l2_alloc_subdev_state(sd);
 	if (IS_ERR(sd_state))
 		return PTR_ERR(sd_state);
 
@@ -288,7 +288,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
 
 	rvin_format_align(vin, pix);
 done:
-	v4l2_subdev_free_state(sd_state);
+	v4l2_free_subdev_state(sd_state);
 
 	return ret;
 }
diff --git a/drivers/media/platform/vsp1/vsp1_entity.c b/drivers/media/platform/vsp1/vsp1_entity.c
index 823c15facd1b..e40bca254b8b 100644
--- a/drivers/media/platform/vsp1/vsp1_entity.c
+++ b/drivers/media/platform/vsp1/vsp1_entity.c
@@ -675,7 +675,7 @@ int vsp1_entity_init(struct vsp1_device *vsp1, struct vsp1_entity *entity,
 	 * Allocate the pad configuration to store formats and selection
 	 * rectangles.
 	 */
-	entity->config = v4l2_subdev_alloc_state(&entity->subdev);
+	entity->config = v4l2_alloc_subdev_state(&entity->subdev);
 	if (IS_ERR(entity->config)) {
 		media_entity_cleanup(&entity->subdev.entity);
 		return PTR_ERR(entity->config);
@@ -690,6 +690,6 @@ void vsp1_entity_destroy(struct vsp1_entity *entity)
 		entity->ops->destroy(entity);
 	if (entity->subdev.ctrl_handler)
 		v4l2_ctrl_handler_free(entity->subdev.ctrl_handler);
-	v4l2_subdev_free_state(entity->config);
+	v4l2_free_subdev_state(entity->config);
 	media_entity_cleanup(&entity->subdev.entity);
 }
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 5d27a27cc2f2..26a34a8e3d37 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -28,7 +28,7 @@ static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
 {
 	struct v4l2_subdev_state *state;
 
-	state = v4l2_subdev_alloc_state(sd);
+	state = v4l2_alloc_subdev_state(sd);
 	if (IS_ERR(state))
 		return PTR_ERR(state);
 
@@ -39,7 +39,7 @@ static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
 
 static void subdev_fh_free(struct v4l2_subdev_fh *fh)
 {
-	v4l2_subdev_free_state(fh->state);
+	v4l2_free_subdev_state(fh->state);
 	fh->state = NULL;
 }
 
@@ -870,7 +870,7 @@ int v4l2_subdev_link_validate(struct media_link *link)
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
 
-struct v4l2_subdev_state *v4l2_subdev_alloc_state(struct v4l2_subdev *sd)
+struct v4l2_subdev_state *v4l2_alloc_subdev_state(struct v4l2_subdev *sd)
 {
 	struct v4l2_subdev_state *state;
 	int ret;
@@ -903,9 +903,9 @@ struct v4l2_subdev_state *v4l2_subdev_alloc_state(struct v4l2_subdev *sd)
 
 	return ERR_PTR(ret);
 }
-EXPORT_SYMBOL_GPL(v4l2_subdev_alloc_state);
+EXPORT_SYMBOL_GPL(v4l2_alloc_subdev_state);
 
-void v4l2_subdev_free_state(struct v4l2_subdev_state *state)
+void v4l2_free_subdev_state(struct v4l2_subdev_state *state)
 {
 	if (!state)
 		return;
@@ -913,7 +913,7 @@ void v4l2_subdev_free_state(struct v4l2_subdev_state *state)
 	kvfree(state->pads);
 	kfree(state);
 }
-EXPORT_SYMBOL_GPL(v4l2_subdev_free_state);
+EXPORT_SYMBOL_GPL(v4l2_free_subdev_state);
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
 
diff --git a/drivers/staging/media/tegra-video/vi.c b/drivers/staging/media/tegra-video/vi.c
index d321790b07d9..a94d19e2a67c 100644
--- a/drivers/staging/media/tegra-video/vi.c
+++ b/drivers/staging/media/tegra-video/vi.c
@@ -507,7 +507,7 @@ static int __tegra_channel_try_format(struct tegra_vi_channel *chan,
 	if (!subdev)
 		return -ENODEV;
 
-	sd_state = v4l2_subdev_alloc_state(subdev);
+	sd_state = v4l2_alloc_subdev_state(subdev);
 	if (IS_ERR(sd_state))
 		return PTR_ERR(sd_state);
 	/*
@@ -558,7 +558,7 @@ static int __tegra_channel_try_format(struct tegra_vi_channel *chan,
 	v4l2_fill_pix_format(pix, &fmt.format);
 	tegra_channel_fmt_align(chan, pix, fmtinfo->bpp);
 
-	v4l2_subdev_free_state(sd_state);
+	v4l2_free_subdev_state(sd_state);
 
 	return 0;
 }
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 95ec18c2f49c..8701d2e7d893 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1135,20 +1135,20 @@ int v4l2_subdev_link_validate_default(struct v4l2_subdev *sd,
 int v4l2_subdev_link_validate(struct media_link *link);
 
 /**
- * v4l2_subdev_alloc_state - allocate v4l2_subdev_state
+ * v4l2_alloc_subdev_state - allocate v4l2_subdev_state
  *
  * @sd: pointer to &struct v4l2_subdev for which the state is being allocated.
  *
- * Must call v4l2_subdev_free_state() when state is no longer needed.
+ * Must call v4l2_free_subdev_state() when state is no longer needed.
  */
-struct v4l2_subdev_state *v4l2_subdev_alloc_state(struct v4l2_subdev *sd);
+struct v4l2_subdev_state *v4l2_alloc_subdev_state(struct v4l2_subdev *sd);
 
 /**
- * v4l2_subdev_free_state - free a v4l2_subdev_state
+ * v4l2_free_subdev_state - free a v4l2_subdev_state
  *
  * @state: v4l2_subdev_state to be freed.
  */
-void v4l2_subdev_free_state(struct v4l2_subdev_state *state);
+void v4l2_free_subdev_state(struct v4l2_subdev_state *state);
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
 
-- 
2.25.1

[PATCH v8 02/36] media: subdev: add active state to struct v4l2_subdev

[Tomi Valkeinen]

Add a new 'state' field to struct v4l2_subdev to which we can store the
active state of a subdev. This will place the subdev configuration into
a known place, allowing us to use the state directly from the v4l2
framework, thus simplifying the drivers.

We also add v4l2_subdev_alloc_state() and v4l2_subdev_free_state(),
which need to be used by the drivers that support subdev state in struct
v4l2_subdev.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 21 ++++++++++++++++
 include/media/v4l2-subdev.h           | 36 +++++++++++++++++++++++++++
 2 files changed, 57 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 26a34a8e3d37..e1a794f69815 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -943,3 +943,24 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
 	v4l2_subdev_notify(sd, V4L2_DEVICE_NOTIFY_EVENT, (void *)ev);
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
+
+int v4l2_subdev_alloc_state(struct v4l2_subdev *sd)
+{
+	struct v4l2_subdev_state *state;
+
+	state = v4l2_alloc_subdev_state(sd);
+	if (IS_ERR(state))
+		return PTR_ERR(state);
+
+	sd->state = state;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_alloc_state);
+
+void v4l2_subdev_free_state(struct v4l2_subdev *sd)
+{
+	v4l2_free_subdev_state(sd->state);
+	sd->state = NULL;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_free_state);
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 8701d2e7d893..ecaf040ead57 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -898,6 +898,8 @@ struct v4l2_subdev_platform_data {
  * @subdev_notifier: A sub-device notifier implicitly registered for the sub-
  *		     device using v4l2_async_register_subdev_sensor().
  * @pdata: common part of subdevice platform data
+ * @state: active state for the subdev (NULL for subdevs tracking the state
+ * 	   internally)
  *
  * Each instance of a subdev driver should create this struct, either
  * stand-alone or embedded in a larger struct.
@@ -929,6 +931,7 @@ struct v4l2_subdev {
 	struct v4l2_async_notifier *notifier;
 	struct v4l2_async_notifier *subdev_notifier;
 	struct v4l2_subdev_platform_data *pdata;
+	struct v4l2_subdev_state *state;
 };
 
 
@@ -1217,4 +1220,37 @@ extern const struct v4l2_subdev_ops v4l2_subdev_call_wrappers;
 void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
 			      const struct v4l2_event *ev);
 
+/**
+ * v4l2_subdev_alloc_state() - Allocate active subdev state for subdevice
+ * @sd: The subdev for which the state is allocated
+ *
+ * This will allocate a subdev state and store it to
+ * &struct v4l2_subdev->state.
+ *
+ * Must call v4l2_subdev_free_state() when the state is no longer needed.
+ */
+int v4l2_subdev_alloc_state(struct v4l2_subdev *sd);
+
+/**
+ * v4l2_subdev_free_state() - Free the active subdev state for subdevice
+ * @sd: The subdevice
+ *
+ * This will free the subdev's state and set
+ * &struct v4l2_subdev->state to NULL.
+ */
+void v4l2_subdev_free_state(struct v4l2_subdev *sd);
+
+/**
+ * v4l2_subdev_get_active_state() - Return the active subdev state for subdevice
+ * @sd: The subdevice
+ *
+ * Return the active state for the subdevice, or NULL if the subdev does not
+ * support active state.
+ */
+static inline struct v4l2_subdev_state *
+v4l2_subdev_get_active_state(struct v4l2_subdev *sd)
+{
+	return sd->state;
+}
+
 #endif
-- 
2.25.1

[PATCH v8 03/36] media: subdev: add 'which' to subdev state

[Tomi Valkeinen]

The subdev state is passed to functions in the media drivers, and
usually either V4L2_SUBDEV_FORMAT_ACTIVE or V4L2_SUBDEV_FORMAT_TRY is
also given to the function in one way or another.

One op where this is not the case is v4l2_subdev_pad_ops.init_cfg. One
could argue that the initialization of the state should be the same for
both ACTIVE and TRY cases, but unfortunately that is not the case:

- Some drivers do also other things than just touch the state when
dealing with ACTIVE, e.g. if there is extra state outside the standard
subdev state.
- Some drivers might need to create, say, struct v4l2_subdev_format
which has 'which' field, and that needs to be filled with either ACTIVE
or TRY.

Currently init_cfg is only called for TRY case from the v4l2 framework,
passing the TRY state. Some drivers call their own init_cfg, passing
NULL as the state, which is used to indicate ACTIVE case.

In the future we want to pass subdev's active state from the v4l2
framework side, so we need a solution to this.

We could change the init_cfg() to include the TRY/ACTIVE value, which
would require changing more or less all the drivers. Instead, I have
added 'which' field to the subdev state itself, filled at state
allocation time, which only requires changes to the drivers that
allocate a state themselves.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/platform/rcar-vin/rcar-v4l2.c |  2 +-
 drivers/media/platform/vsp1/vsp1_entity.c   |  2 +-
 drivers/media/v4l2-core/v4l2-subdev.c       | 10 +++++++---
 drivers/staging/media/tegra-video/vi.c      |  2 +-
 include/media/v4l2-subdev.h                 |  7 ++++++-
 5 files changed, 16 insertions(+), 7 deletions(-)

diff --git a/drivers/media/platform/rcar-vin/rcar-v4l2.c b/drivers/media/platform/rcar-vin/rcar-v4l2.c
index 5f4fa8c48f68..1de30d5b437f 100644
--- a/drivers/media/platform/rcar-vin/rcar-v4l2.c
+++ b/drivers/media/platform/rcar-vin/rcar-v4l2.c
@@ -252,7 +252,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
 	u32 width, height;
 	int ret;
 
-	sd_state = v4l2_alloc_subdev_state(sd);
+	sd_state = v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_ACTIVE);
 	if (IS_ERR(sd_state))
 		return PTR_ERR(sd_state);
 
diff --git a/drivers/media/platform/vsp1/vsp1_entity.c b/drivers/media/platform/vsp1/vsp1_entity.c
index e40bca254b8b..63ea5e472c33 100644
--- a/drivers/media/platform/vsp1/vsp1_entity.c
+++ b/drivers/media/platform/vsp1/vsp1_entity.c
@@ -675,7 +675,7 @@ int vsp1_entity_init(struct vsp1_device *vsp1, struct vsp1_entity *entity,
 	 * Allocate the pad configuration to store formats and selection
 	 * rectangles.
 	 */
-	entity->config = v4l2_alloc_subdev_state(&entity->subdev);
+	entity->config = v4l2_alloc_subdev_state(&entity->subdev, V4L2_SUBDEV_FORMAT_ACTIVE);
 	if (IS_ERR(entity->config)) {
 		media_entity_cleanup(&entity->subdev.entity);
 		return PTR_ERR(entity->config);
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index e1a794f69815..04ad319fb150 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -28,7 +28,7 @@ static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
 {
 	struct v4l2_subdev_state *state;
 
-	state = v4l2_alloc_subdev_state(sd);
+	state = v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_TRY);
 	if (IS_ERR(state))
 		return PTR_ERR(state);
 
@@ -870,7 +870,9 @@ int v4l2_subdev_link_validate(struct media_link *link)
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
 
-struct v4l2_subdev_state *v4l2_alloc_subdev_state(struct v4l2_subdev *sd)
+struct v4l2_subdev_state *
+v4l2_alloc_subdev_state(struct v4l2_subdev *sd,
+			enum v4l2_subdev_format_whence which)
 {
 	struct v4l2_subdev_state *state;
 	int ret;
@@ -879,6 +881,8 @@ struct v4l2_subdev_state *v4l2_alloc_subdev_state(struct v4l2_subdev *sd)
 	if (!state)
 		return ERR_PTR(-ENOMEM);
 
+	state->which = which;
+
 	if (sd->entity.num_pads) {
 		state->pads = kvmalloc_array(sd->entity.num_pads,
 					     sizeof(*state->pads),
@@ -948,7 +952,7 @@ int v4l2_subdev_alloc_state(struct v4l2_subdev *sd)
 {
 	struct v4l2_subdev_state *state;
 
-	state = v4l2_alloc_subdev_state(sd);
+	state = v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_ACTIVE);
 	if (IS_ERR(state))
 		return PTR_ERR(state);
 
diff --git a/drivers/staging/media/tegra-video/vi.c b/drivers/staging/media/tegra-video/vi.c
index a94d19e2a67c..691f5e04b0a1 100644
--- a/drivers/staging/media/tegra-video/vi.c
+++ b/drivers/staging/media/tegra-video/vi.c
@@ -507,7 +507,7 @@ static int __tegra_channel_try_format(struct tegra_vi_channel *chan,
 	if (!subdev)
 		return -ENODEV;
 
-	sd_state = v4l2_alloc_subdev_state(subdev);
+	sd_state = v4l2_alloc_subdev_state(subdev, V4L2_SUBDEV_FORMAT_ACTIVE);
 	if (IS_ERR(sd_state))
 		return PTR_ERR(sd_state);
 	/*
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index ecaf040ead57..5ec78ffda4f5 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -655,6 +655,7 @@ struct v4l2_subdev_pad_config {
 /**
  * struct v4l2_subdev_state - Used for storing subdev state information.
  *
+ * @which: state type (from enum v4l2_subdev_format_whence)
  * @pads: &struct v4l2_subdev_pad_config array
  *
  * This structure only needs to be passed to the pad op if the 'which' field
@@ -662,6 +663,7 @@ struct v4l2_subdev_pad_config {
  * %V4L2_SUBDEV_FORMAT_ACTIVE it is safe to pass %NULL.
  */
 struct v4l2_subdev_state {
+	u32 which;
 	struct v4l2_subdev_pad_config *pads;
 };
 
@@ -1141,10 +1143,13 @@ int v4l2_subdev_link_validate(struct media_link *link);
  * v4l2_alloc_subdev_state - allocate v4l2_subdev_state
  *
  * @sd: pointer to &struct v4l2_subdev for which the state is being allocated.
+ * @which: configuration type for the state (from enum v4l2_subdev_format_whence)
  *
  * Must call v4l2_free_subdev_state() when state is no longer needed.
  */
-struct v4l2_subdev_state *v4l2_alloc_subdev_state(struct v4l2_subdev *sd);
+struct v4l2_subdev_state *
+v4l2_alloc_subdev_state(struct v4l2_subdev *sd,
+			enum v4l2_subdev_format_whence which);
 
 /**
  * v4l2_free_subdev_state - free a v4l2_subdev_state
-- 
2.25.1

[PATCH v8 04/36] media: subdev: pass also the active state to subdevs from ioctls

[Tomi Valkeinen]

At the moment when a subdev op is called, the TRY subdev state
(subdev_fh->state) is passed as a parameter even for ACTIVE case, or
alternatively a NULL can be passed for ACTIVE case. This used to make
sense, as the ACTIVE state was handled internally by the subdev drivers.

We now have a state for ACTIVE case in a standard place, and can pass
that alto to the drivers. This patch changes the subdev ioctls to either
pass the TRY or ACTIVE state to the subdev.

Unfortunately many drivers call ops from other subdevs, and implicitly
pass NULL as the state, so this is just a partial solution. A coccinelle
spatch could perhaps be created which fixes the drivers' subdev calls.

For all current upstream drivers this doesn't matter, as they do not
expect to get a valid state for ACTIVE case. But future drivers which
support multiplexed streaming and routing will depend on getting a state
for both active and try cases, and the simplest way to avoid this
problem is to introduce a helper function, used by the new drivers,
which makes sure the driver has either the TRY or ACTIVE state. This
helper will be introduced in a follow-up patch.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 73 +++++++++++++++++++++++----
 1 file changed, 63 insertions(+), 10 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 04ad319fb150..b3637cddca58 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -353,6 +353,53 @@ const struct v4l2_subdev_ops v4l2_subdev_call_wrappers = {
 EXPORT_SYMBOL(v4l2_subdev_call_wrappers);
 
 #if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
+
+static struct v4l2_subdev_state *
+subdev_ioctl_get_state(struct v4l2_subdev *sd, struct v4l2_subdev_fh *subdev_fh,
+		       unsigned int cmd, void *arg)
+{
+	u32 which;
+
+	switch (cmd) {
+	default:
+		return NULL;
+
+	case VIDIOC_SUBDEV_G_FMT:
+	case VIDIOC_SUBDEV_S_FMT: {
+		which = ((struct v4l2_subdev_format *)arg)->which;
+		break;
+	}
+	case VIDIOC_SUBDEV_G_CROP:
+	case VIDIOC_SUBDEV_S_CROP: {
+		which = ((struct v4l2_subdev_crop *)arg)->which;
+		break;
+	}
+	case VIDIOC_SUBDEV_ENUM_MBUS_CODE: {
+		which = ((struct v4l2_subdev_mbus_code_enum *)arg)->which;
+		break;
+	}
+	case VIDIOC_SUBDEV_ENUM_FRAME_SIZE: {
+		which = ((struct v4l2_subdev_frame_size_enum *)arg)->which;
+		break;
+	}
+
+	case VIDIOC_SUBDEV_ENUM_FRAME_INTERVAL: {
+		which = ((struct v4l2_subdev_frame_interval_enum *)arg)->which;
+		break;
+	}
+
+	case VIDIOC_SUBDEV_G_SELECTION:
+	case VIDIOC_SUBDEV_S_SELECTION: {
+		which = ((struct v4l2_subdev_selection *)arg)->which;
+		break;
+	}
+	}
+
+	return which == V4L2_SUBDEV_FORMAT_TRY ?
+			     subdev_fh->state :
+			     v4l2_subdev_get_active_state(sd);
+}
+
 static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 {
 	struct video_device *vdev = video_devdata(file);
@@ -360,8 +407,11 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 	struct v4l2_fh *vfh = file->private_data;
 	struct v4l2_subdev_fh *subdev_fh = to_v4l2_subdev_fh(vfh);
 	bool ro_subdev = test_bit(V4L2_FL_SUBDEV_RO_DEVNODE, &vdev->flags);
+	struct v4l2_subdev_state *state;
 	int rval;
 
+	state = subdev_ioctl_get_state(sd, subdev_fh, cmd, arg);
+
 	switch (cmd) {
 	case VIDIOC_SUBDEV_QUERYCAP: {
 		struct v4l2_subdev_capability *cap = arg;
@@ -484,7 +534,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 
 		memset(format->reserved, 0, sizeof(format->reserved));
 		memset(format->format.reserved, 0, sizeof(format->format.reserved));
-		return v4l2_subdev_call(sd, pad, get_fmt, subdev_fh->state, format);
+		return v4l2_subdev_call(sd, pad, get_fmt, state, format);
 	}
 
 	case VIDIOC_SUBDEV_S_FMT: {
@@ -495,7 +545,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 
 		memset(format->reserved, 0, sizeof(format->reserved));
 		memset(format->format.reserved, 0, sizeof(format->format.reserved));
-		return v4l2_subdev_call(sd, pad, set_fmt, subdev_fh->state, format);
+		return v4l2_subdev_call(sd, pad, set_fmt, state, format);
 	}
 
 	case VIDIOC_SUBDEV_G_CROP: {
@@ -509,7 +559,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 		sel.target = V4L2_SEL_TGT_CROP;
 
 		rval = v4l2_subdev_call(
-			sd, pad, get_selection, subdev_fh->state, &sel);
+			sd, pad, get_selection, state, &sel);
 
 		crop->rect = sel.r;
 
@@ -531,7 +581,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 		sel.r = crop->rect;
 
 		rval = v4l2_subdev_call(
-			sd, pad, set_selection, subdev_fh->state, &sel);
+			sd, pad, set_selection, state, &sel);
 
 		crop->rect = sel.r;
 
@@ -542,7 +592,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 		struct v4l2_subdev_mbus_code_enum *code = arg;
 
 		memset(code->reserved, 0, sizeof(code->reserved));
-		return v4l2_subdev_call(sd, pad, enum_mbus_code, subdev_fh->state,
+		return v4l2_subdev_call(sd, pad, enum_mbus_code, state,
 					code);
 	}
 
@@ -550,7 +600,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 		struct v4l2_subdev_frame_size_enum *fse = arg;
 
 		memset(fse->reserved, 0, sizeof(fse->reserved));
-		return v4l2_subdev_call(sd, pad, enum_frame_size, subdev_fh->state,
+		return v4l2_subdev_call(sd, pad, enum_frame_size, state,
 					fse);
 	}
 
@@ -575,7 +625,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 		struct v4l2_subdev_frame_interval_enum *fie = arg;
 
 		memset(fie->reserved, 0, sizeof(fie->reserved));
-		return v4l2_subdev_call(sd, pad, enum_frame_interval, subdev_fh->state,
+		return v4l2_subdev_call(sd, pad, enum_frame_interval, state,
 					fie);
 	}
 
@@ -584,7 +634,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 
 		memset(sel->reserved, 0, sizeof(sel->reserved));
 		return v4l2_subdev_call(
-			sd, pad, get_selection, subdev_fh->state, sel);
+			sd, pad, get_selection, state, sel);
 	}
 
 	case VIDIOC_SUBDEV_S_SELECTION: {
@@ -595,7 +645,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 
 		memset(sel->reserved, 0, sizeof(sel->reserved));
 		return v4l2_subdev_call(
-			sd, pad, set_selection, subdev_fh->state, sel);
+			sd, pad, set_selection, state, sel);
 	}
 
 	case VIDIOC_G_EDID: {
@@ -829,10 +879,13 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
 	if (is_media_entity_v4l2_subdev(pad->entity)) {
 		struct v4l2_subdev *sd =
 			media_entity_to_v4l2_subdev(pad->entity);
+		struct v4l2_subdev_state *state;
+
+		state = v4l2_subdev_get_active_state(sd);
 
 		fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
 		fmt->pad = pad->index;
-		return v4l2_subdev_call(sd, pad, get_fmt, NULL, fmt);
+		return v4l2_subdev_call(sd, pad, get_fmt, state, fmt);
 	}
 
 	WARN(pad->entity->function != MEDIA_ENT_F_IO_V4L,
-- 
2.25.1

[PATCH v8 05/36] media: subdev: add subdev state locking

[Tomi Valkeinen]

The V4L2 subdevs have managed without centralized locking for the state
(previously pad_config), as the TRY state is supposedly safe (although I
believe two TRY ioctls for the same fd would race), and the ACTIVE
state, and its locking, is managed by the drivers internally.

We now have ACTIVE state in a centralized position, and need locking.
Strictly speaking the locking is only needed for new drivers that use
the new state, as the current drivers continue behaving as they used to.

Add a mutex to the struct v4l2_subdev_state, along with a few helper
functions for locking/unlocking.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 43 +++++++++++++++++----
 include/media/v4l2-subdev.h           | 55 +++++++++++++++++++++++++--
 2 files changed, 88 insertions(+), 10 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index b3637cddca58..b1e65488210d 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -26,9 +26,11 @@
 #if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
 static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
 {
+	static struct lock_class_key __key;
 	struct v4l2_subdev_state *state;
 
-	state = v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_TRY);
+	state = __v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_TRY,
+					  "v4l2_subdev_fh->state", &__key);
 	if (IS_ERR(state))
 		return PTR_ERR(state);
 
@@ -924,8 +926,10 @@ int v4l2_subdev_link_validate(struct media_link *link)
 EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
 
 struct v4l2_subdev_state *
-v4l2_alloc_subdev_state(struct v4l2_subdev *sd,
-			enum v4l2_subdev_format_whence which)
+__v4l2_alloc_subdev_state(struct v4l2_subdev *sd,
+			  enum v4l2_subdev_format_whence which,
+			  const char *lock_name,
+			  struct lock_class_key *lock_key)
 {
 	struct v4l2_subdev_state *state;
 	int ret;
@@ -934,6 +938,8 @@ v4l2_alloc_subdev_state(struct v4l2_subdev *sd,
 	if (!state)
 		return ERR_PTR(-ENOMEM);
 
+	__mutex_init(&state->lock, lock_name, lock_key);
+
 	state->which = which;
 
 	if (sd->entity.num_pads) {
@@ -960,13 +966,15 @@ v4l2_alloc_subdev_state(struct v4l2_subdev *sd,
 
 	return ERR_PTR(ret);
 }
-EXPORT_SYMBOL_GPL(v4l2_alloc_subdev_state);
+EXPORT_SYMBOL_GPL(__v4l2_alloc_subdev_state);
 
 void v4l2_free_subdev_state(struct v4l2_subdev_state *state)
 {
 	if (!state)
 		return;
 
+	mutex_destroy(&state->lock);
+
 	kvfree(state->pads);
 	kfree(state);
 }
@@ -1001,11 +1009,12 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
 
-int v4l2_subdev_alloc_state(struct v4l2_subdev *sd)
+int __v4l2_subdev_alloc_state(struct v4l2_subdev *sd, const char *name,
+			      struct lock_class_key *key)
 {
 	struct v4l2_subdev_state *state;
 
-	state = v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_ACTIVE);
+	state = __v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_ACTIVE, name, key);
 	if (IS_ERR(state))
 		return PTR_ERR(state);
 
@@ -1013,7 +1022,7 @@ int v4l2_subdev_alloc_state(struct v4l2_subdev *sd)
 
 	return 0;
 }
-EXPORT_SYMBOL_GPL(v4l2_subdev_alloc_state);
+EXPORT_SYMBOL_GPL(__v4l2_subdev_alloc_state);
 
 void v4l2_subdev_free_state(struct v4l2_subdev *sd)
 {
@@ -1021,3 +1030,23 @@ void v4l2_subdev_free_state(struct v4l2_subdev *sd)
 	sd->state = NULL;
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_free_state);
+
+struct v4l2_subdev_state *v4l2_subdev_lock_active_state(struct v4l2_subdev *sd)
+{
+	mutex_lock(&sd->state->lock);
+
+	return sd->state;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_lock_active_state);
+
+void v4l2_subdev_lock_state(struct v4l2_subdev_state *state)
+{
+	mutex_lock(&state->lock);
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_lock_state);
+
+void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state)
+{
+	mutex_unlock(&state->lock);
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_unlock_state);
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 5ec78ffda4f5..52a725281b23 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -655,6 +655,7 @@ struct v4l2_subdev_pad_config {
 /**
  * struct v4l2_subdev_state - Used for storing subdev state information.
  *
+ * @lock: mutex for the state
  * @which: state type (from enum v4l2_subdev_format_whence)
  * @pads: &struct v4l2_subdev_pad_config array
  *
@@ -663,6 +664,7 @@ struct v4l2_subdev_pad_config {
  * %V4L2_SUBDEV_FORMAT_ACTIVE it is safe to pass %NULL.
  */
 struct v4l2_subdev_state {
+	struct mutex lock;
 	u32 which;
 	struct v4l2_subdev_pad_config *pads;
 };
@@ -1147,9 +1149,18 @@ int v4l2_subdev_link_validate(struct media_link *link);
  *
  * Must call v4l2_free_subdev_state() when state is no longer needed.
  */
+#define v4l2_alloc_subdev_state(sd, which)                                     \
+	({                                                                     \
+		static struct lock_class_key __key;                            \
+		const char *name = KBUILD_BASENAME                             \
+			":" __stringify(__LINE__) ":sd->state->lock";          \
+		__v4l2_alloc_subdev_state(sd, which, name, &__key);            \
+	})
+
 struct v4l2_subdev_state *
-v4l2_alloc_subdev_state(struct v4l2_subdev *sd,
-			enum v4l2_subdev_format_whence which);
+__v4l2_alloc_subdev_state(struct v4l2_subdev *sd,
+			  enum v4l2_subdev_format_whence which,
+			  const char *lock_name, struct lock_class_key *key);
 
 /**
  * v4l2_free_subdev_state - free a v4l2_subdev_state
@@ -1234,7 +1245,16 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
  *
  * Must call v4l2_subdev_free_state() when the state is no longer needed.
  */
-int v4l2_subdev_alloc_state(struct v4l2_subdev *sd);
+#define v4l2_subdev_alloc_state(sd)                                            \
+	({                                                                     \
+		static struct lock_class_key __key;                            \
+		const char *name = KBUILD_BASENAME                             \
+			":" __stringify(__LINE__) ":sd->state->lock";          \
+		__v4l2_subdev_alloc_state(sd, name, &__key);                   \
+	})
+
+int __v4l2_subdev_alloc_state(struct v4l2_subdev *sd, const char *name,
+			      struct lock_class_key *key);
 
 /**
  * v4l2_subdev_free_state() - Free the active subdev state for subdevice
@@ -1258,4 +1278,33 @@ v4l2_subdev_get_active_state(struct v4l2_subdev *sd)
 	return sd->state;
 }
 
+/**
+ * v4l2_subdev_lock_active_state() - Lock and return the active subdev state for subdevice
+ * @sd: The subdevice
+ *
+ * Return the locked active state for the subdevice, or NULL if the subdev
+ * does not support active state.
+ *
+ * Must be unlocked with v4l2_subdev_unlock_state() after use.
+ */
+struct v4l2_subdev_state *v4l2_subdev_lock_active_state(struct v4l2_subdev *sd);
+
+/**
+ * v4l2_subdev_lock_state() - Lock the subdev state
+ * @state: The subdevice state
+ *
+ * Lock the given subdev state.
+ *
+ * Must be unlocked with v4l2_subdev_unlock_state() after use.
+ */
+void v4l2_subdev_lock_state(struct v4l2_subdev_state *state);
+
+/**
+ * v4l2_subdev_unlock_state() - Unlock the subdev state
+ * @state: The subdevice state
+ *
+ * Unlock the given subdev state.
+ */
+void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state);
+
 #endif
-- 
2.25.1

[PATCH v8 06/36] media: subdev: Add v4l2_subdev_validate(_and_lock)_state()

[Tomi Valkeinen]

All suitable subdev ops are now passed either the TRY or the ACTIVE
state by the v4l2 core. However, other subdrivers can still call the ops
passing NULL as the state, implying the active case.

Thus all subdev drivers supporting active state need to handle the NULL
state case. Additionally, the subdev drivers usually need to lock the
state.

Add two helper functions to easen the transition to centrally managed
ACTIVE state. v4l2_subdev_validate_state() ensures that the state is not
NULL, and v4l2_subdev_validate_and_lock_state() does the same and
additionally locks the state.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 include/media/v4l2-subdev.h | 41 +++++++++++++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 52a725281b23..2290b5025fc0 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1307,4 +1307,45 @@ void v4l2_subdev_lock_state(struct v4l2_subdev_state *state);
  */
 void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state);
 
+/**
+ * v4l2_subdev_validate_state() - Gets the TRY or ACTIVE subdev state
+ * @sd: subdevice
+ * @state: subdevice state as passed to the subdev op
+ *
+ * Subdev ops used to be sometimes called with NULL as the state for ACTIVE
+ * case. Even if the v4l2 core now passes proper state for both TRY and
+ * ACTIVE cases, a subdev driver may call an op in another subdev driver,
+ * passing NULL.
+ *
+ * This function can be used as a helper to get the state also for the ACTIVE
+ * case. The subdev driver that supports ACTIVE state can use this function
+ * as the first thing in its ops, ensuring that the state variable contains
+ * either the TRY or ACTIVE state.
+ */
+static inline struct v4l2_subdev_state *
+v4l2_subdev_validate_state(struct v4l2_subdev *sd,
+			   struct v4l2_subdev_state *state)
+{
+	return state ? state : sd->state;
+}
+
+/**
+ * v4l2_subdev_validate_and_lock_state() - Gets locked TRY or ACTIVE subdev state
+ * @sd: subdevice
+ * @state: subdevice state as passed to the subdev op
+ *
+ * This is a helper function which does the same as v4l2_subdev_validate_state
+ * () except that it also locks the state.
+ */
+static inline struct v4l2_subdev_state *
+v4l2_subdev_validate_and_lock_state(struct v4l2_subdev *sd,
+				    struct v4l2_subdev_state *state)
+{
+	state = state ? state : sd->state;
+
+	v4l2_subdev_lock_state(state);
+
+	return state;
+}
+
 #endif
-- 
2.25.1

[PATCH v8 07/36] media: Documentation: add documentation about subdev state

[Tomi Valkeinen]

Add documentation about centrally managed subdev state.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 .../driver-api/media/v4l2-subdev.rst          | 25 +++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/Documentation/driver-api/media/v4l2-subdev.rst b/Documentation/driver-api/media/v4l2-subdev.rst
index 7736da077fb8..b28607c3fc79 100644
--- a/Documentation/driver-api/media/v4l2-subdev.rst
+++ b/Documentation/driver-api/media/v4l2-subdev.rst
@@ -518,6 +518,31 @@ The :c:func:`v4l2_i2c_new_subdev` function will call
 :c:type:`i2c_board_info` structure using the ``client_type`` and the
 ``addr`` to fill it.
 
+Centrally managed subdev active state
+-------------------------------------
+
+Traditionally V4L2 subdev drivers maintained internal state for the active
+configuration for the subdev. This is often implemented as an array of struct
+v4l2_mbus_framefmt, on entry for each pad.
+
+In addition to the active configuration, each subdev filehandle has contained
+an array of struct v4l2_subdev_pad_config, managed by V4L2 core, which
+contains the TRY configuration.
+
+To simplify the subdev drivers the V4L2 subdev API now optionally supports a
+centrally managed active configuration. A subdev driver must use
+v4l2_subdev_alloc_state() to initialize the active state between calls to
+media_entity_pads_init() and v4l2_*_register_subdev().
+
+The active state must be locked before access, and can be done with e.g.
+v4l2_subdev_lock_state() or v4l2_subdev_lock_active_state().
+
+The V4L2 core will pass either the TRY or ACTIVE state to various subdev ops.
+Unfortunately all the subdev drivers do not comply with this yet, and may
+pass NULL for the ACTIVE case, so the called subdev ops must also handle the
+NULL case. This can be easily managed by the helpers
+v4l2_subdev_validate_state() or v4l2_subdev_validate_and_lock_state().
+
 V4L2 sub-device functions and data structures
 ---------------------------------------------
 
-- 
2.25.1

[PATCH v8 08/36] media: entity: Use pad as a starting point for graph walk

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

With the upcoming use of the recently added has_route() media entity op, all
the pads in an entity will no longer be considered interconnected. This has
an effect where the media graph is traversed: the starting pad does make a
difference.

Prepare for this change by using pad instead of the entity as an argument
for the graph walk operations. The actual graph traversal algorithm change
is in further patches.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 Documentation/driver-api/media/mc-core.rst    |  2 +-
 drivers/media/mc/mc-entity.c                  | 17 ++++++++---------
 drivers/media/platform/exynos4-is/media-dev.c |  4 ++--
 drivers/media/platform/omap3isp/ispvideo.c    |  2 +-
 drivers/media/platform/vsp1/vsp1_video.c      |  2 +-
 drivers/media/platform/xilinx/xilinx-dma.c    |  2 +-
 drivers/media/v4l2-core/v4l2-mc.c             |  6 +++---
 drivers/staging/media/omap4iss/iss_video.c    |  4 ++--
 include/media/media-entity.h                  | 10 ++++------
 9 files changed, 23 insertions(+), 26 deletions(-)

diff --git a/Documentation/driver-api/media/mc-core.rst b/Documentation/driver-api/media/mc-core.rst
index 57b5bbba944e..ba0aee982124 100644
--- a/Documentation/driver-api/media/mc-core.rst
+++ b/Documentation/driver-api/media/mc-core.rst
@@ -167,7 +167,7 @@ Drivers initiate a graph traversal by calling
 :c:func:`media_graph_walk_start()`
 
 The graph structure, provided by the caller, is initialized to start graph
-traversal at the given entity.
+traversal at the given pad in an entity.
 
 Drivers can then retrieve the next entity by calling
 :c:func:`media_graph_walk_next()`
diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index f40f41977142..47cdaa301838 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -291,17 +291,16 @@ void media_graph_walk_cleanup(struct media_graph *graph)
 }
 EXPORT_SYMBOL_GPL(media_graph_walk_cleanup);
 
-void media_graph_walk_start(struct media_graph *graph,
-			    struct media_entity *entity)
+void media_graph_walk_start(struct media_graph *graph, struct media_pad *pad)
 {
 	media_entity_enum_zero(&graph->ent_enum);
-	media_entity_enum_set(&graph->ent_enum, entity);
+	media_entity_enum_set(&graph->ent_enum, pad->entity);
 
 	graph->top = 0;
 	graph->stack[graph->top].entity = NULL;
-	stack_push(graph, entity);
-	dev_dbg(entity->graph_obj.mdev->dev,
-		"begin graph walk at '%s'\n", entity->name);
+	stack_push(graph, pad->entity);
+	dev_dbg(pad->graph_obj.mdev->dev,
+		"begin graph walk at '%s':%u\n", pad->entity->name, pad->index);
 }
 EXPORT_SYMBOL_GPL(media_graph_walk_start);
 
@@ -420,7 +419,7 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 			goto error_graph_walk_start;
 	}
 
-	media_graph_walk_start(&pipe->graph, entity);
+	media_graph_walk_start(&pipe->graph, entity->pads);
 
 	while ((entity = media_graph_walk_next(graph))) {
 		DECLARE_BITMAP(active, MEDIA_ENTITY_MAX_PADS);
@@ -504,7 +503,7 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 	 * Link validation on graph failed. We revert what we did and
 	 * return the error.
 	 */
-	media_graph_walk_start(graph, entity_err);
+	media_graph_walk_start(graph, entity_err->pads);
 
 	while ((entity_err = media_graph_walk_next(graph))) {
 		/* Sanity check for negative stream_count */
@@ -555,7 +554,7 @@ void __media_pipeline_stop(struct media_entity *entity)
 	if (WARN_ON(!pipe))
 		return;
 
-	media_graph_walk_start(graph, entity);
+	media_graph_walk_start(graph, entity->pads);
 
 	while ((entity = media_graph_walk_next(graph))) {
 		/* Sanity check for negative stream_count */
diff --git a/drivers/media/platform/exynos4-is/media-dev.c b/drivers/media/platform/exynos4-is/media-dev.c
index fa648721eaab..4ec7f7a6b0f6 100644
--- a/drivers/media/platform/exynos4-is/media-dev.c
+++ b/drivers/media/platform/exynos4-is/media-dev.c
@@ -1173,7 +1173,7 @@ static int __fimc_md_modify_pipelines(struct media_entity *entity, bool enable,
 	 * through active links. This is needed as we cannot power on/off the
 	 * subdevs in random order.
 	 */
-	media_graph_walk_start(graph, entity);
+	media_graph_walk_start(graph, entity->pads);
 
 	while ((entity = media_graph_walk_next(graph))) {
 		if (!is_media_entity_v4l2_video_device(entity))
@@ -1188,7 +1188,7 @@ static int __fimc_md_modify_pipelines(struct media_entity *entity, bool enable,
 	return 0;
 
 err:
-	media_graph_walk_start(graph, entity_err);
+	media_graph_walk_start(graph, entity_err->pads);
 
 	while ((entity_err = media_graph_walk_next(graph))) {
 		if (!is_media_entity_v4l2_video_device(entity_err))
diff --git a/drivers/media/platform/omap3isp/ispvideo.c b/drivers/media/platform/omap3isp/ispvideo.c
index 8811d6dd4ee7..3c1485d59404 100644
--- a/drivers/media/platform/omap3isp/ispvideo.c
+++ b/drivers/media/platform/omap3isp/ispvideo.c
@@ -234,7 +234,7 @@ static int isp_video_get_graph_data(struct isp_video *video,
 		return ret;
 	}
 
-	media_graph_walk_start(&graph, entity);
+	media_graph_walk_start(&graph, entity->pads);
 
 	while ((entity = media_graph_walk_next(&graph))) {
 		struct isp_video *__video;
diff --git a/drivers/media/platform/vsp1/vsp1_video.c b/drivers/media/platform/vsp1/vsp1_video.c
index 044eb5778820..61e4fbaba7b7 100644
--- a/drivers/media/platform/vsp1/vsp1_video.c
+++ b/drivers/media/platform/vsp1/vsp1_video.c
@@ -569,7 +569,7 @@ static int vsp1_video_pipeline_build(struct vsp1_pipeline *pipe,
 	if (ret)
 		return ret;
 
-	media_graph_walk_start(&graph, entity);
+	media_graph_walk_start(&graph, entity->pads);
 
 	while ((entity = media_graph_walk_next(&graph))) {
 		struct v4l2_subdev *subdev;
diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
index 338c3661d809..af0d55ab6c15 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.c
+++ b/drivers/media/platform/xilinx/xilinx-dma.c
@@ -189,7 +189,7 @@ static int xvip_pipeline_validate(struct xvip_pipeline *pipe,
 		return ret;
 	}
 
-	media_graph_walk_start(&graph, entity);
+	media_graph_walk_start(&graph, entity->pads);
 
 	while ((entity = media_graph_walk_next(&graph))) {
 		struct xvip_dma *dma;
diff --git a/drivers/media/v4l2-core/v4l2-mc.c b/drivers/media/v4l2-core/v4l2-mc.c
index b01474717dca..d215fe31b9a2 100644
--- a/drivers/media/v4l2-core/v4l2-mc.c
+++ b/drivers/media/v4l2-core/v4l2-mc.c
@@ -436,7 +436,7 @@ static int pipeline_pm_use_count(struct media_entity *entity,
 {
 	int use = 0;
 
-	media_graph_walk_start(graph, entity);
+	media_graph_walk_start(graph, entity->pads);
 
 	while ((entity = media_graph_walk_next(graph))) {
 		if (is_media_entity_v4l2_video_device(entity))
@@ -499,7 +499,7 @@ static int pipeline_pm_power(struct media_entity *entity, int change,
 	if (!change)
 		return 0;
 
-	media_graph_walk_start(graph, entity);
+	media_graph_walk_start(graph, entity->pads);
 
 	while (!ret && (entity = media_graph_walk_next(graph)))
 		if (is_media_entity_v4l2_subdev(entity))
@@ -508,7 +508,7 @@ static int pipeline_pm_power(struct media_entity *entity, int change,
 	if (!ret)
 		return ret;
 
-	media_graph_walk_start(graph, first);
+	media_graph_walk_start(graph, first->pads);
 
 	while ((first = media_graph_walk_next(graph))
 	       && first != entity)
diff --git a/drivers/staging/media/omap4iss/iss_video.c b/drivers/staging/media/omap4iss/iss_video.c
index d0da083deed5..760cd0ab1feb 100644
--- a/drivers/staging/media/omap4iss/iss_video.c
+++ b/drivers/staging/media/omap4iss/iss_video.c
@@ -217,7 +217,7 @@ iss_video_far_end(struct iss_video *video)
 		return NULL;
 	}
 
-	media_graph_walk_start(&graph, entity);
+	media_graph_walk_start(&graph, entity->pads);
 
 	while ((entity = media_graph_walk_next(&graph))) {
 		if (entity == &video->video.entity)
@@ -892,7 +892,7 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 		goto err_media_pipeline_start;
 
 	mutex_lock(&mdev->graph_mutex);
-	media_graph_walk_start(&graph, entity);
+	media_graph_walk_start(&graph, entity->pads);
 	while ((entity = media_graph_walk_next(&graph)))
 		media_entity_enum_set(&pipe->ent_enum, entity);
 	mutex_unlock(&mdev->graph_mutex);
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 09737b47881f..b9bfcf34eb0a 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -902,22 +902,20 @@ __must_check int media_graph_walk_init(
 void media_graph_walk_cleanup(struct media_graph *graph);
 
 /**
- * media_graph_walk_start - Start walking the media graph at a
- *	given entity
+ * media_graph_walk_start - Start walking the media graph at a given pad
  *
  * @graph: Media graph structure that will be used to walk the graph
- * @entity: Starting entity
+ * @pad: Starting pad
  *
  * Before using this function, media_graph_walk_init() must be
  * used to allocate resources used for walking the graph. This
  * function initializes the graph traversal structure to walk the
- * entities graph starting at the given entity. The traversal
+ * entities graph starting at the given pad. The traversal
  * structure must not be modified by the caller during graph
  * traversal. After the graph walk, the resources must be released
  * using media_graph_walk_cleanup().
  */
-void media_graph_walk_start(struct media_graph *graph,
-			    struct media_entity *entity);
+void media_graph_walk_start(struct media_graph *graph, struct media_pad *pad);
 
 /**
  * media_graph_walk_next - Get the next entity in the graph
-- 
2.25.1

[PATCH v8 09/36] media: entity: Use pads instead of entities in the media graph walk stack

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

Change the media graph walk stack structure to use media pads instead of
using media entities. In addition to the entity, the pad contains the
information which pad in the entity are being dealt with.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 55 ++++++++++++++++++------------------
 include/media/media-entity.h |  8 +++---
 2 files changed, 31 insertions(+), 32 deletions(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 47cdaa301838..e3a932ab9abd 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -228,40 +228,39 @@ EXPORT_SYMBOL_GPL(media_entity_pads_init);
  * Graph traversal
  */
 
-static struct media_entity *
-media_entity_other(struct media_entity *entity, struct media_link *link)
+static struct media_pad *
+media_pad_other(struct media_pad *pad, struct media_link *link)
 {
-	if (link->source->entity == entity)
-		return link->sink->entity;
+	if (link->source == pad)
+		return link->sink;
 	else
-		return link->source->entity;
+		return link->source;
 }
 
 /* push an entity to traversal stack */
-static void stack_push(struct media_graph *graph,
-		       struct media_entity *entity)
+static void stack_push(struct media_graph *graph, struct media_pad *pad)
 {
 	if (graph->top == MEDIA_ENTITY_ENUM_MAX_DEPTH - 1) {
 		WARN_ON(1);
 		return;
 	}
 	graph->top++;
-	graph->stack[graph->top].link = entity->links.next;
-	graph->stack[graph->top].entity = entity;
+	graph->stack[graph->top].link = pad->entity->links.next;
+	graph->stack[graph->top].pad = pad;
 }
 
-static struct media_entity *stack_pop(struct media_graph *graph)
+static struct media_pad *stack_pop(struct media_graph *graph)
 {
-	struct media_entity *entity;
+	struct media_pad *pad;
 
-	entity = graph->stack[graph->top].entity;
+	pad = graph->stack[graph->top].pad;
 	graph->top--;
 
-	return entity;
+	return pad;
 }
 
 #define link_top(en)	((en)->stack[(en)->top].link)
-#define stack_top(en)	((en)->stack[(en)->top].entity)
+#define stack_top(en)	((en)->stack[(en)->top].pad)
 
 /**
  * media_graph_walk_init - Allocate resources for graph walk
@@ -297,8 +296,8 @@ void media_graph_walk_start(struct media_graph *graph, struct media_pad *pad)
 	media_entity_enum_set(&graph->ent_enum, pad->entity);
 
 	graph->top = 0;
-	graph->stack[graph->top].entity = NULL;
-	stack_push(graph, pad->entity);
+	graph->stack[graph->top].pad = NULL;
+	stack_push(graph, pad);
 	dev_dbg(pad->graph_obj.mdev->dev,
 		"begin graph walk at '%s':%u\n", pad->entity->name, pad->index);
 }
@@ -306,16 +305,16 @@ EXPORT_SYMBOL_GPL(media_graph_walk_start);
 
 static void media_graph_walk_iter(struct media_graph *graph)
 {
-	struct media_entity *entity = stack_top(graph);
+	struct media_pad *pad = stack_top(graph);
 	struct media_link *link;
-	struct media_entity *next;
+	struct media_pad *next;
 
 	link = list_entry(link_top(graph), typeof(*link), list);
 
 	/* The link is not enabled so we do not follow. */
 	if (!(link->flags & MEDIA_LNK_FL_ENABLED)) {
 		link_top(graph) = link_top(graph)->next;
-		dev_dbg(entity->graph_obj.mdev->dev,
+		dev_dbg(pad->graph_obj.mdev->dev,
 			"walk: skipping disabled link '%s':%u -> '%s':%u\n",
 			link->source->entity->name, link->source->index,
 			link->sink->entity->name, link->sink->index);
@@ -323,23 +322,23 @@ static void media_graph_walk_iter(struct media_graph *graph)
 	}
 
 	/* Get the entity at the other end of the link. */
-	next = media_entity_other(entity, link);
+	next = media_pad_other(pad, link);
 
 	/* Has the entity already been visited? */
-	if (media_entity_enum_test_and_set(&graph->ent_enum, next)) {
+	if (media_entity_enum_test_and_set(&graph->ent_enum, next->entity)) {
 		link_top(graph) = link_top(graph)->next;
-		dev_dbg(entity->graph_obj.mdev->dev,
+		dev_dbg(pad->graph_obj.mdev->dev,
 			"walk: skipping entity '%s' (already seen)\n",
-			next->name);
+			next->entity->name);
 		return;
 	}
 
 	/* Push the new entity to stack and start over. */
 	link_top(graph) = link_top(graph)->next;
 	stack_push(graph, next);
-	dev_dbg(entity->graph_obj.mdev->dev, "walk: pushing '%s' on stack\n",
-		next->name);
-	lockdep_assert_held(&entity->graph_obj.mdev->graph_mutex);
+	dev_dbg(next->graph_obj.mdev->dev, "walk: pushing '%s':%u on stack\n",
+		next->entity->name, next->index);
+	lockdep_assert_held(&next->graph_obj.mdev->graph_mutex);
 }
 
 struct media_entity *media_graph_walk_next(struct media_graph *graph)
@@ -354,10 +353,10 @@ struct media_entity *media_graph_walk_next(struct media_graph *graph)
 	 * top of the stack until no more entities on the level can be
 	 * found.
 	 */
-	while (link_top(graph) != &stack_top(graph)->links)
+	while (link_top(graph) != &stack_top(graph)->entity->links)
 		media_graph_walk_iter(graph);
 
-	entity = stack_pop(graph);
+	entity = stack_pop(graph)->entity;
 	dev_dbg(entity->graph_obj.mdev->dev,
 		"walk: returning entity '%s'\n", entity->name);
 
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index b9bfcf34eb0a..5b55d6179e13 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -78,16 +78,16 @@ struct media_entity_enum {
  * struct media_graph - Media graph traversal state
  *
  * @stack:		Graph traversal stack; the stack contains information
- *			on the path the media entities to be walked and the
- *			links through which they were reached.
- * @stack.entity:	pointer to &struct media_entity at the graph.
+ *			on the media pads to be walked and the links through
+ *			which they were reached.
+ * @stack.pad:		pointer to &struct media_pad at the graph.
  * @stack.link:		pointer to &struct list_head.
  * @ent_enum:		Visited entities
  * @top:		The top of the stack
  */
 struct media_graph {
 	struct {
-		struct media_entity *entity;
+		struct media_pad *pad;
 		struct list_head *link;
 	} stack[MEDIA_ENTITY_ENUM_MAX_DEPTH];
 
-- 
2.25.1

[PATCH v8 10/36] media: entity: Walk the graph based on pads

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

Instead of iterating over graph entities during the walk, iterate the pads
through which the entity was first reached. This is required in order to
make the entity pipeline pad-based rather than entity based.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 Documentation/driver-api/media/mc-core.rst    |  7 ++-
 drivers/media/mc/mc-entity.c                  | 51 +++++++++++--------
 drivers/media/platform/exynos4-is/media-dev.c | 20 ++++----
 drivers/media/platform/omap3isp/ispvideo.c    | 17 ++++---
 drivers/media/platform/vsp1/vsp1_video.c      | 12 ++---
 drivers/media/platform/xilinx/xilinx-dma.c    | 12 ++---
 drivers/media/v4l2-core/v4l2-mc.c             | 24 ++++-----
 drivers/staging/media/omap4iss/iss_video.c    | 36 ++++++-------
 include/media/media-entity.h                  |  7 +--
 9 files changed, 101 insertions(+), 85 deletions(-)

diff --git a/Documentation/driver-api/media/mc-core.rst b/Documentation/driver-api/media/mc-core.rst
index ba0aee982124..8a13640bed56 100644
--- a/Documentation/driver-api/media/mc-core.rst
+++ b/Documentation/driver-api/media/mc-core.rst
@@ -169,8 +169,11 @@ Drivers initiate a graph traversal by calling
 The graph structure, provided by the caller, is initialized to start graph
 traversal at the given pad in an entity.
 
-Drivers can then retrieve the next entity by calling
-:c:func:`media_graph_walk_next()`
+Drivers can then retrieve the next pad by calling
+:c:func:`media_graph_walk_next()`. Only the pad through which the entity
+is first reached is returned. If the caller is interested in knowing which
+further pads would be connected, the :c:func:`media_entity_has_route()`
+function can be used for that.
 
 When the graph traversal is complete the function will return ``NULL``.
 
diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index e3a932ab9abd..06147f38ce2e 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -341,9 +341,9 @@ static void media_graph_walk_iter(struct media_graph *graph)
 	lockdep_assert_held(&next->graph_obj.mdev->graph_mutex);
 }
 
-struct media_entity *media_graph_walk_next(struct media_graph *graph)
+struct media_pad *media_graph_walk_next(struct media_graph *graph)
 {
-	struct media_entity *entity;
+	struct media_pad *pad;
 
 	if (stack_top(graph) == NULL)
 		return NULL;
@@ -356,11 +356,11 @@ struct media_entity *media_graph_walk_next(struct media_graph *graph)
 	while (link_top(graph) != &stack_top(graph)->entity->links)
 		media_graph_walk_iter(graph);
 
-	entity = stack_pop(graph)->entity;
-	dev_dbg(entity->graph_obj.mdev->dev,
-		"walk: returning entity '%s'\n", entity->name);
+	pad = stack_pop(graph);
+	dev_dbg(pad->graph_obj.mdev->dev,
+		"walk: returning pad '%s':%u\n", pad->entity->name, pad->index);
 
-	return entity;
+	return pad;
 }
 EXPORT_SYMBOL_GPL(media_graph_walk_next);
 
@@ -408,7 +408,8 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 {
 	struct media_device *mdev = entity->graph_obj.mdev;
 	struct media_graph *graph = &pipe->graph;
-	struct media_entity *entity_err = entity;
+	struct media_pad *pad = entity->pads;
+	struct media_pad *pad_err = pad;
 	struct media_link *link;
 	int ret;
 
@@ -418,9 +419,11 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 			goto error_graph_walk_start;
 	}
 
-	media_graph_walk_start(&pipe->graph, entity->pads);
+	media_graph_walk_start(&pipe->graph, pad);
+
+	while ((pad = media_graph_walk_next(graph))) {
+		struct media_entity *entity = pad->entity;
 
-	while ((entity = media_graph_walk_next(graph))) {
 		DECLARE_BITMAP(active, MEDIA_ENTITY_MAX_PADS);
 		DECLARE_BITMAP(has_no_links, MEDIA_ENTITY_MAX_PADS);
 
@@ -429,7 +432,7 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 		if (entity->pipe && entity->pipe != pipe) {
 			pr_err("Pipe active for %s. Can't start for %s\n",
 				entity->name,
-				entity_err->name);
+				pad_err->entity->name);
 			ret = -EBUSY;
 			goto error;
 		}
@@ -447,26 +450,27 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 		bitmap_fill(has_no_links, entity->num_pads);
 
 		list_for_each_entry(link, &entity->links, list) {
-			struct media_pad *pad = link->sink->entity == entity
-						? link->sink : link->source;
+			struct media_pad *other_pad =
+				link->sink->entity == entity ?
+				link->sink : link->source;
 
 			/* Mark that a pad is connected by a link. */
-			bitmap_clear(has_no_links, pad->index, 1);
+			bitmap_clear(has_no_links, other_pad->index, 1);
 
 			/*
 			 * Pads that either do not need to connect or
 			 * are connected through an enabled link are
 			 * fine.
 			 */
-			if (!(pad->flags & MEDIA_PAD_FL_MUST_CONNECT) ||
+			if (!(other_pad->flags & MEDIA_PAD_FL_MUST_CONNECT) ||
 			    link->flags & MEDIA_LNK_FL_ENABLED)
-				bitmap_set(active, pad->index, 1);
+				bitmap_set(active, other_pad->index, 1);
 
 			/*
 			 * Link validation will only take place for
 			 * sink ends of the link that are enabled.
 			 */
-			if (link->sink != pad ||
+			if (link->sink != other_pad ||
 			    !(link->flags & MEDIA_LNK_FL_ENABLED))
 				continue;
 
@@ -502,9 +506,11 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 	 * Link validation on graph failed. We revert what we did and
 	 * return the error.
 	 */
-	media_graph_walk_start(graph, entity_err->pads);
+	media_graph_walk_start(graph, pad_err);
+
+	while ((pad_err = media_graph_walk_next(graph))) {
+		struct media_entity *entity_err = pad_err->entity;
 
-	while ((entity_err = media_graph_walk_next(graph))) {
 		/* Sanity check for negative stream_count */
 		if (!WARN_ON_ONCE(entity_err->stream_count <= 0)) {
 			entity_err->stream_count--;
@@ -516,7 +522,7 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 		 * We haven't increased stream_count further than this
 		 * so we quit here.
 		 */
-		if (entity_err == entity)
+		if (pad_err == pad)
 			break;
 	}
 
@@ -543,8 +549,9 @@ EXPORT_SYMBOL_GPL(media_pipeline_start);
 
 void __media_pipeline_stop(struct media_entity *entity)
 {
-	struct media_graph *graph = &entity->pipe->graph;
 	struct media_pipeline *pipe = entity->pipe;
+	struct media_graph *graph = &pipe->graph;
+	struct media_pad *pad;
 
 	/*
 	 * If the following check fails, the driver has performed an
@@ -555,7 +562,9 @@ void __media_pipeline_stop(struct media_entity *entity)
 
 	media_graph_walk_start(graph, entity->pads);
 
-	while ((entity = media_graph_walk_next(graph))) {
+	while ((pad = media_graph_walk_next(graph))) {
+		struct media_entity *entity = pad->entity;
+
 		/* Sanity check for negative stream_count */
 		if (!WARN_ON_ONCE(entity->stream_count <= 0)) {
 			entity->stream_count--;
diff --git a/drivers/media/platform/exynos4-is/media-dev.c b/drivers/media/platform/exynos4-is/media-dev.c
index 4ec7f7a6b0f6..10429466a394 100644
--- a/drivers/media/platform/exynos4-is/media-dev.c
+++ b/drivers/media/platform/exynos4-is/media-dev.c
@@ -1164,7 +1164,7 @@ static int __fimc_md_modify_pipeline(struct media_entity *entity, bool enable)
 static int __fimc_md_modify_pipelines(struct media_entity *entity, bool enable,
 				      struct media_graph *graph)
 {
-	struct media_entity *entity_err = entity;
+	struct media_pad *pad, *pad_err = entity->pads;
 	int ret;
 
 	/*
@@ -1173,13 +1173,13 @@ static int __fimc_md_modify_pipelines(struct media_entity *entity, bool enable,
 	 * through active links. This is needed as we cannot power on/off the
 	 * subdevs in random order.
 	 */
-	media_graph_walk_start(graph, entity->pads);
+	media_graph_walk_start(graph, pad_err);
 
-	while ((entity = media_graph_walk_next(graph))) {
-		if (!is_media_entity_v4l2_video_device(entity))
+	while ((pad = media_graph_walk_next(graph))) {
+		if (!is_media_entity_v4l2_video_device(pad->entity))
 			continue;
 
-		ret  = __fimc_md_modify_pipeline(entity, enable);
+		ret  = __fimc_md_modify_pipeline(pad->entity, enable);
 
 		if (ret < 0)
 			goto err;
@@ -1188,15 +1188,15 @@ static int __fimc_md_modify_pipelines(struct media_entity *entity, bool enable,
 	return 0;
 
 err:
-	media_graph_walk_start(graph, entity_err->pads);
+	media_graph_walk_start(graph, pad_err);
 
-	while ((entity_err = media_graph_walk_next(graph))) {
-		if (!is_media_entity_v4l2_video_device(entity_err))
+	while ((pad_err = media_graph_walk_next(graph))) {
+		if (!is_media_entity_v4l2_video_device(pad_err->entity))
 			continue;
 
-		__fimc_md_modify_pipeline(entity_err, !enable);
+		__fimc_md_modify_pipeline(pad_err->entity, !enable);
 
-		if (entity_err == entity)
+		if (pad_err == pad)
 			break;
 	}
 
diff --git a/drivers/media/platform/omap3isp/ispvideo.c b/drivers/media/platform/omap3isp/ispvideo.c
index 3c1485d59404..5c1cbb1a9003 100644
--- a/drivers/media/platform/omap3isp/ispvideo.c
+++ b/drivers/media/platform/omap3isp/ispvideo.c
@@ -222,8 +222,8 @@ static int isp_video_get_graph_data(struct isp_video *video,
 				    struct isp_pipeline *pipe)
 {
 	struct media_graph graph;
-	struct media_entity *entity = &video->video.entity;
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_pad *pad = video->video.entity.pads;
+	struct media_device *mdev = video->video.entity.graph_obj.mdev;
 	struct isp_video *far_end = NULL;
 	int ret;
 
@@ -234,23 +234,24 @@ static int isp_video_get_graph_data(struct isp_video *video,
 		return ret;
 	}
 
-	media_graph_walk_start(&graph, entity->pads);
+	media_graph_walk_start(&graph, pad);
 
-	while ((entity = media_graph_walk_next(&graph))) {
+	while ((pad = media_graph_walk_next(&graph))) {
 		struct isp_video *__video;
 
-		media_entity_enum_set(&pipe->ent_enum, entity);
+		media_entity_enum_set(&pipe->ent_enum, pad->entity);
 
 		if (far_end != NULL)
 			continue;
 
-		if (entity == &video->video.entity)
+		if (pad == video->video.entity.pads)
 			continue;
 
-		if (!is_media_entity_v4l2_video_device(entity))
+		if (!is_media_entity_v4l2_video_device(pad->entity))
 			continue;
 
-		__video = to_isp_video(media_entity_to_video_device(entity));
+		__video = to_isp_video(media_entity_to_video_device(
+					       pad->entity));
 		if (__video->type != video->type)
 			far_end = __video;
 	}
diff --git a/drivers/media/platform/vsp1/vsp1_video.c b/drivers/media/platform/vsp1/vsp1_video.c
index 61e4fbaba7b7..f2c36f2fdf53 100644
--- a/drivers/media/platform/vsp1/vsp1_video.c
+++ b/drivers/media/platform/vsp1/vsp1_video.c
@@ -559,8 +559,8 @@ static int vsp1_video_pipeline_build(struct vsp1_pipeline *pipe,
 				     struct vsp1_video *video)
 {
 	struct media_graph graph;
-	struct media_entity *entity = &video->video.entity;
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_pad *pad = video->video.entity.pads;
+	struct media_device *mdev = video->video.entity.graph_obj.mdev;
 	unsigned int i;
 	int ret;
 
@@ -569,17 +569,17 @@ static int vsp1_video_pipeline_build(struct vsp1_pipeline *pipe,
 	if (ret)
 		return ret;
 
-	media_graph_walk_start(&graph, entity->pads);
+	media_graph_walk_start(&graph, pad);
 
-	while ((entity = media_graph_walk_next(&graph))) {
+	while ((pad = media_graph_walk_next(&graph))) {
 		struct v4l2_subdev *subdev;
 		struct vsp1_rwpf *rwpf;
 		struct vsp1_entity *e;
 
-		if (!is_media_entity_v4l2_subdev(entity))
+		if (!is_media_entity_v4l2_subdev(pad->entity))
 			continue;
 
-		subdev = media_entity_to_v4l2_subdev(entity);
+		subdev = media_entity_to_v4l2_subdev(pad->entity);
 		e = to_vsp1_entity(subdev);
 		list_add_tail(&e->list_pipe, &pipe->entities);
 		e->pipe = pipe;
diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
index af0d55ab6c15..d33f99c6ffa4 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.c
+++ b/drivers/media/platform/xilinx/xilinx-dma.c
@@ -174,8 +174,8 @@ static int xvip_pipeline_validate(struct xvip_pipeline *pipe,
 				  struct xvip_dma *start)
 {
 	struct media_graph graph;
-	struct media_entity *entity = &start->video.entity;
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_pad *pad = start->video.entity.pads;
+	struct media_device *mdev = pad->entity->graph_obj.mdev;
 	unsigned int num_inputs = 0;
 	unsigned int num_outputs = 0;
 	int ret;
@@ -189,15 +189,15 @@ static int xvip_pipeline_validate(struct xvip_pipeline *pipe,
 		return ret;
 	}
 
-	media_graph_walk_start(&graph, entity->pads);
+	media_graph_walk_start(&graph, pad);
 
-	while ((entity = media_graph_walk_next(&graph))) {
+	while ((pad = media_graph_walk_next(&graph))) {
 		struct xvip_dma *dma;
 
-		if (entity->function != MEDIA_ENT_F_IO_V4L)
+		if (pad->entity->function != MEDIA_ENT_F_IO_V4L)
 			continue;
 
-		dma = to_xvip_dma(media_entity_to_video_device(entity));
+		dma = to_xvip_dma(media_entity_to_video_device(pad->entity));
 
 		if (dma->pad.flags & MEDIA_PAD_FL_SINK) {
 			pipe->output = dma;
diff --git a/drivers/media/v4l2-core/v4l2-mc.c b/drivers/media/v4l2-core/v4l2-mc.c
index d215fe31b9a2..cbeb580c6754 100644
--- a/drivers/media/v4l2-core/v4l2-mc.c
+++ b/drivers/media/v4l2-core/v4l2-mc.c
@@ -434,13 +434,14 @@ EXPORT_SYMBOL_GPL(v4l2_create_fwnode_links);
 static int pipeline_pm_use_count(struct media_entity *entity,
 	struct media_graph *graph)
 {
+	struct media_pad *pad;
 	int use = 0;
 
 	media_graph_walk_start(graph, entity->pads);
 
-	while ((entity = media_graph_walk_next(graph))) {
-		if (is_media_entity_v4l2_video_device(entity))
-			use += entity->use_count;
+	while ((pad = media_graph_walk_next(graph))) {
+		if (is_media_entity_v4l2_video_device(pad->entity))
+			use += pad->entity->use_count;
 	}
 
 	return use;
@@ -493,7 +494,7 @@ static int pipeline_pm_power_one(struct media_entity *entity, int change)
 static int pipeline_pm_power(struct media_entity *entity, int change,
 	struct media_graph *graph)
 {
-	struct media_entity *first = entity;
+	struct media_pad *tmp_pad, *pad;
 	int ret = 0;
 
 	if (!change)
@@ -501,19 +502,18 @@ static int pipeline_pm_power(struct media_entity *entity, int change,
 
 	media_graph_walk_start(graph, entity->pads);
 
-	while (!ret && (entity = media_graph_walk_next(graph)))
-		if (is_media_entity_v4l2_subdev(entity))
-			ret = pipeline_pm_power_one(entity, change);
+	while (!ret && (pad = media_graph_walk_next(graph)))
+		if (is_media_entity_v4l2_subdev(pad->entity))
+			ret = pipeline_pm_power_one(pad->entity, change);
 
 	if (!ret)
 		return ret;
 
-	media_graph_walk_start(graph, first->pads);
+	media_graph_walk_start(graph, entity->pads);
 
-	while ((first = media_graph_walk_next(graph))
-	       && first != entity)
-		if (is_media_entity_v4l2_subdev(first))
-			pipeline_pm_power_one(first, -change);
+	while ((tmp_pad = media_graph_walk_next(graph)) && tmp_pad != pad)
+		if (is_media_entity_v4l2_subdev(tmp_pad->entity))
+			pipeline_pm_power_one(tmp_pad->entity, -change);
 
 	return ret;
 }
diff --git a/drivers/staging/media/omap4iss/iss_video.c b/drivers/staging/media/omap4iss/iss_video.c
index 760cd0ab1feb..8c25ad73a81e 100644
--- a/drivers/staging/media/omap4iss/iss_video.c
+++ b/drivers/staging/media/omap4iss/iss_video.c
@@ -206,8 +206,8 @@ static struct iss_video *
 iss_video_far_end(struct iss_video *video)
 {
 	struct media_graph graph;
-	struct media_entity *entity = &video->video.entity;
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_pad *pad = video->video.entity.pads;
+	struct media_device *mdev = video->video.entity.graph_obj.mdev;
 	struct iss_video *far_end = NULL;
 
 	mutex_lock(&mdev->graph_mutex);
@@ -217,16 +217,17 @@ iss_video_far_end(struct iss_video *video)
 		return NULL;
 	}
 
-	media_graph_walk_start(&graph, entity->pads);
+	media_graph_walk_start(&graph, pad);
 
-	while ((entity = media_graph_walk_next(&graph))) {
-		if (entity == &video->video.entity)
+	while ((pad = media_graph_walk_next(&graph))) {
+		if (pad->entity == &video->video.entity)
 			continue;
 
-		if (!is_media_entity_v4l2_video_device(entity))
+		if (!is_media_entity_v4l2_video_device(pad->entity))
 			continue;
 
-		far_end = to_iss_video(media_entity_to_video_device(entity));
+		far_end = to_iss_video(media_entity_to_video_device(
+						pad->entity));
 		if (far_end->type != video->type)
 			break;
 
@@ -853,8 +854,8 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	struct iss_video_fh *vfh = to_iss_video_fh(fh);
 	struct iss_video *video = video_drvdata(file);
 	struct media_graph graph;
-	struct media_entity *entity = &video->video.entity;
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_pad *pad = video->video.entity.pads;
+	struct media_device *mdev = video->video.entity.graph_obj.mdev;
 	enum iss_pipeline_state state;
 	struct iss_pipeline *pipe;
 	struct iss_video *far_end;
@@ -870,31 +871,32 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	 * Start streaming on the pipeline. No link touching an entity in the
 	 * pipeline can be activated or deactivated once streaming is started.
 	 */
-	pipe = entity->pipe
-	     ? to_iss_pipeline(entity) : &video->pipe;
+	pipe = pad->entity->pipe
+	     ? to_iss_pipeline(pad->entity) : &video->pipe;
 	pipe->external = NULL;
 	pipe->external_rate = 0;
 	pipe->external_bpp = 0;
 
-	ret = media_entity_enum_init(&pipe->ent_enum, entity->graph_obj.mdev);
+	ret = media_entity_enum_init(&pipe->ent_enum,
+				     pad->entity->graph_obj.mdev);
 	if (ret)
 		goto err_graph_walk_init;
 
-	ret = media_graph_walk_init(&graph, entity->graph_obj.mdev);
+	ret = media_graph_walk_init(&graph, pad->entity->graph_obj.mdev);
 	if (ret)
 		goto err_graph_walk_init;
 
 	if (video->iss->pdata->set_constraints)
 		video->iss->pdata->set_constraints(video->iss, true);
 
-	ret = media_pipeline_start(entity, &pipe->pipe);
+	ret = media_pipeline_start(pad->entity, &pipe->pipe);
 	if (ret < 0)
 		goto err_media_pipeline_start;
 
 	mutex_lock(&mdev->graph_mutex);
-	media_graph_walk_start(&graph, entity->pads);
-	while ((entity = media_graph_walk_next(&graph)))
-		media_entity_enum_set(&pipe->ent_enum, entity);
+	media_graph_walk_start(&graph, pad);
+	while ((pad = media_graph_walk_next(&graph)))
+		media_entity_enum_set(&pipe->ent_enum, pad->entity);
 	mutex_unlock(&mdev->graph_mutex);
 
 	/*
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 5b55d6179e13..926fd201eae3 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -926,10 +926,11 @@ void media_graph_walk_start(struct media_graph *graph, struct media_pad *pad);
  * The graph structure must have been previously initialized with a call to
  * media_graph_walk_start().
  *
- * Return: returns the next entity in the graph or %NULL if the whole graph
- * have been traversed.
+ * Return: returns the next entity in the graph, identified by the pad through
+ * which it has been reached. If the whole graph has been traversed, return
+ * %NULL.
  */
-struct media_entity *media_graph_walk_next(struct media_graph *graph);
+struct media_pad *media_graph_walk_next(struct media_graph *graph);
 
 /**
  * media_pipeline_start - Mark a pipeline as streaming
-- 
2.25.1

[PATCH v8 11/36] media: mc: Start walk from a specific pad in use count calculation

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

With the addition of the recent has_route() media entity op, the pads of a
media entity are no longer all interconnected. This has to be taken into
account in power management.

Prepare for the addition of a helper function supporting S_ROUTING.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-mc.c | 27 +++++++++++++--------------
 1 file changed, 13 insertions(+), 14 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-mc.c b/drivers/media/v4l2-core/v4l2-mc.c
index cbeb580c6754..35d18ed89fa5 100644
--- a/drivers/media/v4l2-core/v4l2-mc.c
+++ b/drivers/media/v4l2-core/v4l2-mc.c
@@ -427,17 +427,16 @@ EXPORT_SYMBOL_GPL(v4l2_create_fwnode_links);
 
 /*
  * pipeline_pm_use_count - Count the number of users of a pipeline
- * @entity: The entity
+ * @pad: Any pad along the pipeline
  *
  * Return the total number of users of all video device nodes in the pipeline.
  */
-static int pipeline_pm_use_count(struct media_entity *entity,
-	struct media_graph *graph)
+static int pipeline_pm_use_count(struct media_pad *pad,
+				 struct media_graph *graph)
 {
-	struct media_pad *pad;
 	int use = 0;
 
-	media_graph_walk_start(graph, entity->pads);
+	media_graph_walk_start(graph, pad);
 
 	while ((pad = media_graph_walk_next(graph))) {
 		if (is_media_entity_v4l2_video_device(pad->entity))
@@ -483,7 +482,7 @@ static int pipeline_pm_power_one(struct media_entity *entity, int change)
 
 /*
  * pipeline_pm_power - Apply power change to all entities in a pipeline
- * @entity: The entity
+ * @pad: Any pad along the pipeline
  * @change: Use count change
  *
  * Walk the pipeline to update the use count and the power state of all non-node
@@ -491,16 +490,16 @@ static int pipeline_pm_power_one(struct media_entity *entity, int change)
  *
  * Return 0 on success or a negative error code on failure.
  */
-static int pipeline_pm_power(struct media_entity *entity, int change,
-	struct media_graph *graph)
+static int pipeline_pm_power(struct media_pad *pad, int change,
+			     struct media_graph *graph)
 {
-	struct media_pad *tmp_pad, *pad;
+	struct media_pad *tmp_pad, *first = pad;
 	int ret = 0;
 
 	if (!change)
 		return 0;
 
-	media_graph_walk_start(graph, entity->pads);
+	media_graph_walk_start(graph, pad);
 
 	while (!ret && (pad = media_graph_walk_next(graph)))
 		if (is_media_entity_v4l2_subdev(pad->entity))
@@ -509,7 +508,7 @@ static int pipeline_pm_power(struct media_entity *entity, int change,
 	if (!ret)
 		return ret;
 
-	media_graph_walk_start(graph, entity->pads);
+	media_graph_walk_start(graph, first);
 
 	while ((tmp_pad = media_graph_walk_next(graph)) && tmp_pad != pad)
 		if (is_media_entity_v4l2_subdev(tmp_pad->entity))
@@ -531,7 +530,7 @@ static int v4l2_pipeline_pm_use(struct media_entity *entity, unsigned int use)
 	WARN_ON(entity->use_count < 0);
 
 	/* Apply power change to connected non-nodes. */
-	ret = pipeline_pm_power(entity, change, &mdev->pm_count_walk);
+	ret = pipeline_pm_power(entity->pads, change, &mdev->pm_count_walk);
 	if (ret < 0)
 		entity->use_count -= change;
 
@@ -557,8 +556,8 @@ int v4l2_pipeline_link_notify(struct media_link *link, u32 flags,
 			      unsigned int notification)
 {
 	struct media_graph *graph = &link->graph_obj.mdev->pm_count_walk;
-	struct media_entity *source = link->source->entity;
-	struct media_entity *sink = link->sink->entity;
+	struct media_pad *source = link->source;
+	struct media_pad *sink = link->sink;
 	int source_use;
 	int sink_use;
 	int ret = 0;
-- 
2.25.1

[PATCH v8 12/36] media: entity: Add iterator helper for entity pads

[Tomi Valkeinen]

From: Jacopo Mondi <jacopo+renesas@jmondi.org>

Add an iterator helper to easily cycle through all pads in an entity and
use it in media-entity and media-device code where appropriate.

Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-device.c | 13 ++++++-------
 drivers/media/mc/mc-entity.c | 11 ++++++-----
 include/media/media-entity.h | 12 ++++++++++++
 3 files changed, 24 insertions(+), 12 deletions(-)

diff --git a/drivers/media/mc/mc-device.c b/drivers/media/mc/mc-device.c
index cf5e459b1d96..cb569beab99e 100644
--- a/drivers/media/mc/mc-device.c
+++ b/drivers/media/mc/mc-device.c
@@ -581,7 +581,7 @@ static void __media_device_unregister_entity(struct media_entity *entity)
 	struct media_device *mdev = entity->graph_obj.mdev;
 	struct media_link *link, *tmp;
 	struct media_interface *intf;
-	unsigned int i;
+	struct media_pad *iter;
 
 	ida_free(&mdev->entity_internal_idx, entity->internal_idx);
 
@@ -597,8 +597,8 @@ static void __media_device_unregister_entity(struct media_entity *entity)
 	__media_entity_remove_links(entity);
 
 	/* Remove all pads that belong to this entity */
-	for (i = 0; i < entity->num_pads; i++)
-		media_gobj_destroy(&entity->pads[i].graph_obj);
+	media_entity_for_each_pad(entity, iter)
+		media_gobj_destroy(&iter->graph_obj);
 
 	/* Remove the entity */
 	media_gobj_destroy(&entity->graph_obj);
@@ -617,7 +617,7 @@ int __must_check media_device_register_entity(struct media_device *mdev,
 					      struct media_entity *entity)
 {
 	struct media_entity_notify *notify, *next;
-	unsigned int i;
+	struct media_pad *iter;
 	int ret;
 
 	if (entity->function == MEDIA_ENT_F_V4L2_SUBDEV_UNKNOWN ||
@@ -646,9 +646,8 @@ int __must_check media_device_register_entity(struct media_device *mdev,
 	media_gobj_create(mdev, MEDIA_GRAPH_ENTITY, &entity->graph_obj);
 
 	/* Initialize objects at the pads */
-	for (i = 0; i < entity->num_pads; i++)
-		media_gobj_create(mdev, MEDIA_GRAPH_PAD,
-			       &entity->pads[i].graph_obj);
+	media_entity_for_each_pad(entity, iter)
+		media_gobj_create(mdev, MEDIA_GRAPH_PAD, &iter->graph_obj);
 
 	/* invoke entity_notify callbacks */
 	list_for_each_entry_safe(notify, next, &mdev->entity_notify, list)
diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 06147f38ce2e..2b438c481812 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -198,7 +198,8 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 			   struct media_pad *pads)
 {
 	struct media_device *mdev = entity->graph_obj.mdev;
-	unsigned int i;
+	struct media_pad *iter;
+	unsigned int i = 0;
 
 	if (num_pads >= MEDIA_ENTITY_MAX_PADS)
 		return -E2BIG;
@@ -209,12 +210,12 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 	if (mdev)
 		mutex_lock(&mdev->graph_mutex);
 
-	for (i = 0; i < num_pads; i++) {
-		pads[i].entity = entity;
-		pads[i].index = i;
+	media_entity_for_each_pad(entity, iter) {
+		iter->entity = entity;
+		iter->index = i++;
 		if (mdev)
 			media_gobj_create(mdev, MEDIA_GRAPH_PAD,
-					&entity->pads[i].graph_obj);
+					&iter->graph_obj);
 	}
 
 	if (mdev)
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 926fd201eae3..5f6eed24e63f 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -1107,3 +1107,15 @@ void media_remove_intf_links(struct media_interface *intf);
 	 (entity)->ops->operation((entity) , ##args) : -ENOIOCTLCMD)
 
 #endif
+
+/**
+ * media_entity_for_each_pad - Iterate on all pads in an entity
+ * @entity: The entity the pads belong to
+ * @iter: The iterator pad
+ *
+ * Iterate on all pads in a media entity.
+ */
+#define media_entity_for_each_pad(entity, iter)			\
+	for (iter = (entity)->pads;				\
+	     iter < &(entity)->pads[(entity)->num_pads];	\
+	     ++iter)
-- 
2.25.1

[PATCH v8 13/36] media: entity: Move the pipeline from entity to pads

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

This moves the pipe and stream_count fields from struct media_entity to
struct media_pad. Effectively streams become pad-specific rather than
being entity specific, allowing several independent streams to traverse a
single entity and an entity to be part of several streams.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>

- Update documentation to use 'pads'
- Use the media pad iterator in media_entity.c
- Update rcar-dma.c to use the new per-pad stream count
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>

- Fix cleanup in the error path of __media_pipeline_start()
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c                  | 68 +++++++++++--------
 drivers/media/platform/exynos4-is/fimc-isp.c  |  2 +-
 drivers/media/platform/exynos4-is/fimc-lite.c |  2 +-
 drivers/media/platform/omap3isp/isp.c         |  2 +-
 drivers/media/platform/omap3isp/ispvideo.c    |  2 +-
 drivers/media/platform/omap3isp/ispvideo.h    |  2 +-
 drivers/media/platform/rcar-vin/rcar-core.c   | 16 +++--
 drivers/media/platform/rcar-vin/rcar-dma.c    |  2 +-
 drivers/media/platform/xilinx/xilinx-dma.c    |  2 +-
 drivers/media/platform/xilinx/xilinx-dma.h    |  2 +-
 drivers/staging/media/imx/imx-media-utils.c   |  2 +-
 drivers/staging/media/omap4iss/iss.c          |  2 +-
 drivers/staging/media/omap4iss/iss_video.c    |  2 +-
 drivers/staging/media/omap4iss/iss_video.h    |  2 +-
 include/media/media-entity.h                  | 21 +++---
 15 files changed, 74 insertions(+), 55 deletions(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 2b438c481812..8ad4cb845f4a 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -424,24 +424,30 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 
 	while ((pad = media_graph_walk_next(graph))) {
 		struct media_entity *entity = pad->entity;
+		bool skip_validation = pad->pipe != NULL;
+		struct media_pad *iter;
 
 		DECLARE_BITMAP(active, MEDIA_ENTITY_MAX_PADS);
 		DECLARE_BITMAP(has_no_links, MEDIA_ENTITY_MAX_PADS);
 
-		entity->stream_count++;
+		ret = 0;
 
-		if (entity->pipe && entity->pipe != pipe) {
-			pr_err("Pipe active for %s. Can't start for %s\n",
-				entity->name,
-				pad_err->entity->name);
-			ret = -EBUSY;
-			goto error;
+		media_entity_for_each_pad(entity, iter) {
+			if (iter->pipe && iter->pipe != pipe) {
+				pr_err("Pipe active for %s. Can't start for %s\n",
+				       entity->name, iter->entity->name);
+				ret = -EBUSY;
+			} else {
+				iter->pipe = pipe;
+			}
+			iter->stream_count++;
 		}
 
-		entity->pipe = pipe;
+		if (ret)
+			goto error;
 
-		/* Already streaming --- no need to check. */
-		if (entity->stream_count > 1)
+		/* Already part of the pipeline, skip validation. */
+		if (skip_validation)
 			continue;
 
 		if (!entity->ops || !entity->ops->link_validate)
@@ -510,20 +516,23 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 	media_graph_walk_start(graph, pad_err);
 
 	while ((pad_err = media_graph_walk_next(graph))) {
-		struct media_entity *entity_err = pad_err->entity;
-
-		/* Sanity check for negative stream_count */
-		if (!WARN_ON_ONCE(entity_err->stream_count <= 0)) {
-			entity_err->stream_count--;
-			if (entity_err->stream_count == 0)
-				entity_err->pipe = NULL;
+		struct media_entity *entity = pad_err->entity;
+		struct media_pad *iter;
+
+		media_entity_for_each_pad(entity, iter) {
+			/* Sanity check for negative stream_count */
+			if (!WARN_ON_ONCE(iter->stream_count <= 0)) {
+				--iter->stream_count;
+				if (iter->stream_count == 0)
+					iter->pipe = NULL;
+			}
 		}
 
 		/*
 		 * We haven't increased stream_count further than this
 		 * so we quit here.
 		 */
-		if (pad_err == pad)
+		if (pad_err->entity == pad->entity)
 			break;
 	}
 
@@ -550,7 +559,7 @@ EXPORT_SYMBOL_GPL(media_pipeline_start);
 
 void __media_pipeline_stop(struct media_entity *entity)
 {
-	struct media_pipeline *pipe = entity->pipe;
+	struct media_pipeline *pipe = entity->pads->pipe;
 	struct media_graph *graph = &pipe->graph;
 	struct media_pad *pad;
 
@@ -565,12 +574,15 @@ void __media_pipeline_stop(struct media_entity *entity)
 
 	while ((pad = media_graph_walk_next(graph))) {
 		struct media_entity *entity = pad->entity;
-
-		/* Sanity check for negative stream_count */
-		if (!WARN_ON_ONCE(entity->stream_count <= 0)) {
-			entity->stream_count--;
-			if (entity->stream_count == 0)
-				entity->pipe = NULL;
+		struct media_pad *iter;
+
+		media_entity_for_each_pad(entity, iter) {
+			/* Sanity check for negative stream_count */
+			if (!WARN_ON_ONCE(iter->stream_count <= 0)) {
+				iter->stream_count--;
+				if (iter->stream_count == 0)
+					iter->pipe = NULL;
+			}
 		}
 	}
 
@@ -840,7 +852,7 @@ int __media_entity_setup_link(struct media_link *link, u32 flags)
 {
 	const u32 mask = MEDIA_LNK_FL_ENABLED;
 	struct media_device *mdev;
-	struct media_entity *source, *sink;
+	struct media_pad *source, *sink;
 	int ret = -EBUSY;
 
 	if (link == NULL)
@@ -856,8 +868,8 @@ int __media_entity_setup_link(struct media_link *link, u32 flags)
 	if (link->flags == flags)
 		return 0;
 
-	source = link->source->entity;
-	sink = link->sink->entity;
+	source = link->source;
+	sink = link->sink;
 
 	if (!(link->flags & MEDIA_LNK_FL_DYNAMIC) &&
 	    (source->stream_count || sink->stream_count))
diff --git a/drivers/media/platform/exynos4-is/fimc-isp.c b/drivers/media/platform/exynos4-is/fimc-isp.c
index 855235bea46d..80274e29ccc5 100644
--- a/drivers/media/platform/exynos4-is/fimc-isp.c
+++ b/drivers/media/platform/exynos4-is/fimc-isp.c
@@ -226,7 +226,7 @@ static int fimc_isp_subdev_set_fmt(struct v4l2_subdev *sd,
 			}
 		}
 	} else {
-		if (sd->entity.stream_count == 0) {
+		if (sd->entity.pads->stream_count == 0) {
 			if (fmt->pad == FIMC_ISP_SD_PAD_SINK) {
 				struct v4l2_subdev_format format = *fmt;
 
diff --git a/drivers/media/platform/exynos4-is/fimc-lite.c b/drivers/media/platform/exynos4-is/fimc-lite.c
index aaa3af0493ce..67bfb1ad2ba2 100644
--- a/drivers/media/platform/exynos4-is/fimc-lite.c
+++ b/drivers/media/platform/exynos4-is/fimc-lite.c
@@ -1073,7 +1073,7 @@ static int fimc_lite_subdev_set_fmt(struct v4l2_subdev *sd,
 	mutex_lock(&fimc->lock);
 
 	if ((atomic_read(&fimc->out_path) == FIMC_IO_ISP &&
-	    sd->entity.stream_count > 0) ||
+	    sd->entity.pads->stream_count > 0) ||
 	    (atomic_read(&fimc->out_path) == FIMC_IO_DMA &&
 	    vb2_is_busy(&fimc->vb_queue))) {
 		mutex_unlock(&fimc->lock);
diff --git a/drivers/media/platform/omap3isp/isp.c b/drivers/media/platform/omap3isp/isp.c
index 20f59c59ff8a..28aab16d7662 100644
--- a/drivers/media/platform/omap3isp/isp.c
+++ b/drivers/media/platform/omap3isp/isp.c
@@ -936,7 +936,7 @@ static int isp_pipeline_is_last(struct media_entity *me)
 	struct isp_pipeline *pipe;
 	struct media_pad *pad;
 
-	if (!me->pipe)
+	if (!me->pads->pipe)
 		return 0;
 	pipe = to_isp_pipeline(me);
 	if (pipe->stream_state == ISP_PIPELINE_STREAM_STOPPED)
diff --git a/drivers/media/platform/omap3isp/ispvideo.c b/drivers/media/platform/omap3isp/ispvideo.c
index 5c1cbb1a9003..a8438040c4aa 100644
--- a/drivers/media/platform/omap3isp/ispvideo.c
+++ b/drivers/media/platform/omap3isp/ispvideo.c
@@ -1094,7 +1094,7 @@ isp_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	/* Start streaming on the pipeline. No link touching an entity in the
 	 * pipeline can be activated or deactivated once streaming is started.
 	 */
-	pipe = video->video.entity.pipe
+	pipe = video->video.entity.pads->pipe
 	     ? to_isp_pipeline(&video->video.entity) : &video->pipe;
 
 	ret = media_entity_enum_init(&pipe->ent_enum, &video->isp->media_dev);
diff --git a/drivers/media/platform/omap3isp/ispvideo.h b/drivers/media/platform/omap3isp/ispvideo.h
index a0908670c0cf..4c9c5b719ec5 100644
--- a/drivers/media/platform/omap3isp/ispvideo.h
+++ b/drivers/media/platform/omap3isp/ispvideo.h
@@ -100,7 +100,7 @@ struct isp_pipeline {
 };
 
 #define to_isp_pipeline(__e) \
-	container_of((__e)->pipe, struct isp_pipeline, pipe)
+	container_of((__e)->pads->pipe, struct isp_pipeline, pipe)
 
 static inline int isp_pipeline_ready(struct isp_pipeline *pipe)
 {
diff --git a/drivers/media/platform/rcar-vin/rcar-core.c b/drivers/media/platform/rcar-vin/rcar-core.c
index 33957cc9118c..e59453d6b7c3 100644
--- a/drivers/media/platform/rcar-vin/rcar-core.c
+++ b/drivers/media/platform/rcar-vin/rcar-core.c
@@ -132,13 +132,17 @@ static int rvin_group_link_notify(struct media_link *link, u32 flags,
 		return 0;
 
 	/*
-	 * Don't allow link changes if any entity in the graph is
-	 * streaming, modifying the CHSEL register fields can disrupt
-	 * running streams.
+	 * Don't allow link changes if any stream in the graph is active as
+	 * modifying the CHSEL register fields can disrupt running streams.
 	 */
-	media_device_for_each_entity(entity, &group->mdev)
-		if (entity->stream_count)
-			return -EBUSY;
+	media_device_for_each_entity(entity, &group->mdev) {
+		struct media_pad *iter;
+
+		media_entity_for_each_pad(entity, iter) {
+			if (iter->stream_count)
+				return -EBUSY;
+		}
+	}
 
 	mutex_lock(&group->lock);
 
diff --git a/drivers/media/platform/rcar-vin/rcar-dma.c b/drivers/media/platform/rcar-vin/rcar-dma.c
index f5f722ab1d4e..80b7ae47d165 100644
--- a/drivers/media/platform/rcar-vin/rcar-dma.c
+++ b/drivers/media/platform/rcar-vin/rcar-dma.c
@@ -1231,7 +1231,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
 	 */
 	mdev = vin->vdev.entity.graph_obj.mdev;
 	mutex_lock(&mdev->graph_mutex);
-	pipe = sd->entity.pipe ? sd->entity.pipe : &vin->vdev.pipe;
+	pipe = sd->entity.pads->pipe ? sd->entity.pads->pipe : &vin->vdev.pipe;
 	ret = __media_pipeline_start(&vin->vdev.entity, pipe);
 	mutex_unlock(&mdev->graph_mutex);
 	if (ret)
diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
index d33f99c6ffa4..03ee19d00041 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.c
+++ b/drivers/media/platform/xilinx/xilinx-dma.c
@@ -402,7 +402,7 @@ static int xvip_dma_start_streaming(struct vb2_queue *vq, unsigned int count)
 	 * Use the pipeline object embedded in the first DMA object that starts
 	 * streaming.
 	 */
-	pipe = dma->video.entity.pipe
+	pipe = dma->video.entity.pads->pipe
 	     ? to_xvip_pipeline(&dma->video.entity) : &dma->pipe;
 
 	ret = media_pipeline_start(&dma->video.entity, &pipe->pipe);
diff --git a/drivers/media/platform/xilinx/xilinx-dma.h b/drivers/media/platform/xilinx/xilinx-dma.h
index 2378bdae57ae..69ced71a5696 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.h
+++ b/drivers/media/platform/xilinx/xilinx-dma.h
@@ -47,7 +47,7 @@ struct xvip_pipeline {
 
 static inline struct xvip_pipeline *to_xvip_pipeline(struct media_entity *e)
 {
-	return container_of(e->pipe, struct xvip_pipeline, pipe);
+	return container_of(e->pads->pipe, struct xvip_pipeline, pipe);
 }
 
 /**
diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c
index 6f90acf9c725..535da4dda3c6 100644
--- a/drivers/staging/media/imx/imx-media-utils.c
+++ b/drivers/staging/media/imx/imx-media-utils.c
@@ -913,7 +913,7 @@ int imx_media_pipeline_set_stream(struct imx_media_dev *imxmd,
 			__media_pipeline_stop(entity);
 	} else {
 		v4l2_subdev_call(sd, video, s_stream, 0);
-		if (entity->pipe)
+		if (entity->pads->pipe)
 			__media_pipeline_stop(entity);
 	}
 
diff --git a/drivers/staging/media/omap4iss/iss.c b/drivers/staging/media/omap4iss/iss.c
index 68588e9dab0b..4c6f25aa8b57 100644
--- a/drivers/staging/media/omap4iss/iss.c
+++ b/drivers/staging/media/omap4iss/iss.c
@@ -548,7 +548,7 @@ static int iss_pipeline_is_last(struct media_entity *me)
 	struct iss_pipeline *pipe;
 	struct media_pad *pad;
 
-	if (!me->pipe)
+	if (!me->pads->pipe)
 		return 0;
 	pipe = to_iss_pipeline(me);
 	if (pipe->stream_state == ISS_PIPELINE_STREAM_STOPPED)
diff --git a/drivers/staging/media/omap4iss/iss_video.c b/drivers/staging/media/omap4iss/iss_video.c
index 8c25ad73a81e..b74f7891711d 100644
--- a/drivers/staging/media/omap4iss/iss_video.c
+++ b/drivers/staging/media/omap4iss/iss_video.c
@@ -871,7 +871,7 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	 * Start streaming on the pipeline. No link touching an entity in the
 	 * pipeline can be activated or deactivated once streaming is started.
 	 */
-	pipe = pad->entity->pipe
+	pipe = pad->pipe
 	     ? to_iss_pipeline(pad->entity) : &video->pipe;
 	pipe->external = NULL;
 	pipe->external_rate = 0;
diff --git a/drivers/staging/media/omap4iss/iss_video.h b/drivers/staging/media/omap4iss/iss_video.h
index 526281bf0051..9b8ec27bf87d 100644
--- a/drivers/staging/media/omap4iss/iss_video.h
+++ b/drivers/staging/media/omap4iss/iss_video.h
@@ -91,7 +91,7 @@ struct iss_pipeline {
 };
 
 #define to_iss_pipeline(__e) \
-	container_of((__e)->pipe, struct iss_pipeline, pipe)
+	container_of((__e)->pads->pipe, struct iss_pipeline, pipe)
 
 static inline int iss_pipeline_ready(struct iss_pipeline *pipe)
 {
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 5f6eed24e63f..c9d97c902d05 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -180,15 +180,24 @@ enum media_pad_signal_type {
  *
  * @graph_obj:	Embedded structure containing the media object common data
  * @entity:	Entity this pad belongs to
+ * @pipe:	Pipeline this pad belongs to
+ * @stream_count: Stream count for the pad
  * @index:	Pad index in the entity pads array, numbered from 0 to n
  * @sig_type:	Type of the signal inside a media pad
  * @flags:	Pad flags, as defined in
  *		:ref:`include/uapi/linux/media.h <media_header>`
  *		(seek for ``MEDIA_PAD_FL_*``)
+ * .. note::
+ *
+ *    @stream_count reference count must never be negative, but is a signed
+ *    integer on purpose: a simple ``WARN_ON(<0)`` check can be used to
+ *    detect reference count bugs that would make it negative.
  */
 struct media_pad {
 	struct media_gobj graph_obj;	/* must be first field in struct */
 	struct media_entity *entity;
+	struct media_pipeline *pipe;
+	int stream_count;
 	u16 index;
 	enum media_pad_signal_type sig_type;
 	unsigned long flags;
@@ -267,9 +276,7 @@ enum media_entity_type {
  * @pads:	Pads array with the size defined by @num_pads.
  * @links:	List of data links.
  * @ops:	Entity operations.
- * @stream_count: Stream count for the entity.
  * @use_count:	Use count for the entity.
- * @pipe:	Pipeline this entity belongs to.
  * @info:	Union with devnode information.  Kept just for backward
  *		compatibility.
  * @info.dev:	Contains device major and minor info.
@@ -282,10 +289,9 @@ enum media_entity_type {
  *
  * .. note::
  *
- *    @stream_count and @use_count reference counts must never be
- *    negative, but are signed integers on purpose: a simple ``WARN_ON(<0)``
- *    check can be used to detect reference count bugs that would make them
- *    negative.
+ *    @use_count reference count must never be negative, but is a signed
+ *    integer on purpose: a simple ``WARN_ON(<0)`` check can be used to
+ *    detect reference count bugs that would make it negative.
  */
 struct media_entity {
 	struct media_gobj graph_obj;	/* must be first field in struct */
@@ -304,11 +310,8 @@ struct media_entity {
 
 	const struct media_entity_operations *ops;
 
-	int stream_count;
 	int use_count;
 
-	struct media_pipeline *pipe;
-
 	union {
 		struct {
 			u32 major;
-- 
2.25.1

[PATCH v8 14/36] media: entity: Use pad as the starting point for a pipeline

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

The pipeline has been moved from the entity to the pads; reflect this in
the media pipeline function API.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 Documentation/driver-api/media/mc-core.rst    |  9 +++--
 drivers/media/mc/mc-entity.c                  | 24 ++++++-------
 drivers/media/pci/intel/ipu3/ipu3-cio2-main.c |  6 ++--
 .../media/platform/exynos4-is/fimc-capture.c  |  8 ++---
 .../platform/exynos4-is/fimc-isp-video.c      |  8 ++---
 drivers/media/platform/exynos4-is/fimc-lite.c |  8 ++---
 drivers/media/platform/omap3isp/ispvideo.c    |  6 ++--
 .../media/platform/qcom/camss/camss-video.c   |  6 ++--
 drivers/media/platform/rcar-vin/rcar-dma.c    |  6 ++--
 .../platform/rockchip/rkisp1/rkisp1-capture.c |  6 ++--
 .../media/platform/s3c-camif/camif-capture.c  |  6 ++--
 drivers/media/platform/stm32/stm32-dcmi.c     |  6 ++--
 .../platform/sunxi/sun4i-csi/sun4i_dma.c      |  6 ++--
 .../platform/sunxi/sun6i-csi/sun6i_video.c    |  6 ++--
 drivers/media/platform/ti-vpe/cal-video.c     |  6 ++--
 drivers/media/platform/vsp1/vsp1_video.c      |  6 ++--
 drivers/media/platform/xilinx/xilinx-dma.c    |  6 ++--
 .../media/test-drivers/vimc/vimc-capture.c    |  6 ++--
 drivers/media/usb/au0828/au0828-core.c        |  8 ++---
 drivers/staging/media/imx/imx-media-utils.c   |  6 ++--
 drivers/staging/media/ipu3/ipu3-v4l2.c        |  6 ++--
 drivers/staging/media/omap4iss/iss_video.c    |  6 ++--
 drivers/staging/media/tegra-video/tegra210.c  |  6 ++--
 include/media/media-entity.h                  | 34 +++++++++----------
 24 files changed, 99 insertions(+), 102 deletions(-)

diff --git a/Documentation/driver-api/media/mc-core.rst b/Documentation/driver-api/media/mc-core.rst
index 8a13640bed56..136047a22744 100644
--- a/Documentation/driver-api/media/mc-core.rst
+++ b/Documentation/driver-api/media/mc-core.rst
@@ -213,15 +213,14 @@ When starting streaming, drivers must notify all entities in the pipeline to
 prevent link states from being modified during streaming by calling
 :c:func:`media_pipeline_start()`.
 
-The function will mark all entities connected to the given entity through
-enabled links, either directly or indirectly, as streaming.
+The function will mark all the pads connected to the given pad through
+enabled routes and links, either directly or indirectly, as streaming.
 
 The struct media_pipeline instance pointed to by
-the pipe argument will be stored in every entity in the pipeline.
+the pipe argument will be stored in every pad in the pipeline.
 Drivers should embed the struct media_pipeline
 in higher-level pipeline structures and can then access the
-pipeline through the struct media_entity
-pipe field.
+pipeline through the struct media_pad pipe field.
 
 Calls to :c:func:`media_pipeline_start()` can be nested.
 The pipeline pointer must be identical for all nested calls to the function.
diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 8ad4cb845f4a..b44ab423b49b 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -404,12 +404,11 @@ EXPORT_SYMBOL_GPL(media_entity_get_fwnode_pad);
  * Pipeline management
  */
 
-__must_check int __media_pipeline_start(struct media_entity *entity,
+__must_check int __media_pipeline_start(struct media_pad *pad,
 					struct media_pipeline *pipe)
 {
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_device *mdev = pad->graph_obj.mdev;
 	struct media_graph *graph = &pipe->graph;
-	struct media_pad *pad = entity->pads;
 	struct media_pad *pad_err = pad;
 	struct media_link *link;
 	int ret;
@@ -544,24 +543,23 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 }
 EXPORT_SYMBOL_GPL(__media_pipeline_start);
 
-__must_check int media_pipeline_start(struct media_entity *entity,
+__must_check int media_pipeline_start(struct media_pad *pad,
 				      struct media_pipeline *pipe)
 {
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_device *mdev = pad->graph_obj.mdev;
 	int ret;
 
 	mutex_lock(&mdev->graph_mutex);
-	ret = __media_pipeline_start(entity, pipe);
+	ret = __media_pipeline_start(pad, pipe);
 	mutex_unlock(&mdev->graph_mutex);
 	return ret;
 }
 EXPORT_SYMBOL_GPL(media_pipeline_start);
 
-void __media_pipeline_stop(struct media_entity *entity)
+void __media_pipeline_stop(struct media_pad *pad)
 {
-	struct media_pipeline *pipe = entity->pads->pipe;
+	struct media_pipeline *pipe = pad->pipe;
 	struct media_graph *graph = &pipe->graph;
-	struct media_pad *pad;
 
 	/*
 	 * If the following check fails, the driver has performed an
@@ -570,7 +568,7 @@ void __media_pipeline_stop(struct media_entity *entity)
 	if (WARN_ON(!pipe))
 		return;
 
-	media_graph_walk_start(graph, entity->pads);
+	media_graph_walk_start(graph, pad);
 
 	while ((pad = media_graph_walk_next(graph))) {
 		struct media_entity *entity = pad->entity;
@@ -592,12 +590,12 @@ void __media_pipeline_stop(struct media_entity *entity)
 }
 EXPORT_SYMBOL_GPL(__media_pipeline_stop);
 
-void media_pipeline_stop(struct media_entity *entity)
+void media_pipeline_stop(struct media_pad *pad)
 {
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_device *mdev = pad->graph_obj.mdev;
 
 	mutex_lock(&mdev->graph_mutex);
-	__media_pipeline_stop(entity);
+	__media_pipeline_stop(pad);
 	mutex_unlock(&mdev->graph_mutex);
 }
 EXPORT_SYMBOL_GPL(media_pipeline_stop);
diff --git a/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c b/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c
index 47db0ee0fcbf..74db79b2990f 100644
--- a/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c
+++ b/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c
@@ -981,7 +981,7 @@ static int cio2_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
 		return r;
 	}
 
-	r = media_pipeline_start(&q->vdev.entity, &q->pipe);
+	r = media_pipeline_start(q->vdev.entity.pads, &q->pipe);
 	if (r)
 		goto fail_pipeline;
 
@@ -1001,7 +1001,7 @@ static int cio2_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
 fail_csi2_subdev:
 	cio2_hw_exit(cio2, q);
 fail_hw:
-	media_pipeline_stop(&q->vdev.entity);
+	media_pipeline_stop(q->vdev.entity.pads);
 fail_pipeline:
 	dev_dbg(&cio2->pci_dev->dev, "failed to start streaming (%d)\n", r);
 	cio2_vb2_return_all_buffers(q, VB2_BUF_STATE_QUEUED);
@@ -1022,7 +1022,7 @@ static void cio2_vb2_stop_streaming(struct vb2_queue *vq)
 	cio2_hw_exit(cio2, q);
 	synchronize_irq(cio2->pci_dev->irq);
 	cio2_vb2_return_all_buffers(q, VB2_BUF_STATE_ERROR);
-	media_pipeline_stop(&q->vdev.entity);
+	media_pipeline_stop(q->vdev.entity.pads);
 	pm_runtime_put(&cio2->pci_dev->dev);
 	cio2->streaming = false;
 }
diff --git a/drivers/media/platform/exynos4-is/fimc-capture.c b/drivers/media/platform/exynos4-is/fimc-capture.c
index 7ff4024003f4..eaac0ac4e406 100644
--- a/drivers/media/platform/exynos4-is/fimc-capture.c
+++ b/drivers/media/platform/exynos4-is/fimc-capture.c
@@ -524,7 +524,7 @@ static int fimc_capture_release(struct file *file)
 	mutex_lock(&fimc->lock);
 
 	if (close && vc->streaming) {
-		media_pipeline_stop(&vc->ve.vdev.entity);
+		media_pipeline_stop(vc->ve.vdev.entity.pads);
 		vc->streaming = false;
 	}
 
@@ -1184,7 +1184,7 @@ static int fimc_cap_streamon(struct file *file, void *priv,
 	if (fimc_capture_active(fimc))
 		return -EBUSY;
 
-	ret = media_pipeline_start(entity, &vc->ve.pipe->mp);
+	ret = media_pipeline_start(entity->pads, &vc->ve.pipe->mp);
 	if (ret < 0)
 		return ret;
 
@@ -1218,7 +1218,7 @@ static int fimc_cap_streamon(struct file *file, void *priv,
 	}
 
 err_p_stop:
-	media_pipeline_stop(entity);
+	media_pipeline_stop(entity->pads);
 	return ret;
 }
 
@@ -1234,7 +1234,7 @@ static int fimc_cap_streamoff(struct file *file, void *priv,
 		return ret;
 
 	if (vc->streaming) {
-		media_pipeline_stop(&vc->ve.vdev.entity);
+		media_pipeline_stop(vc->ve.vdev.entity.pads);
 		vc->streaming = false;
 	}
 
diff --git a/drivers/media/platform/exynos4-is/fimc-isp-video.c b/drivers/media/platform/exynos4-is/fimc-isp-video.c
index 83688a7982f7..e2862b3dcdfc 100644
--- a/drivers/media/platform/exynos4-is/fimc-isp-video.c
+++ b/drivers/media/platform/exynos4-is/fimc-isp-video.c
@@ -312,7 +312,7 @@ static int isp_video_release(struct file *file)
 	is_singular_file = v4l2_fh_is_singular_file(file);
 
 	if (is_singular_file && ivc->streaming) {
-		media_pipeline_stop(entity);
+		media_pipeline_stop(entity->pads);
 		ivc->streaming = 0;
 	}
 
@@ -493,7 +493,7 @@ static int isp_video_streamon(struct file *file, void *priv,
 	struct media_entity *me = &ve->vdev.entity;
 	int ret;
 
-	ret = media_pipeline_start(me, &ve->pipe->mp);
+	ret = media_pipeline_start(me->pads, &ve->pipe->mp);
 	if (ret < 0)
 		return ret;
 
@@ -508,7 +508,7 @@ static int isp_video_streamon(struct file *file, void *priv,
 	isp->video_capture.streaming = 1;
 	return 0;
 p_stop:
-	media_pipeline_stop(me);
+	media_pipeline_stop(me->pads);
 	return ret;
 }
 
@@ -523,7 +523,7 @@ static int isp_video_streamoff(struct file *file, void *priv,
 	if (ret < 0)
 		return ret;
 
-	media_pipeline_stop(&video->ve.vdev.entity);
+	media_pipeline_stop(video->ve.vdev.entity.pads);
 	video->streaming = 0;
 	return 0;
 }
diff --git a/drivers/media/platform/exynos4-is/fimc-lite.c b/drivers/media/platform/exynos4-is/fimc-lite.c
index 67bfb1ad2ba2..a979600ff6e6 100644
--- a/drivers/media/platform/exynos4-is/fimc-lite.c
+++ b/drivers/media/platform/exynos4-is/fimc-lite.c
@@ -516,7 +516,7 @@ static int fimc_lite_release(struct file *file)
 	if (v4l2_fh_is_singular_file(file) &&
 	    atomic_read(&fimc->out_path) == FIMC_IO_DMA) {
 		if (fimc->streaming) {
-			media_pipeline_stop(entity);
+			media_pipeline_stop(entity->pads);
 			fimc->streaming = false;
 		}
 		fimc_lite_stop_capture(fimc, false);
@@ -822,7 +822,7 @@ static int fimc_lite_streamon(struct file *file, void *priv,
 	if (fimc_lite_active(fimc))
 		return -EBUSY;
 
-	ret = media_pipeline_start(entity, &fimc->ve.pipe->mp);
+	ret = media_pipeline_start(entity->pads, &fimc->ve.pipe->mp);
 	if (ret < 0)
 		return ret;
 
@@ -839,7 +839,7 @@ static int fimc_lite_streamon(struct file *file, void *priv,
 	}
 
 err_p_stop:
-	media_pipeline_stop(entity);
+	media_pipeline_stop(entity->pads);
 	return 0;
 }
 
@@ -853,7 +853,7 @@ static int fimc_lite_streamoff(struct file *file, void *priv,
 	if (ret < 0)
 		return ret;
 
-	media_pipeline_stop(&fimc->ve.vdev.entity);
+	media_pipeline_stop(fimc->ve.vdev.entity.pads);
 	fimc->streaming = false;
 	return 0;
 }
diff --git a/drivers/media/platform/omap3isp/ispvideo.c b/drivers/media/platform/omap3isp/ispvideo.c
index a8438040c4aa..87334477f223 100644
--- a/drivers/media/platform/omap3isp/ispvideo.c
+++ b/drivers/media/platform/omap3isp/ispvideo.c
@@ -1105,7 +1105,7 @@ isp_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	pipe->l3_ick = clk_get_rate(video->isp->clock[ISP_CLK_L3_ICK]);
 	pipe->max_rate = pipe->l3_ick;
 
-	ret = media_pipeline_start(&video->video.entity, &pipe->pipe);
+	ret = media_pipeline_start(video->video.entity.pads, &pipe->pipe);
 	if (ret < 0)
 		goto err_pipeline_start;
 
@@ -1162,7 +1162,7 @@ isp_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	return 0;
 
 err_check_format:
-	media_pipeline_stop(&video->video.entity);
+	media_pipeline_stop(video->video.entity.pads);
 err_pipeline_start:
 	/* TODO: Implement PM QoS */
 	/* The DMA queue must be emptied here, otherwise CCDC interrupts that
@@ -1229,7 +1229,7 @@ isp_video_streamoff(struct file *file, void *fh, enum v4l2_buf_type type)
 	video->error = false;
 
 	/* TODO: Implement PM QoS */
-	media_pipeline_stop(&video->video.entity);
+	media_pipeline_stop(video->video.entity.pads);
 
 	media_entity_enum_cleanup(&pipe->ent_enum);
 
diff --git a/drivers/media/platform/qcom/camss/camss-video.c b/drivers/media/platform/qcom/camss/camss-video.c
index f282275af626..5cd494f17589 100644
--- a/drivers/media/platform/qcom/camss/camss-video.c
+++ b/drivers/media/platform/qcom/camss/camss-video.c
@@ -491,7 +491,7 @@ static int video_start_streaming(struct vb2_queue *q, unsigned int count)
 	struct v4l2_subdev *subdev;
 	int ret;
 
-	ret = media_pipeline_start(&vdev->entity, &video->pipe);
+	ret = media_pipeline_start(vdev->entity.pads, &video->pipe);
 	if (ret < 0)
 		return ret;
 
@@ -520,7 +520,7 @@ static int video_start_streaming(struct vb2_queue *q, unsigned int count)
 	return 0;
 
 error:
-	media_pipeline_stop(&vdev->entity);
+	media_pipeline_stop(vdev->entity.pads);
 
 	video->ops->flush_buffers(video, VB2_BUF_STATE_QUEUED);
 
@@ -551,7 +551,7 @@ static void video_stop_streaming(struct vb2_queue *q)
 		v4l2_subdev_call(subdev, video, s_stream, 0);
 	}
 
-	media_pipeline_stop(&vdev->entity);
+	media_pipeline_stop(vdev->entity.pads);
 
 	video->ops->flush_buffers(video, VB2_BUF_STATE_ERROR);
 }
diff --git a/drivers/media/platform/rcar-vin/rcar-dma.c b/drivers/media/platform/rcar-vin/rcar-dma.c
index 80b7ae47d165..83b2f923cf98 100644
--- a/drivers/media/platform/rcar-vin/rcar-dma.c
+++ b/drivers/media/platform/rcar-vin/rcar-dma.c
@@ -1215,7 +1215,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
 	sd = media_entity_to_v4l2_subdev(pad->entity);
 
 	if (!on) {
-		media_pipeline_stop(&vin->vdev.entity);
+		media_pipeline_stop(vin->vdev.entity.pads);
 		return v4l2_subdev_call(sd, video, s_stream, 0);
 	}
 
@@ -1232,7 +1232,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
 	mdev = vin->vdev.entity.graph_obj.mdev;
 	mutex_lock(&mdev->graph_mutex);
 	pipe = sd->entity.pads->pipe ? sd->entity.pads->pipe : &vin->vdev.pipe;
-	ret = __media_pipeline_start(&vin->vdev.entity, pipe);
+	ret = __media_pipeline_start(vin->vdev.entity.pads, pipe);
 	mutex_unlock(&mdev->graph_mutex);
 	if (ret)
 		return ret;
@@ -1241,7 +1241,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
 	if (ret == -ENOIOCTLCMD)
 		ret = 0;
 	if (ret)
-		media_pipeline_stop(&vin->vdev.entity);
+		media_pipeline_stop(vin->vdev.entity.pads);
 
 	return ret;
 }
diff --git a/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c b/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
index 41988eb0ec0a..2efbe181b6f1 100644
--- a/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
+++ b/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
@@ -988,7 +988,7 @@ static void rkisp1_vb2_stop_streaming(struct vb2_queue *queue)
 
 	rkisp1_dummy_buf_destroy(cap);
 
-	media_pipeline_stop(&node->vdev.entity);
+	media_pipeline_stop(node->vdev.entity.pads);
 
 	mutex_unlock(&cap->rkisp1->stream_lock);
 }
@@ -1002,7 +1002,7 @@ rkisp1_vb2_start_streaming(struct vb2_queue *queue, unsigned int count)
 
 	mutex_lock(&cap->rkisp1->stream_lock);
 
-	ret = media_pipeline_start(entity, &cap->rkisp1->pipe);
+	ret = media_pipeline_start(entity->pads, &cap->rkisp1->pipe);
 	if (ret) {
 		dev_err(cap->rkisp1->dev, "start pipeline failed %d\n", ret);
 		goto err_ret_buffers;
@@ -1038,7 +1038,7 @@ rkisp1_vb2_start_streaming(struct vb2_queue *queue, unsigned int count)
 err_destroy_dummy:
 	rkisp1_dummy_buf_destroy(cap);
 err_pipeline_stop:
-	media_pipeline_stop(entity);
+	media_pipeline_stop(entity->pads);
 err_ret_buffers:
 	rkisp1_return_all_buffers(cap, VB2_BUF_STATE_QUEUED);
 	mutex_unlock(&cap->rkisp1->stream_lock);
diff --git a/drivers/media/platform/s3c-camif/camif-capture.c b/drivers/media/platform/s3c-camif/camif-capture.c
index 140854ab4dd8..0189b8a33032 100644
--- a/drivers/media/platform/s3c-camif/camif-capture.c
+++ b/drivers/media/platform/s3c-camif/camif-capture.c
@@ -848,13 +848,13 @@ static int s3c_camif_streamon(struct file *file, void *priv,
 	if (s3c_vp_active(vp))
 		return 0;
 
-	ret = media_pipeline_start(sensor, camif->m_pipeline);
+	ret = media_pipeline_start(sensor->pads, camif->m_pipeline);
 	if (ret < 0)
 		return ret;
 
 	ret = camif_pipeline_validate(camif);
 	if (ret < 0) {
-		media_pipeline_stop(sensor);
+		media_pipeline_stop(sensor->pads);
 		return ret;
 	}
 
@@ -878,7 +878,7 @@ static int s3c_camif_streamoff(struct file *file, void *priv,
 
 	ret = vb2_streamoff(&vp->vb_queue, type);
 	if (ret == 0)
-		media_pipeline_stop(&camif->sensor.sd->entity);
+		media_pipeline_stop(camif->sensor.sd->entity.pads);
 	return ret;
 }
 
diff --git a/drivers/media/platform/stm32/stm32-dcmi.c b/drivers/media/platform/stm32/stm32-dcmi.c
index d914ccef9831..457bc838bb73 100644
--- a/drivers/media/platform/stm32/stm32-dcmi.c
+++ b/drivers/media/platform/stm32/stm32-dcmi.c
@@ -730,7 +730,7 @@ static int dcmi_start_streaming(struct vb2_queue *vq, unsigned int count)
 		goto err_unlocked;
 	}
 
-	ret = media_pipeline_start(&dcmi->vdev->entity, &dcmi->pipeline);
+	ret = media_pipeline_start(dcmi->vdev->entity.pads, &dcmi->pipeline);
 	if (ret < 0) {
 		dev_err(dcmi->dev, "%s: Failed to start streaming, media pipeline start error (%d)\n",
 			__func__, ret);
@@ -844,7 +844,7 @@ static int dcmi_start_streaming(struct vb2_queue *vq, unsigned int count)
 	dcmi_pipeline_stop(dcmi);
 
 err_media_pipeline_stop:
-	media_pipeline_stop(&dcmi->vdev->entity);
+	media_pipeline_stop(dcmi->vdev->entity.pads);
 
 err_pm_put:
 	pm_runtime_put(dcmi->dev);
@@ -871,7 +871,7 @@ static void dcmi_stop_streaming(struct vb2_queue *vq)
 
 	dcmi_pipeline_stop(dcmi);
 
-	media_pipeline_stop(&dcmi->vdev->entity);
+	media_pipeline_stop(dcmi->vdev->entity.pads);
 
 	spin_lock_irq(&dcmi->irqlock);
 
diff --git a/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c b/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
index 2c39cd7f2862..be0defdf74f1 100644
--- a/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
+++ b/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
@@ -266,7 +266,7 @@ static int sun4i_csi_start_streaming(struct vb2_queue *vq, unsigned int count)
 		goto err_clear_dma_queue;
 	}
 
-	ret = media_pipeline_start(&csi->vdev.entity, &csi->vdev.pipe);
+	ret = media_pipeline_start(csi->vdev.entity.pads, &csi->vdev.pipe);
 	if (ret < 0)
 		goto err_free_scratch_buffer;
 
@@ -330,7 +330,7 @@ static int sun4i_csi_start_streaming(struct vb2_queue *vq, unsigned int count)
 	sun4i_csi_capture_stop(csi);
 
 err_disable_pipeline:
-	media_pipeline_stop(&csi->vdev.entity);
+	media_pipeline_stop(csi->vdev.entity.pads);
 
 err_free_scratch_buffer:
 	dma_free_coherent(csi->dev, csi->scratch.size, csi->scratch.vaddr,
@@ -359,7 +359,7 @@ static void sun4i_csi_stop_streaming(struct vb2_queue *vq)
 	return_all_buffers(csi, VB2_BUF_STATE_ERROR);
 	spin_unlock_irqrestore(&csi->qlock, flags);
 
-	media_pipeline_stop(&csi->vdev.entity);
+	media_pipeline_stop(csi->vdev.entity.pads);
 
 	dma_free_coherent(csi->dev, csi->scratch.size, csi->scratch.vaddr,
 			  csi->scratch.paddr);
diff --git a/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c b/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c
index 07b2161392d2..6baa597a4dad 100644
--- a/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c
+++ b/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c
@@ -141,7 +141,7 @@ static int sun6i_video_start_streaming(struct vb2_queue *vq, unsigned int count)
 
 	video->sequence = 0;
 
-	ret = media_pipeline_start(&video->vdev.entity, &video->vdev.pipe);
+	ret = media_pipeline_start(video->vdev.entity.pads, &video->vdev.pipe);
 	if (ret < 0)
 		goto clear_dma_queue;
 
@@ -207,7 +207,7 @@ static int sun6i_video_start_streaming(struct vb2_queue *vq, unsigned int count)
 stop_csi_stream:
 	sun6i_csi_set_stream(video->csi, false);
 stop_media_pipeline:
-	media_pipeline_stop(&video->vdev.entity);
+	media_pipeline_stop(video->vdev.entity.pads);
 clear_dma_queue:
 	spin_lock_irqsave(&video->dma_queue_lock, flags);
 	list_for_each_entry(buf, &video->dma_queue, list)
@@ -231,7 +231,7 @@ static void sun6i_video_stop_streaming(struct vb2_queue *vq)
 
 	sun6i_csi_set_stream(video->csi, false);
 
-	media_pipeline_stop(&video->vdev.entity);
+	media_pipeline_stop(video->vdev.entity.pads);
 
 	/* Release all active buffers */
 	spin_lock_irqsave(&video->dma_queue_lock, flags);
diff --git a/drivers/media/platform/ti-vpe/cal-video.c b/drivers/media/platform/ti-vpe/cal-video.c
index 7799da1cc261..861a20ee6885 100644
--- a/drivers/media/platform/ti-vpe/cal-video.c
+++ b/drivers/media/platform/ti-vpe/cal-video.c
@@ -711,7 +711,7 @@ static int cal_start_streaming(struct vb2_queue *vq, unsigned int count)
 	dma_addr_t addr;
 	int ret;
 
-	ret = media_pipeline_start(&ctx->vdev.entity, &ctx->phy->pipe);
+	ret = media_pipeline_start(ctx->vdev.entity.pads, &ctx->phy->pipe);
 	if (ret < 0) {
 		ctx_err(ctx, "Failed to start media pipeline: %d\n", ret);
 		goto error_release_buffers;
@@ -764,7 +764,7 @@ static int cal_start_streaming(struct vb2_queue *vq, unsigned int count)
 	cal_ctx_unprepare(ctx);
 
 error_pipeline:
-	media_pipeline_stop(&ctx->vdev.entity);
+	media_pipeline_stop(ctx->vdev.entity.pads);
 error_release_buffers:
 	cal_release_buffers(ctx, VB2_BUF_STATE_QUEUED);
 
@@ -785,7 +785,7 @@ static void cal_stop_streaming(struct vb2_queue *vq)
 
 	cal_release_buffers(ctx, VB2_BUF_STATE_ERROR);
 
-	media_pipeline_stop(&ctx->vdev.entity);
+	media_pipeline_stop(ctx->vdev.entity.pads);
 }
 
 static const struct vb2_ops cal_video_qops = {
diff --git a/drivers/media/platform/vsp1/vsp1_video.c b/drivers/media/platform/vsp1/vsp1_video.c
index f2c36f2fdf53..978f820b0f34 100644
--- a/drivers/media/platform/vsp1/vsp1_video.c
+++ b/drivers/media/platform/vsp1/vsp1_video.c
@@ -927,7 +927,7 @@ static void vsp1_video_stop_streaming(struct vb2_queue *vq)
 	}
 	mutex_unlock(&pipe->lock);
 
-	media_pipeline_stop(&video->video.entity);
+	media_pipeline_stop(video->video.entity.pads);
 	vsp1_video_release_buffers(video);
 	vsp1_video_pipeline_put(pipe);
 }
@@ -1048,7 +1048,7 @@ vsp1_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 		return PTR_ERR(pipe);
 	}
 
-	ret = __media_pipeline_start(&video->video.entity, &pipe->pipe);
+	ret = __media_pipeline_start(video->video.entity.pads, &pipe->pipe);
 	if (ret < 0) {
 		mutex_unlock(&mdev->graph_mutex);
 		goto err_pipe;
@@ -1072,7 +1072,7 @@ vsp1_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	return 0;
 
 err_stop:
-	media_pipeline_stop(&video->video.entity);
+	media_pipeline_stop(video->video.entity.pads);
 err_pipe:
 	vsp1_video_pipeline_put(pipe);
 	return ret;
diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
index 03ee19d00041..f04b3d190562 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.c
+++ b/drivers/media/platform/xilinx/xilinx-dma.c
@@ -405,7 +405,7 @@ static int xvip_dma_start_streaming(struct vb2_queue *vq, unsigned int count)
 	pipe = dma->video.entity.pads->pipe
 	     ? to_xvip_pipeline(&dma->video.entity) : &dma->pipe;
 
-	ret = media_pipeline_start(&dma->video.entity, &pipe->pipe);
+	ret = media_pipeline_start(dma->video.entity.pads, &pipe->pipe);
 	if (ret < 0)
 		goto error;
 
@@ -431,7 +431,7 @@ static int xvip_dma_start_streaming(struct vb2_queue *vq, unsigned int count)
 	return 0;
 
 error_stop:
-	media_pipeline_stop(&dma->video.entity);
+	media_pipeline_stop(dma->video.entity.pads);
 
 error:
 	/* Give back all queued buffers to videobuf2. */
@@ -459,7 +459,7 @@ static void xvip_dma_stop_streaming(struct vb2_queue *vq)
 
 	/* Cleanup the pipeline and mark it as being stopped. */
 	xvip_pipeline_cleanup(pipe);
-	media_pipeline_stop(&dma->video.entity);
+	media_pipeline_stop(dma->video.entity.pads);
 
 	/* Give back all queued buffers to videobuf2. */
 	spin_lock_irq(&dma->queued_lock);
diff --git a/drivers/media/test-drivers/vimc/vimc-capture.c b/drivers/media/test-drivers/vimc/vimc-capture.c
index 5e9fd902cd37..10724b0a868c 100644
--- a/drivers/media/test-drivers/vimc/vimc-capture.c
+++ b/drivers/media/test-drivers/vimc/vimc-capture.c
@@ -246,7 +246,7 @@ static int vimc_cap_start_streaming(struct vb2_queue *vq, unsigned int count)
 	vcap->sequence = 0;
 
 	/* Start the media pipeline */
-	ret = media_pipeline_start(entity, &vcap->stream.pipe);
+	ret = media_pipeline_start(entity->pads, &vcap->stream.pipe);
 	if (ret) {
 		vimc_cap_return_all_buffers(vcap, VB2_BUF_STATE_QUEUED);
 		return ret;
@@ -254,7 +254,7 @@ static int vimc_cap_start_streaming(struct vb2_queue *vq, unsigned int count)
 
 	ret = vimc_streamer_s_stream(&vcap->stream, &vcap->ved, 1);
 	if (ret) {
-		media_pipeline_stop(entity);
+		media_pipeline_stop(entity->pads);
 		vimc_cap_return_all_buffers(vcap, VB2_BUF_STATE_QUEUED);
 		return ret;
 	}
@@ -273,7 +273,7 @@ static void vimc_cap_stop_streaming(struct vb2_queue *vq)
 	vimc_streamer_s_stream(&vcap->stream, &vcap->ved, 0);
 
 	/* Stop the media pipeline */
-	media_pipeline_stop(&vcap->vdev.entity);
+	media_pipeline_stop(vcap->vdev.entity.pads);
 
 	/* Release all active buffers */
 	vimc_cap_return_all_buffers(vcap, VB2_BUF_STATE_ERROR);
diff --git a/drivers/media/usb/au0828/au0828-core.c b/drivers/media/usb/au0828/au0828-core.c
index caefac07af92..877e85a451cb 100644
--- a/drivers/media/usb/au0828/au0828-core.c
+++ b/drivers/media/usb/au0828/au0828-core.c
@@ -410,7 +410,7 @@ static int au0828_enable_source(struct media_entity *entity,
 		goto end;
 	}
 
-	ret = __media_pipeline_start(entity, pipe);
+	ret = __media_pipeline_start(entity->pads, pipe);
 	if (ret) {
 		pr_err("Start Pipeline: %s->%s Error %d\n",
 			source->name, entity->name, ret);
@@ -501,12 +501,12 @@ static void au0828_disable_source(struct media_entity *entity)
 				return;
 
 			/* stop pipeline */
-			__media_pipeline_stop(dev->active_link_owner);
+			__media_pipeline_stop(dev->active_link_owner->pads);
 			pr_debug("Pipeline stop for %s\n",
 				dev->active_link_owner->name);
 
 			ret = __media_pipeline_start(
-					dev->active_link_user,
+					dev->active_link_user->pads,
 					dev->active_link_user_pipe);
 			if (ret) {
 				pr_err("Start Pipeline: %s->%s %d\n",
@@ -532,7 +532,7 @@ static void au0828_disable_source(struct media_entity *entity)
 			return;
 
 		/* stop pipeline */
-		__media_pipeline_stop(dev->active_link_owner);
+		__media_pipeline_stop(dev->active_link_owner->pads);
 		pr_debug("Pipeline stop for %s\n",
 			dev->active_link_owner->name);
 
diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c
index 535da4dda3c6..74218af45551 100644
--- a/drivers/staging/media/imx/imx-media-utils.c
+++ b/drivers/staging/media/imx/imx-media-utils.c
@@ -905,16 +905,16 @@ int imx_media_pipeline_set_stream(struct imx_media_dev *imxmd,
 	mutex_lock(&imxmd->md.graph_mutex);
 
 	if (on) {
-		ret = __media_pipeline_start(entity, &imxmd->pipe);
+		ret = __media_pipeline_start(entity->pads, &imxmd->pipe);
 		if (ret)
 			goto out;
 		ret = v4l2_subdev_call(sd, video, s_stream, 1);
 		if (ret)
-			__media_pipeline_stop(entity);
+			__media_pipeline_stop(entity->pads);
 	} else {
 		v4l2_subdev_call(sd, video, s_stream, 0);
 		if (entity->pads->pipe)
-			__media_pipeline_stop(entity);
+			__media_pipeline_stop(entity->pads);
 	}
 
 out:
diff --git a/drivers/staging/media/ipu3/ipu3-v4l2.c b/drivers/staging/media/ipu3/ipu3-v4l2.c
index 38a240764509..db5867da3f11 100644
--- a/drivers/staging/media/ipu3/ipu3-v4l2.c
+++ b/drivers/staging/media/ipu3/ipu3-v4l2.c
@@ -485,7 +485,7 @@ static int imgu_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
 
 	pipe = node->pipe;
 	imgu_pipe = &imgu->imgu_pipe[pipe];
-	r = media_pipeline_start(&node->vdev.entity, &imgu_pipe->pipeline);
+	r = media_pipeline_start(node->vdev.entity.pads, &imgu_pipe->pipeline);
 	if (r < 0)
 		goto fail_return_bufs;
 
@@ -510,7 +510,7 @@ static int imgu_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
 	return 0;
 
 fail_stop_pipeline:
-	media_pipeline_stop(&node->vdev.entity);
+	media_pipeline_stop(node->vdev.entity.pads);
 fail_return_bufs:
 	imgu_return_all_buffers(imgu, node, VB2_BUF_STATE_QUEUED);
 
@@ -550,7 +550,7 @@ static void imgu_vb2_stop_streaming(struct vb2_queue *vq)
 	imgu_return_all_buffers(imgu, node, VB2_BUF_STATE_ERROR);
 	mutex_unlock(&imgu->streaming_lock);
 
-	media_pipeline_stop(&node->vdev.entity);
+	media_pipeline_stop(node->vdev.entity.pads);
 }
 
 /******************** v4l2_ioctl_ops ********************/
diff --git a/drivers/staging/media/omap4iss/iss_video.c b/drivers/staging/media/omap4iss/iss_video.c
index b74f7891711d..20fac40581c6 100644
--- a/drivers/staging/media/omap4iss/iss_video.c
+++ b/drivers/staging/media/omap4iss/iss_video.c
@@ -889,7 +889,7 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	if (video->iss->pdata->set_constraints)
 		video->iss->pdata->set_constraints(video->iss, true);
 
-	ret = media_pipeline_start(pad->entity, &pipe->pipe);
+	ret = media_pipeline_start(pad, &pipe->pipe);
 	if (ret < 0)
 		goto err_media_pipeline_start;
 
@@ -980,7 +980,7 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 err_omap4iss_set_stream:
 	vb2_streamoff(&vfh->queue, type);
 err_iss_video_check_format:
-	media_pipeline_stop(&video->video.entity);
+	media_pipeline_stop(video->video.entity.pads);
 err_media_pipeline_start:
 	if (video->iss->pdata->set_constraints)
 		video->iss->pdata->set_constraints(video->iss, false);
@@ -1034,7 +1034,7 @@ iss_video_streamoff(struct file *file, void *fh, enum v4l2_buf_type type)
 
 	if (video->iss->pdata->set_constraints)
 		video->iss->pdata->set_constraints(video->iss, false);
-	media_pipeline_stop(&video->video.entity);
+	media_pipeline_stop(video->video.entity.pads);
 
 done:
 	mutex_unlock(&video->stream_lock);
diff --git a/drivers/staging/media/tegra-video/tegra210.c b/drivers/staging/media/tegra-video/tegra210.c
index f10a041e3e6c..d2d7dd0e8624 100644
--- a/drivers/staging/media/tegra-video/tegra210.c
+++ b/drivers/staging/media/tegra-video/tegra210.c
@@ -547,7 +547,7 @@ static int tegra210_vi_start_streaming(struct vb2_queue *vq, u32 count)
 		       VI_INCR_SYNCPT_NO_STALL);
 
 	/* start the pipeline */
-	ret = media_pipeline_start(&chan->video.entity, pipe);
+	ret = media_pipeline_start(chan->video.entity.pads, pipe);
 	if (ret < 0)
 		goto error_pipeline_start;
 
@@ -595,7 +595,7 @@ static int tegra210_vi_start_streaming(struct vb2_queue *vq, u32 count)
 error_kthread_start:
 	tegra_channel_set_stream(chan, false);
 error_set_stream:
-	media_pipeline_stop(&chan->video.entity);
+	media_pipeline_stop(chan->video.entity.pads);
 error_pipeline_start:
 	tegra_channel_release_buffers(chan, VB2_BUF_STATE_QUEUED);
 	return ret;
@@ -617,7 +617,7 @@ static void tegra210_vi_stop_streaming(struct vb2_queue *vq)
 
 	tegra_channel_release_buffers(chan, VB2_BUF_STATE_ERROR);
 	tegra_channel_set_stream(chan, false);
-	media_pipeline_stop(&chan->video.entity);
+	media_pipeline_stop(chan->video.entity.pads);
 }
 
 /*
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index c9d97c902d05..516d73a2941e 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -937,53 +937,53 @@ struct media_pad *media_graph_walk_next(struct media_graph *graph);
 
 /**
  * media_pipeline_start - Mark a pipeline as streaming
- * @entity: Starting entity
- * @pipe: Media pipeline to be assigned to all entities in the pipeline.
+ * @pad: Starting pad
+ * @pipe: Media pipeline to be assigned to all pads in the pipeline.
  *
- * Mark all entities connected to a given entity through enabled links, either
- * directly or indirectly, as streaming. The given pipeline object is assigned
- * to every entity in the pipeline and stored in the media_entity pipe field.
+ * Mark all pads connected to a given pad through enabled routes or links,
+ * either directly or indirectly, as streaming. The given pipeline object is
+ * assigned to every pad in the pipeline and stored in the media_pad pipe
+ * field.
  *
  * Calls to this function can be nested, in which case the same number of
  * media_pipeline_stop() calls will be required to stop streaming. The
  * pipeline pointer must be identical for all nested calls to
  * media_pipeline_start().
  */
-__must_check int media_pipeline_start(struct media_entity *entity,
+__must_check int media_pipeline_start(struct media_pad *pad,
 				      struct media_pipeline *pipe);
 /**
  * __media_pipeline_start - Mark a pipeline as streaming
  *
- * @entity: Starting entity
- * @pipe: Media pipeline to be assigned to all entities in the pipeline.
+ * @pad: Starting pad
+ * @pipe: Media pipeline to be assigned to all pads in the pipeline.
  *
  * ..note:: This is the non-locking version of media_pipeline_start()
  */
-__must_check int __media_pipeline_start(struct media_entity *entity,
+__must_check int __media_pipeline_start(struct media_pad *pad,
 					struct media_pipeline *pipe);
 
 /**
  * media_pipeline_stop - Mark a pipeline as not streaming
- * @entity: Starting entity
+ * @pad: Starting pad
  *
- * Mark all entities connected to a given entity through enabled links, either
- * directly or indirectly, as not streaming. The media_entity pipe field is
- * reset to %NULL.
+ * Mark all pads connected to a given pad through enabled routes or links,
+ * either directly or indirectly, as not streaming.
  *
  * If multiple calls to media_pipeline_start() have been made, the same
  * number of calls to this function are required to mark the pipeline as not
- * streaming.
+ * streaming and reset the media_pad pipe field to %NULL.
  */
-void media_pipeline_stop(struct media_entity *entity);
+void media_pipeline_stop(struct media_pad *pad);
 
 /**
  * __media_pipeline_stop - Mark a pipeline as not streaming
  *
- * @entity: Starting entity
+ * @pad: Starting pad
  *
  * .. note:: This is the non-locking version of media_pipeline_stop()
  */
-void __media_pipeline_stop(struct media_entity *entity);
+void __media_pipeline_stop(struct media_pad *pad);
 
 /**
  * media_devnode_create() - creates and initializes a device node interface
-- 
2.25.1

[PATCH v8 15/36] media: entity: Add has_route entity operation

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

The optional operation can be used by entities to report whether two
pads are internally connected.

While at there, fix a Sphinx compiler warning in a comment block a few
lines above.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Michal Simek <michal.simek@xilinx.com>
Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 include/media/media-entity.h | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 516d73a2941e..ad4020b2df65 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -187,6 +187,7 @@ enum media_pad_signal_type {
  * @flags:	Pad flags, as defined in
  *		:ref:`include/uapi/linux/media.h <media_header>`
  *		(seek for ``MEDIA_PAD_FL_*``)
+ *
  * .. note::
  *
  *    @stream_count reference count must never be negative, but is a signed
@@ -214,6 +215,10 @@ struct media_pad {
  * @link_validate:	Return whether a link is valid from the entity point of
  *			view. The media_pipeline_start() function
  *			validates all links by calling this operation. Optional.
+ * @has_route:		Return whether a route exists inside the entity between
+ *			two given pads. Pads are passed to the operation ordered
+ *			by index. Optional: If the operation isn't implemented
+ *			all pads will be considered as connected.
  *
  * .. note::
  *
@@ -227,6 +232,8 @@ struct media_entity_operations {
 			  const struct media_pad *local,
 			  const struct media_pad *remote, u32 flags);
 	int (*link_validate)(struct media_link *link);
+	bool (*has_route)(struct media_entity *entity, unsigned int pad0,
+			  unsigned int pad1);
 };
 
 /**
-- 
2.25.1

[PATCH v8 16/36] media: entity: Add media_entity_has_route() function

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

This is a wrapper around the media entity has_route operation.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Michal Simek <michal.simek@xilinx.com>
Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 19 +++++++++++++++++++
 include/media/media-entity.h | 17 +++++++++++++++++
 2 files changed, 36 insertions(+)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index b44ab423b49b..a83f004efd37 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -229,6 +229,25 @@ EXPORT_SYMBOL_GPL(media_entity_pads_init);
  * Graph traversal
  */
 
+bool media_entity_has_route(struct media_entity *entity, unsigned int pad0,
+			    unsigned int pad1)
+{
+	if (pad0 >= entity->num_pads || pad1 >= entity->num_pads)
+		return false;
+
+	if (pad0 == pad1)
+		return true;
+
+	if (!entity->ops || !entity->ops->has_route)
+		return true;
+
+	if (entity->pads[pad1].index < entity->pads[pad0].index)
+		swap(pad0, pad1);
+
+	return entity->ops->has_route(entity, pad0, pad1);
+}
+EXPORT_SYMBOL_GPL(media_entity_has_route);
+
 static struct media_pad *
 media_pad_other(struct media_pad *pad, struct media_link *link)
 {
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index ad4020b2df65..b3069eef7fdb 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -904,6 +904,23 @@ int media_entity_get_fwnode_pad(struct media_entity *entity,
 __must_check int media_graph_walk_init(
 	struct media_graph *graph, struct media_device *mdev);
 
+/**
+ * media_entity_has_route - Check if two entity pads are connected internally
+ *
+ * @entity: The entity
+ * @pad0: The first pad index
+ * @pad1: The second pad index
+ *
+ * This function can be used to check whether two pads of an entity are
+ * connected internally in the entity.
+ *
+ * The caller must hold entity->graph_obj.mdev->mutex.
+ *
+ * Return: true if the pads are connected internally and false otherwise.
+ */
+bool media_entity_has_route(struct media_entity *entity, unsigned int pad0,
+			    unsigned int pad1);
+
 /**
  * media_graph_walk_cleanup - Release resources used by graph walk.
  *
-- 
2.25.1

[PATCH v8 17/36] media: entity: Use routing information during graph traversal

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Take internal routing information as reported by the entity has_route
operation into account during graph traversal to avoid following
unrelated links.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Michal Simek <michal.simek@xilinx.com>
Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 46 ++++++++++++++++++++++--------------
 1 file changed, 28 insertions(+), 18 deletions(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index a83f004efd37..58cdc9c6b342 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -248,15 +248,6 @@ bool media_entity_has_route(struct media_entity *entity, unsigned int pad0,
 }
 EXPORT_SYMBOL_GPL(media_entity_has_route);
 
-static struct media_pad *
-media_pad_other(struct media_pad *pad, struct media_link *link)
-{
-	if (link->source == pad)
-		return link->sink;
-	else
-		return link->source;
-}
-
 /* push an entity to traversal stack */
 static void stack_push(struct media_graph *graph, struct media_pad *pad)
 {
@@ -327,7 +318,8 @@ static void media_graph_walk_iter(struct media_graph *graph)
 {
 	struct media_pad *pad = stack_top(graph);
 	struct media_link *link;
-	struct media_pad *next;
+	struct media_pad *remote;
+	struct media_pad *local;
 
 	link = list_entry(link_top(graph), typeof(*link), list);
 
@@ -341,24 +333,42 @@ static void media_graph_walk_iter(struct media_graph *graph)
 		return;
 	}
 
-	/* Get the entity at the other end of the link. */
-	next = media_pad_other(pad, link);
+	/*
+	 * Get the local pad, the remote pad and the entity at the other
+	 * end of the link.
+	 */
+	if (link->source->entity == pad->entity) {
+		remote = link->sink;
+		local = link->source;
+	} else {
+		remote = link->source;
+		local = link->sink;
+	}
+
+	/*
+	 * Are the local pad and the pad we came from connected
+	 * internally in the entity ?
+	 */
+	if (!media_entity_has_route(pad->entity, pad->index, local->index)) {
+		link_top(graph) = link_top(graph)->next;
+		return;
+	}
 
 	/* Has the entity already been visited? */
-	if (media_entity_enum_test_and_set(&graph->ent_enum, next->entity)) {
+	if (media_entity_enum_test_and_set(&graph->ent_enum, remote->entity)) {
 		link_top(graph) = link_top(graph)->next;
 		dev_dbg(pad->graph_obj.mdev->dev,
 			"walk: skipping entity '%s' (already seen)\n",
-			next->entity->name);
+			remote->entity->name);
 		return;
 	}
 
 	/* Push the new entity to stack and start over. */
 	link_top(graph) = link_top(graph)->next;
-	stack_push(graph, next);
-	dev_dbg(next->graph_obj.mdev->dev, "walk: pushing '%s':%u on stack\n",
-		next->entity->name, next->index);
-	lockdep_assert_held(&next->graph_obj.mdev->graph_mutex);
+	stack_push(graph, remote);
+	dev_dbg(remote->graph_obj.mdev->dev, "walk: pushing '%s':%u on stack\n",
+		remote->entity->name, remote->index);
+	lockdep_assert_held(&remote->graph_obj.mdev->graph_mutex);
 }
 
 struct media_pad *media_graph_walk_next(struct media_graph *graph)
-- 
2.25.1

[PATCH v8 18/36] media: entity: Skip link validation for pads to which there is no route

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

Links are validated along the pipeline which is about to start streaming.
Not all the pads in entities that are traversed along that pipeline are
part of the pipeline, however. Skip the link validation for such pads.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 58cdc9c6b342..e963850b32df 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -489,6 +489,11 @@ __must_check int __media_pipeline_start(struct media_pad *pad,
 				link->sink->entity == entity ?
 				link->sink : link->source;
 
+			/* Ignore pads to which there is no route. */
+			if (!media_entity_has_route(entity, pad->index,
+						    other_pad->index))
+				continue;
+
 			/* Mark that a pad is connected by a link. */
 			bitmap_clear(has_no_links, other_pad->index, 1);
 
-- 
2.25.1

[PATCH v8 19/36] media: entity: Add an iterator helper for connected pads

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

Add a helper macro for iterating over pads that are connected through
enabled routes. This can be used to find all the connected pads within an
entity, for instance starting from the pad which has been obtained during
the graph walk.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>

- Make __media_entity_next_routed_pad() return NULL and adjust the
  iterator to handle that
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 14 ++++++++++++++
 include/media/media-entity.h | 26 ++++++++++++++++++++++++++
 2 files changed, 40 insertions(+)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index e963850b32df..072f017b399a 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -248,6 +248,20 @@ bool media_entity_has_route(struct media_entity *entity, unsigned int pad0,
 }
 EXPORT_SYMBOL_GPL(media_entity_has_route);
 
+struct media_pad *__media_entity_next_routed_pad(struct media_pad *root,
+						 struct media_pad *iter)
+{
+	struct media_entity *entity = root->entity;
+
+	for (; iter < &entity->pads[entity->num_pads]; iter++) {
+		if (media_entity_has_route(entity, root->index, iter->index))
+			return iter;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(__media_entity_next_routed_pad);
+
 /* push an entity to traversal stack */
 static void stack_push(struct media_graph *graph, struct media_pad *pad)
 {
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index b3069eef7fdb..cd1750e495df 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -921,6 +921,32 @@ __must_check int media_graph_walk_init(
 bool media_entity_has_route(struct media_entity *entity, unsigned int pad0,
 			    unsigned int pad1);
 
+/**
+ * __media_entity_next_routed_pad - Get next pad connected to @root
+ *
+ * @root: The root pad to which the iterated pads have a route
+ * @iter: The iterator pad
+ *
+ * Get next pad which has a route to @root.
+ */
+struct media_pad *__media_entity_next_routed_pad(struct media_pad *root,
+						 struct media_pad *iter);
+
+/**
+ * media_entity_for_each_routed_pad - Iterate over entity pads connected by routes
+ *
+ * @root: The root pad to which the iterated pads have a route
+ * @iter: The iterator pad
+ *
+ * Iterate over all pads of an entity which have an internal route to @root pad.
+ * The iteration will include the @root pad itself.
+ */
+#define media_entity_for_each_routed_pad(root, iter)                           \
+	for (iter = __media_entity_next_routed_pad(root,                       \
+						   (root)->entity->pads);      \
+	     iter != NULL;                                                     \
+	     iter = __media_entity_next_routed_pad(root, iter + 1))
+
 /**
  * media_graph_walk_cleanup - Release resources used by graph walk.
  *
-- 
2.25.1

[PATCH v8 20/36] media: entity: Add only connected pads to the pipeline

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

A single entity may contain multiple pipelines. Only add pads that were
connected to the pad through which the entity was reached to the pipeline.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 072f017b399a..4eb4b94c09e2 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -474,7 +474,7 @@ __must_check int __media_pipeline_start(struct media_pad *pad,
 
 		ret = 0;
 
-		media_entity_for_each_pad(entity, iter) {
+		media_entity_for_each_routed_pad(pad, iter) {
 			if (iter->pipe && iter->pipe != pipe) {
 				pr_err("Pipe active for %s. Can't start for %s\n",
 				       entity->name, iter->entity->name);
@@ -563,10 +563,9 @@ __must_check int __media_pipeline_start(struct media_pad *pad,
 	media_graph_walk_start(graph, pad_err);
 
 	while ((pad_err = media_graph_walk_next(graph))) {
-		struct media_entity *entity = pad_err->entity;
 		struct media_pad *iter;
 
-		media_entity_for_each_pad(entity, iter) {
+		media_entity_for_each_routed_pad(pad_err, iter) {
 			/* Sanity check for negative stream_count */
 			if (!WARN_ON_ONCE(iter->stream_count <= 0)) {
 				--iter->stream_count;
@@ -619,10 +618,9 @@ void __media_pipeline_stop(struct media_pad *pad)
 	media_graph_walk_start(graph, pad);
 
 	while ((pad = media_graph_walk_next(graph))) {
-		struct media_entity *entity = pad->entity;
 		struct media_pad *iter;
 
-		media_entity_for_each_pad(entity, iter) {
+		media_entity_for_each_routed_pad(pad, iter) {
 			/* Sanity check for negative stream_count */
 			if (!WARN_ON_ONCE(iter->stream_count <= 0)) {
 				iter->stream_count--;
-- 
2.25.1

[PATCH v8 21/36] media: entity: Add debug information in graph walk route check

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

Add debug printout in graph walk route check.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 4eb4b94c09e2..663773f6bb2f 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -365,6 +365,9 @@ static void media_graph_walk_iter(struct media_graph *graph)
 	 */
 	if (!media_entity_has_route(pad->entity, pad->index, local->index)) {
 		link_top(graph) = link_top(graph)->next;
+		dev_dbg(pad->graph_obj.mdev->dev,
+			"walk: skipping \"%s\":%u -> %u (no route)\n",
+			pad->entity->name, pad->index, local->index);
 		return;
 	}
 
-- 
2.25.1

[PATCH v8 22/36] media: Add bus type to frame descriptors

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

Add the media bus type to the frame descriptor. CSI-2 specific
information will be added in next patch to the frame descriptor.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>

- Make the bus type a named enum
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
---
 include/media/v4l2-subdev.h | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 2290b5025fc0..65471e579815 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -344,12 +344,32 @@ struct v4l2_mbus_frame_desc_entry {
 
 #define V4L2_FRAME_DESC_ENTRY_MAX	4
 
+/**
+ * enum v4l2_mbus_frame_desc_type - media bus frame description type
+ *
+ * @V4L2_MBUS_FRAME_DESC_TYPE_UNDEFINED:
+ *	Undefined frame desc type. Drivers should not use this, it is
+ *	for backwards compatibility.
+ * @V4L2_MBUS_FRAME_DESC_TYPE_PARALLEL:
+ *	Parallel media bus.
+ * @V4L2_MBUS_FRAME_DESC_TYPE_CSI2:
+ *	CSI-2 media bus. Frame desc parameters must be set in
+ *	&struct v4l2_mbus_frame_desc_entry->csi2.
+ */
+enum v4l2_mbus_frame_desc_type {
+	V4L2_MBUS_FRAME_DESC_TYPE_UNDEFINED = 0,
+	V4L2_MBUS_FRAME_DESC_TYPE_PARALLEL,
+	V4L2_MBUS_FRAME_DESC_TYPE_CSI2,
+};
+
 /**
  * struct v4l2_mbus_frame_desc - media bus data frame description
+ * @type: type of the bus (enum v4l2_mbus_frame_desc_type)
  * @entry: frame descriptors array
  * @num_entries: number of entries in @entry array
  */
 struct v4l2_mbus_frame_desc {
+	enum v4l2_mbus_frame_desc_type type;
 	struct v4l2_mbus_frame_desc_entry entry[V4L2_FRAME_DESC_ENTRY_MAX];
 	unsigned short num_entries;
 };
-- 
2.25.1

[PATCH v8 23/36] media: Add CSI-2 bus configuration to frame descriptors

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

Add CSI-2 bus specific configuration to the frame descriptors. This allows
obtaining the virtual channel and data type information for each stream
the transmitter is sending.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
---
 include/media/v4l2-subdev.h | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 65471e579815..3071d202597d 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -312,6 +312,17 @@ struct v4l2_subdev_audio_ops {
 	int (*s_stream)(struct v4l2_subdev *sd, int enable);
 };
 
+/**
+ * struct v4l2_mbus_frame_desc_entry_csi2
+ *
+ * @vc: CSI-2 virtual channel
+ * @dt: CSI-2 data type ID
+ */
+struct v4l2_mbus_frame_desc_entry_csi2 {
+	u8 vc;
+	u8 dt;
+};
+
 /**
  * enum v4l2_mbus_frame_desc_flags - media bus frame description flags
  *
@@ -335,11 +346,16 @@ enum v4l2_mbus_frame_desc_flags {
  *		%FRAME_DESC_FL_BLOB is not set.
  * @length:	number of octets per frame, valid if @flags
  *		%V4L2_MBUS_FRAME_DESC_FL_LEN_MAX is set.
+ * @bus:	Bus-specific frame descriptor parameters
+ * @bus.csi2:	CSI-2-specific bus configuration
  */
 struct v4l2_mbus_frame_desc_entry {
 	enum v4l2_mbus_frame_desc_flags flags;
 	u32 pixelcode;
 	u32 length;
+	union {
+		struct v4l2_mbus_frame_desc_entry_csi2 csi2;
+	} bus;
 };
 
 #define V4L2_FRAME_DESC_ENTRY_MAX	4
-- 
2.25.1

[PATCH v8 24/36] media: Add stream to frame descriptor

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

The stream field identifies the stream this frame descriptor applies to in
routing configuration across a multiplexed link.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 include/media/v4l2-subdev.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 3071d202597d..f232b9f52817 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -342,6 +342,7 @@ enum v4l2_mbus_frame_desc_flags {
  * struct v4l2_mbus_frame_desc_entry - media bus frame description structure
  *
  * @flags:	bitmask flags, as defined by &enum v4l2_mbus_frame_desc_flags.
+ * @stream:	stream in routing configuration
  * @pixelcode:	media bus pixel code, valid if @flags
  *		%FRAME_DESC_FL_BLOB is not set.
  * @length:	number of octets per frame, valid if @flags
@@ -351,6 +352,7 @@ enum v4l2_mbus_frame_desc_flags {
  */
 struct v4l2_mbus_frame_desc_entry {
 	enum v4l2_mbus_frame_desc_flags flags;
+	u32 stream;
 	u32 pixelcode;
 	u32 length;
 	union {
-- 
2.25.1

[PATCH v8 25/36] media: subdev: increase V4L2_FRAME_DESC_ENTRY_MAX to 8

[Tomi Valkeinen]

V4L2_FRAME_DESC_ENTRY_MAX is currently set to 4. In theory it's possible
to have an arbitrary amount of streams in a single pad, so preferably
there should be no hardcoded maximum number.

However, I believe a reasonable max is 8, which would cover a CSI-2 pad
with 4 streams of pixel data and 4 streams of metadata.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
---
 include/media/v4l2-subdev.h | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index f232b9f52817..1c022d1a6896 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -360,7 +360,11 @@ struct v4l2_mbus_frame_desc_entry {
 	} bus;
 };
 
-#define V4L2_FRAME_DESC_ENTRY_MAX	4
+ /*
+  * FIXME: If this number is too small, it should be dropped altogether and the
+  * API switched to a dynamic number of frame descriptor entries.
+  */
+#define V4L2_FRAME_DESC_ENTRY_MAX	8
 
 /**
  * enum v4l2_mbus_frame_desc_type - media bus frame description type
-- 
2.25.1

[PATCH v8 26/36] media: add V4L2_SUBDEV_FL_MULTIPLEXED

[Tomi Valkeinen]

Add subdev flag V4L2_SUBDEV_FL_MULTIPLEXED. It is used to indicate that
the subdev supports the new API with multiplexed streams (routing,
stream configs).

TODO: describe better what the flag does.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 include/media/v4l2-subdev.h | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 1c022d1a6896..356901d8a948 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -890,6 +890,12 @@ struct v4l2_subdev_internal_ops {
  * should set this flag.
  */
 #define V4L2_SUBDEV_FL_HAS_EVENTS		(1U << 3)
+/*
+ * Set this flag if this subdev supports multiplexed streams. This means
+ * that the driver supports routing and handles the stream parameter in its
+ * v4l2_subdev_pad_ops handlers.
+ */
+#define V4L2_SUBDEV_FL_MULTIPLEXED		(1U << 4)
 
 struct regulator_bulk_data;
 
-- 
2.25.1

[PATCH v8 27/36] media: Documentation: Add GS_ROUTING documentation

[Tomi Valkeinen]

From: Jacopo Mondi <jacopo+renesas@jmondi.org>

Add documentation for VIDIOC_SUBDEV_G/S_ROUTING ioctl and add
description of multiplexed media pads and internal routing to the
V4L2-subdev documentation section.

Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 .../userspace-api/media/v4l/dev-subdev.rst    |   2 +
 .../userspace-api/media/v4l/user-func.rst     |   1 +
 .../media/v4l/vidioc-subdev-g-routing.rst     | 146 ++++++++++++++++++
 3 files changed, 149 insertions(+)
 create mode 100644 Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst

diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index fd1de0a73a9f..a67c2749089a 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -29,6 +29,8 @@ will feature a character device node on which ioctls can be called to
 
 -  negotiate image formats on individual pads
 
+-  inspect and modify internal data routing between pads of the same entity
+
 Sub-device character device nodes, conventionally named
 ``/dev/v4l-subdev*``, use major number 81.
 
diff --git a/Documentation/userspace-api/media/v4l/user-func.rst b/Documentation/userspace-api/media/v4l/user-func.rst
index 53e604bd7d60..228c1521f190 100644
--- a/Documentation/userspace-api/media/v4l/user-func.rst
+++ b/Documentation/userspace-api/media/v4l/user-func.rst
@@ -70,6 +70,7 @@ Function Reference
     vidioc-subdev-g-crop
     vidioc-subdev-g-fmt
     vidioc-subdev-g-frame-interval
+    vidioc-subdev-g-routing
     vidioc-subdev-g-selection
     vidioc-subdev-querycap
     vidioc-subscribe-event
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
new file mode 100644
index 000000000000..41f4873c49f7
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
@@ -0,0 +1,146 @@
+.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
+.. c:namespace:: V4L
+
+.. _VIDIOC_SUBDEV_G_ROUTING:
+
+******************************************************
+ioctl VIDIOC_SUBDEV_G_ROUTING, VIDIOC_SUBDEV_S_ROUTING
+******************************************************
+
+Name
+====
+
+VIDIOC_SUBDEV_G_ROUTING - VIDIOC_SUBDEV_S_ROUTING - Get or set routing between streams of media pads in a media entity.
+
+
+Synopsis
+========
+
+.. c:function:: int ioctl( int fd, VIDIOC_SUBDEV_G_ROUTING, struct v4l2_subdev_routing *argp )
+    :name: VIDIOC_SUBDEV_G_ROUTING
+
+.. c:function:: int ioctl( int fd, VIDIOC_SUBDEV_S_ROUTING, struct v4l2_subdev_routing *argp )
+    :name: VIDIOC_SUBDEV_S_ROUTING
+
+
+Arguments
+=========
+
+``fd``
+    File descriptor returned by :ref:`open() <func-open>`.
+
+``argp``
+    Pointer to struct :c:type:`v4l2_subdev_routing`.
+
+
+Description
+===========
+
+These ioctls are used to get and set the routing in a media entity.
+The routing configuration determines the flows of data inside an entity.
+
+Drivers report their current routing tables using the
+``VIDIOC_SUBDEV_G_ROUTING`` ioctl and application may enable or disable routes
+with the VIDIOC_SUBDEV_S_ROUTING ioctl, by adding or removing routes and setting
+or clearing the ``V4L2_SUBDEV_ROUTE_FL_ACTIVE`` flag of the  ``flags`` field of
+a struct :c:type:`v4l2_subdev_route`.
+
+A special case for routing are routes marked with
+``V4L2_SUBDEV_ROUTE_FL_SOURCE`` flag. These routes are used to describe
+source endpoints on sensors and the sink fields are unused.
+
+When inspecting routes through VIDIOC_SUBDEV_G_ROUTING and the application
+provided ``num_routes`` is not big enough to contain all the available routes
+the subdevice exposes, drivers return the ENOSPC error code and adjust the
+value of the ``num_routes`` field. Application should then reserve enough memory
+for all the route entries and call VIDIOC_SUBDEV_G_ROUTING again.
+
+.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
+
+.. c:type:: v4l2_subdev_routing
+
+.. flat-table:: struct v4l2_subdev_routing
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u32
+      - ``which``
+      - Format to modified, from enum
+        :ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
+    * - struct :c:type:`v4l2_subdev_route`
+      - ``routes[]``
+      - Array of struct :c:type:`v4l2_subdev_route` entries
+    * - __u32
+      - ``num_routes``
+      - Number of entries of the routes array
+    * - __u32
+      - ``reserved``\ [5]
+      - Reserved for future extensions. Applications and drivers must set
+	the array to zero.
+
+.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
+
+.. c:type:: v4l2_subdev_route
+
+.. flat-table:: struct v4l2_subdev_route
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u32
+      - ``sink_pad``
+      - Sink pad number.
+    * - __u32
+      - ``sink_stream``
+      - Sink pad stream number.
+    * - __u32
+      - ``source_pad``
+      - Source pad number.
+    * - __u32
+      - ``source_stream``
+      - Source pad stream number.
+    * - __u32
+      - ``flags``
+      - Route enable/disable flags
+	:ref:`v4l2_subdev_routing_flags <v4l2-subdev-routing-flags>`.
+    * - __u32
+      - ``reserved``\ [5]
+      - Reserved for future extensions. Applications and drivers must set
+	the array to zero.
+
+.. tabularcolumns:: |p{6.6cm}|p{2.2cm}|p{8.7cm}|
+
+.. _v4l2-subdev-routing-flags:
+
+.. flat-table:: enum v4l2_subdev_routing_flags
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       3 1 4
+
+    * - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - 0
+      - The route is enabled. Set by applications.
+    * - V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
+      - 1
+      - The route is immutable. Set by the driver.
+    * - V4L2_SUBDEV_ROUTE_FL_SOURCE
+      - 2
+      - The route is a source route, and the ``sink_pad`` and ``sink_stream``
+        fields are unused. Set by the driver.
+
+Return Value
+============
+
+On success 0 is returned, on error -1 and the ``errno`` variable is set
+appropriately. The generic error codes are described at the
+:ref:`Generic Error Codes <gen-errors>` chapter.
+
+ENOSPC
+   The number of provided route entries is less than the available ones.
+
+EINVAL
+   The sink or source pad identifiers reference a non-existing pad, or reference
+   pads of different types (ie. the sink_pad identifiers refers to a source pad)
+   or the sink or source stream identifiers reference a non-existing stream on
+   the sink or source pad.
-- 
2.25.1

[PATCH v8 28/36] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Add support for subdev internal routing. A route is defined as a single
stream from a sink pad to a source pad.

The userspace can configure the routing via two new ioctls,
VIDIOC_SUBDEV_G_ROUTING and VIDIOC_SUBDEV_S_ROUTING, and subdevs can
implement the functionality with v4l2_subdev_pad_ops.set_routing().

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Michal Simek <michal.simek@xilinx.com>

- Add sink and source streams for multiplexed links
- Copy the argument back in case of an error. This is needed to let the
  caller know the number of routes.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>

- Expand and refine documentation.
- Make the 'routes' pointer a __u64 __user pointer so that a compat32
  version of the ioctl is not required.
- Add struct v4l2_subdev_krouting to be used for subdevice operations.

Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>

- Fix typecasing warnings
- Check sink & source pad types
- Add 'which' field
- Add V4L2_SUBDEV_ROUTE_FL_SOURCE
- Routing to subdev state
- Dropped get_routing subdev op

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-ioctl.c  | 25 ++++++++-
 drivers/media/v4l2-core/v4l2-subdev.c | 75 +++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 24 +++++++++
 include/uapi/linux/v4l2-subdev.h      | 57 ++++++++++++++++++++
 4 files changed, 180 insertions(+), 1 deletion(-)

diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index 05d5db3d85e5..8e9315ffcb99 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -16,6 +16,7 @@
 #include <linux/kernel.h>
 #include <linux/version.h>
 
+#include <linux/v4l2-subdev.h>
 #include <linux/videodev2.h>
 
 #include <media/v4l2-common.h>
@@ -3065,6 +3066,21 @@ static int check_array_args(unsigned int cmd, void *parg, size_t *array_size,
 		ret = 1;
 		break;
 	}
+
+	case VIDIOC_SUBDEV_G_ROUTING:
+	case VIDIOC_SUBDEV_S_ROUTING: {
+		struct v4l2_subdev_routing *routing = parg;
+
+		if (routing->num_routes > 256)
+			return -EINVAL;
+
+		*user_ptr = u64_to_user_ptr(routing->routes);
+		*kernel_ptr = (void **)&routing->routes;
+		*array_size = sizeof(struct v4l2_subdev_route)
+			    * routing->num_routes;
+		ret = 1;
+		break;
+	}
 	}
 
 	return ret;
@@ -3328,8 +3344,15 @@ video_usercopy(struct file *file, unsigned int orig_cmd, unsigned long arg,
 	/*
 	 * Some ioctls can return an error, but still have valid
 	 * results that must be returned.
+	 *
+	 * FIXME: subdev IOCTLS are partially handled here and partially in
+	 * v4l2-subdev.c and the 'always_copy' flag can only be set for IOCTLS
+	 * defined here as part of the 'v4l2_ioctls' array. As
+	 * VIDIOC_SUBDEV_G_ROUTING needs to return results to applications even
+	 * in case of failure, but it is not defined here as part of the
+	 * 'v4l2_ioctls' array, insert an ad-hoc check to address that.
 	 */
-	if (err < 0 && !always_copy)
+	if (err < 0 && !always_copy && cmd != VIDIOC_SUBDEV_G_ROUTING)
 		goto out;
 
 out_array_args:
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index b1e65488210d..0e1f325b3159 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -395,6 +395,12 @@ subdev_ioctl_get_state(struct v4l2_subdev *sd, struct v4l2_subdev_fh *subdev_fh,
 		which = ((struct v4l2_subdev_selection *)arg)->which;
 		break;
 	}
+
+	case VIDIOC_SUBDEV_G_ROUTING:
+	case VIDIOC_SUBDEV_S_ROUTING: {
+		which = ((struct v4l2_subdev_routing *)arg)->which;
+		break;
+	}
 	}
 
 	return which == V4L2_SUBDEV_FORMAT_TRY ?
@@ -711,6 +717,74 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 	case VIDIOC_SUBDEV_QUERYSTD:
 		return v4l2_subdev_call(sd, video, querystd, arg);
 
+	case VIDIOC_SUBDEV_G_ROUTING: {
+		struct v4l2_subdev_routing *routing = arg;
+		struct v4l2_subdev_krouting *krouting;
+
+		if (!(sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED))
+			return -ENOIOCTLCMD;
+
+		memset(routing->reserved, 0, sizeof(routing->reserved));
+
+		krouting = &state->routing;
+
+		if (routing->num_routes < krouting->num_routes) {
+			routing->num_routes = krouting->num_routes;
+			return -ENOSPC;
+		}
+
+		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
+		       krouting->routes,
+		       krouting->num_routes * sizeof(*krouting->routes));
+		routing->num_routes = krouting->num_routes;
+
+		return 0;
+	}
+
+	case VIDIOC_SUBDEV_S_ROUTING: {
+		struct v4l2_subdev_routing *routing = arg;
+		struct v4l2_subdev_route *routes =
+			(struct v4l2_subdev_route *)(uintptr_t)routing->routes;
+		struct v4l2_subdev_krouting krouting = {};
+		unsigned int i;
+
+		if (!(sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED))
+			return -ENOIOCTLCMD;
+
+		if (routing->which != V4L2_SUBDEV_FORMAT_TRY && ro_subdev)
+			return -EPERM;
+
+		memset(routing->reserved, 0, sizeof(routing->reserved));
+
+		for (i = 0; i < routing->num_routes; ++i) {
+			const struct v4l2_subdev_route *route = &routes[i];
+			const struct media_pad *pads = sd->entity.pads;
+
+			/* Do not check sink pad for source routes */
+			if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE)) {
+				if (route->sink_pad >= sd->entity.num_pads)
+					return -EINVAL;
+
+				if (!(pads[route->sink_pad].flags &
+				      MEDIA_PAD_FL_SINK))
+					return -EINVAL;
+			}
+
+			if (route->source_pad >= sd->entity.num_pads)
+				return -EINVAL;
+
+			if (!(pads[route->source_pad].flags &
+			      MEDIA_PAD_FL_SOURCE))
+				return -EINVAL;
+		}
+
+		krouting.which = routing->which;
+		krouting.num_routes = routing->num_routes;
+		krouting.routes = routes;
+
+		return v4l2_subdev_call(sd, pad, set_routing, state, &krouting);
+	}
+
 	default:
 		return v4l2_subdev_call(sd, core, ioctl, cmd, arg);
 	}
@@ -975,6 +1049,7 @@ void v4l2_free_subdev_state(struct v4l2_subdev_state *state)
 
 	mutex_destroy(&state->lock);
 
+	kvfree(state->routing.routes);
 	kvfree(state->pads);
 	kfree(state);
 }
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 356901d8a948..cd6aad21ae0c 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -694,12 +694,29 @@ struct v4l2_subdev_pad_config {
 	struct v4l2_rect try_compose;
 };
 
+/**
+ * struct v4l2_subdev_krouting - subdev routing table
+ *
+ * @which: format type (from enum v4l2_subdev_format_whence)
+ * @routes: &struct v4l2_subdev_route
+ * @num_routes: number of routes
+ *
+ * This structure is used to translate arguments received from
+ * VIDIOC_SUBDEV_G/S_ROUTING() ioctl to subdev device drivers operations.
+ */
+struct v4l2_subdev_krouting {
+	u32 which;
+	struct v4l2_subdev_route *routes;
+	unsigned int num_routes;
+};
+
 /**
  * struct v4l2_subdev_state - Used for storing subdev state information.
  *
  * @lock: mutex for the state
  * @which: state type (from enum v4l2_subdev_format_whence)
  * @pads: &struct v4l2_subdev_pad_config array
+ * @routing: routing table for the subdev
  *
  * This structure only needs to be passed to the pad op if the 'which' field
  * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
@@ -709,6 +726,7 @@ struct v4l2_subdev_state {
 	struct mutex lock;
 	u32 which;
 	struct v4l2_subdev_pad_config *pads;
+	struct v4l2_subdev_krouting routing;
 };
 
 /**
@@ -772,6 +790,9 @@ struct v4l2_subdev_state {
  *		     applied to the hardware. The operation shall fail if the
  *		     pad index it has been called on is not valid or in case of
  *		     unrecoverable failures.
+ *
+ * @set_routing: enable or disable data connection routes described in the
+ *		 subdevice routing table.
  */
 struct v4l2_subdev_pad_ops {
 	int (*init_cfg)(struct v4l2_subdev *sd,
@@ -816,6 +837,9 @@ struct v4l2_subdev_pad_ops {
 			       struct v4l2_mbus_config *config);
 	int (*set_mbus_config)(struct v4l2_subdev *sd, unsigned int pad,
 			       struct v4l2_mbus_config *config);
+	int (*set_routing)(struct v4l2_subdev *sd,
+			   struct v4l2_subdev_state *state,
+			   struct v4l2_subdev_krouting *route);
 };
 
 /**
diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
index 658106f5b5dc..3aa623e0e5f9 100644
--- a/include/uapi/linux/v4l2-subdev.h
+++ b/include/uapi/linux/v4l2-subdev.h
@@ -188,6 +188,61 @@ struct v4l2_subdev_capability {
 /* The v4l2 sub-device video device node is registered in read-only mode. */
 #define V4L2_SUBDEV_CAP_RO_SUBDEV		0x00000001
 
+/*
+ * Is the route active? An active route will start when streaming is enabled
+ * on a video node.
+ */
+#define V4L2_SUBDEV_ROUTE_FL_ACTIVE		BIT(0)
+
+/*
+ * Is the route immutable, i.e. can it be activated and inactivated?
+ * Set by the driver.
+ */
+#define V4L2_SUBDEV_ROUTE_FL_IMMUTABLE		BIT(1)
+
+/*
+ * Is the route a source endpoint? A source endpoint route refers to a stream
+ * generated internally by the subdevice (usually a sensor), and thus there
+ * is no sink-side endpoint for the route. The sink_pad and sink_stream
+ * fields are unused.
+ * Set by the driver.
+ */
+#define V4L2_SUBDEV_ROUTE_FL_SOURCE		BIT(2)
+
+/**
+ * struct v4l2_subdev_route - A route inside a subdev
+ *
+ * @sink_pad: the sink pad index
+ * @sink_stream: the sink stream identifier
+ * @source_pad: the source pad index
+ * @source_stream: the source stream identifier
+ * @flags: route flags V4L2_SUBDEV_ROUTE_FL_*
+ * @reserved: drivers and applications must zero this array
+ */
+struct v4l2_subdev_route {
+	__u32 sink_pad;
+	__u32 sink_stream;
+	__u32 source_pad;
+	__u32 source_stream;
+	__u32 flags;
+	__u32 reserved[5];
+};
+
+/**
+ * struct v4l2_subdev_routing - Subdev routing information
+ *
+ * @which: configuration type (from enum v4l2_subdev_format_whence)
+ * @routes: pointer to the routes array
+ * @num_routes: the total number of routes in the routes array
+ * @reserved: drivers and applications must zero this array
+ */
+struct v4l2_subdev_routing {
+	__u32 which;
+	__u64 routes;
+	__u32 num_routes;
+	__u32 reserved[5];
+};
+
 /* Backwards compatibility define --- to be removed */
 #define v4l2_subdev_edid v4l2_edid
 
@@ -203,6 +258,8 @@ struct v4l2_subdev_capability {
 #define VIDIOC_SUBDEV_S_CROP			_IOWR('V', 60, struct v4l2_subdev_crop)
 #define VIDIOC_SUBDEV_G_SELECTION		_IOWR('V', 61, struct v4l2_subdev_selection)
 #define VIDIOC_SUBDEV_S_SELECTION		_IOWR('V', 62, struct v4l2_subdev_selection)
+#define VIDIOC_SUBDEV_G_ROUTING			_IOWR('V', 38, struct v4l2_subdev_routing)
+#define VIDIOC_SUBDEV_S_ROUTING			_IOWR('V', 39, struct v4l2_subdev_routing)
 /* The following ioctls are identical to the ioctls in videodev2.h */
 #define VIDIOC_SUBDEV_G_STD			_IOR('V', 23, v4l2_std_id)
 #define VIDIOC_SUBDEV_S_STD			_IOW('V', 24, v4l2_std_id)
-- 
2.25.1

[PATCH v8 29/36] media: subdev: add v4l2_subdev_has_route()

[Tomi Valkeinen]

Add a v4l2_subdev_has_route() helper function which can be used for
media_entity_operations.has_route op.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 30 +++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 16 ++++++++++++++
 2 files changed, 46 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 0e1f325b3159..7497ad0f6024 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -999,6 +999,36 @@ int v4l2_subdev_link_validate(struct media_link *link)
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
 
+bool v4l2_subdev_has_route(struct media_entity *entity, unsigned int pad0, unsigned int pad1)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct v4l2_subdev_krouting *routing;
+	unsigned int i;
+	struct v4l2_subdev_state *state;
+
+	state = v4l2_subdev_lock_active_state(sd);
+
+	routing = &state->routing;
+
+	for (i = 0; i < routing->num_routes; ++i) {
+		struct v4l2_subdev_route *route = &routing->routes[i];
+
+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
+			continue;
+
+		if ((route->sink_pad == pad0 && route->source_pad == pad1) ||
+		    (route->source_pad == pad0 && route->sink_pad == pad1)) {
+			v4l2_subdev_unlock_state(state);
+			return true;
+		}
+	}
+
+	v4l2_subdev_unlock_state(state);
+
+	return false;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_has_route);
+
 struct v4l2_subdev_state *
 __v4l2_alloc_subdev_state(struct v4l2_subdev *sd,
 			  enum v4l2_subdev_format_whence which,
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index cd6aad21ae0c..858e18aa5b20 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1213,6 +1213,22 @@ int v4l2_subdev_link_validate_default(struct v4l2_subdev *sd,
  */
 int v4l2_subdev_link_validate(struct media_link *link);
 
+/**
+ * v4l2_subdev_has_route - MC has_route implementation for subdevs
+ *
+ * @entity: pointer to &struct media_entity
+ * @pad0: pad number for the first pad
+ * @pad1: pad number for the second pad
+ *
+ * This function looks at the routing in subdev's active state and returns if
+ * there is a route connecting pad0 and pad1.
+ *
+ * This function can be used as implementation for
+ * media_entity_operations.has_route.
+ */
+bool v4l2_subdev_has_route(struct media_entity *entity, unsigned int pad0,
+			   unsigned int pad1);
+
 /**
  * v4l2_alloc_subdev_state - allocate v4l2_subdev_state
  *
-- 
2.25.1

[PATCH v8 30/36] media: subdev: add v4l2_subdev_set_routing helper()

[Tomi Valkeinen]

Add a helper function to set the subdev routing. The helper can be used
from subdev driver's set_routing op to store the routing table.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 31 +++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 16 ++++++++++++++
 2 files changed, 47 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 7497ad0f6024..eb1103afbfbc 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1155,3 +1155,34 @@ void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state)
 	mutex_unlock(&state->lock);
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_unlock_state);
+
+int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_state *state,
+			    struct v4l2_subdev_krouting *routing)
+{
+	struct v4l2_subdev_krouting *dst = &state->routing;
+	const struct v4l2_subdev_krouting *src = routing;
+
+	lockdep_assert_held(&state->lock);
+
+	kvfree(dst->routes);
+	dst->routes = NULL;
+	dst->num_routes = 0;
+
+	if (src->num_routes == 0) {
+		dst->which = src->which;
+	} else {
+		dst->routes = kvmalloc_array(src->num_routes, sizeof(*src->routes),
+					     GFP_KERNEL);
+		if (!dst->routes)
+			return -ENOMEM;
+
+		memcpy(dst->routes, src->routes,
+		       src->num_routes * sizeof(*src->routes));
+		dst->num_routes = src->num_routes;
+		dst->which = src->which;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing);
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 858e18aa5b20..90e9fda6babb 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1436,4 +1436,20 @@ v4l2_subdev_validate_and_lock_state(struct v4l2_subdev *sd,
 	return state;
 }
 
+/**
+ * v4l2_subdev_set_routing() - Set given routing to subdev state
+ * @sd: The subdevice
+ * @state: The subdevice state
+ * @routing: Routing that will be copied to subdev state
+ *
+ * This will release old routing table (if any) from the state, allocate
+ * enough space for the given routing, and copy the routing.
+ *
+ * This can be used from the subdev driver's set_routing op, after validating
+ * the routing.
+ */
+int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_state *state,
+			    struct v4l2_subdev_krouting *routing);
+
 #endif
-- 
2.25.1

[PATCH v8 31/36] media: subdev: add stream based configuration

[Tomi Valkeinen]

Add support to manage configurations (format, crop, compose) per stream,
instead of per pad. This is accomplished with data structures that hold
an array of all subdev's stream configurations.

The number of streams can vary at runtime based on routing. Every time
the routing is changed, the stream configurations need to be
re-initialized.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 .../v4l/vidioc-subdev-enum-frame-interval.rst |   5 +-
 .../v4l/vidioc-subdev-enum-frame-size.rst     |   5 +-
 .../v4l/vidioc-subdev-enum-mbus-code.rst      |   5 +-
 .../media/v4l/vidioc-subdev-g-crop.rst        |   5 +-
 .../media/v4l/vidioc-subdev-g-fmt.rst         |   5 +-
 .../v4l/vidioc-subdev-g-frame-interval.rst    |   5 +-
 .../media/v4l/vidioc-subdev-g-selection.rst   |   5 +-
 drivers/media/v4l2-core/v4l2-subdev.c         | 136 ++++++++++++++++--
 include/media/v4l2-subdev.h                   |  48 +++++++
 include/uapi/linux/v4l2-subdev.h              |  28 +++-
 10 files changed, 225 insertions(+), 22 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
index 3703943b412f..8def4c05d3da 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
@@ -92,7 +92,10 @@ multiple pads of the same sub-device is not defined.
       - Frame intervals to be enumerated, from enum
 	:ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
index c25a9896df0e..3ef361c0dca7 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
@@ -97,7 +97,10 @@ information about try formats.
       - Frame sizes to be enumerated, from enum
 	:ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
index 417f1a19bcc4..248f6f9ee7c5 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
@@ -73,7 +73,10 @@ information about the try formats.
       - ``flags``
       - See :ref:`v4l2-subdev-mbus-code-flags`
     * - __u32
-      - ``reserved``\ [7]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [6]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
index bd15c0a5a66b..1d267f7e7991 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
@@ -96,7 +96,10 @@ modified format should be as close as possible to the original request.
       - ``rect``
       - Crop rectangle boundaries, in pixels.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
index 7acdbb939d89..ed253a1e44b7 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
@@ -102,7 +102,10 @@ should be as close as possible to the original request.
       - Definition of an image format, see :c:type:`v4l2_mbus_framefmt` for
 	details.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
index d7fe7543c506..842f962d2aea 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
@@ -90,7 +90,10 @@ the same sub-device is not defined.
       - ``interval``
       - Period, in seconds, between consecutive video frames.
     * - __u32
-      - ``reserved``\ [9]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [8]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
index f9172a42f036..6b629c19168c 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
@@ -94,7 +94,10 @@ Selection targets and flags are documented in
       - ``r``
       - Selection rectangle, in pixels.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index eb1103afbfbc..3df30e7e0fef 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -150,14 +150,45 @@ static inline int check_pad(struct v4l2_subdev *sd, u32 pad)
 	return 0;
 }
 
-static int check_state_pads(u32 which, struct v4l2_subdev_state *state)
+static int check_state_pads(struct v4l2_subdev *sd, u32 which,
+			    struct v4l2_subdev_state *state)
 {
+	if (sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED)
+		return 0;
+
 	if (which == V4L2_SUBDEV_FORMAT_TRY && (!state || !state->pads))
 		return -EINVAL;
 
 	return 0;
 }
 
+static int check_state_pad_stream(struct v4l2_subdev *sd,
+				  struct v4l2_subdev_state *state, u32 pad,
+				  u32 stream)
+{
+	struct v4l2_mbus_framefmt *fmt;
+
+	if (!(sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED))
+		return 0;
+
+	/*
+	 * We need to take the state lock to access the format, but as we then
+	 * have to unlock, nothing prevents someone changing the state before
+	 * this call thread enters the driver's op and the driver has the
+	 * change to lock the state.
+	 */
+	v4l2_subdev_lock_state(state);
+
+	fmt = v4l2_state_get_stream_format(state, pad, stream);
+
+	v4l2_subdev_unlock_state(state);
+
+	if (!fmt)
+		return -EINVAL;
+
+	return 0;
+}
+
 static inline int check_format(struct v4l2_subdev *sd,
 			       struct v4l2_subdev_state *state,
 			       struct v4l2_subdev_format *format)
@@ -166,7 +197,8 @@ static inline int check_format(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	return check_which(format->which) ? : check_pad(sd, format->pad) ? :
-	       check_state_pads(format->which, state);
+	       check_state_pads(sd, format->which, state) ? :
+	       check_state_pad_stream(sd, state, format->pad, format->stream);
 }
 
 static int call_get_fmt(struct v4l2_subdev *sd,
@@ -193,7 +225,8 @@ static int call_enum_mbus_code(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	return check_which(code->which) ? : check_pad(sd, code->pad) ? :
-	       check_state_pads(code->which, state) ? :
+	       check_state_pads(sd, code->which, state) ? :
+	       check_state_pad_stream(sd, state, code->pad, code->stream) ? :
 	       sd->ops->pad->enum_mbus_code(sd, state, code);
 }
 
@@ -205,7 +238,8 @@ static int call_enum_frame_size(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	return check_which(fse->which) ? : check_pad(sd, fse->pad) ? :
-	       check_state_pads(fse->which, state) ? :
+	       check_state_pads(sd, fse->which, state) ? :
+	       check_state_pad_stream(sd, state, fse->pad, fse->stream) ? :
 	       sd->ops->pad->enum_frame_size(sd, state, fse);
 }
 
@@ -240,7 +274,8 @@ static int call_enum_frame_interval(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	return check_which(fie->which) ? : check_pad(sd, fie->pad) ? :
-	       check_state_pads(fie->which, state) ? :
+	       check_state_pads(sd, fie->which, state) ? :
+	       check_state_pad_stream(sd, state, fie->pad, fie->stream) ? :
 	       sd->ops->pad->enum_frame_interval(sd, state, fie);
 }
 
@@ -252,7 +287,8 @@ static inline int check_selection(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	return check_which(sel->which) ? : check_pad(sd, sel->pad) ? :
-	       check_state_pads(sel->which, state);
+	       check_state_pads(sd, sel->which, state) ? :
+	       check_state_pad_stream(sd, state, sel->pad, sel->stream);
 }
 
 static int call_get_selection(struct v4l2_subdev *sd,
@@ -868,6 +904,67 @@ const struct v4l2_file_operations v4l2_subdev_fops = {
 	.poll = subdev_poll,
 };
 
+static int
+v4l2_init_stream_configs(struct v4l2_subdev_stream_configs *stream_configs,
+			 const struct v4l2_subdev_krouting *routing)
+{
+	u32 num_configs = 0;
+	unsigned int i;
+	u32 format_idx = 0;
+
+	kvfree(stream_configs->configs);
+	stream_configs->configs = NULL;
+	stream_configs->num_configs = 0;
+
+	/* Count number of formats needed */
+	for (i = 0; i < routing->num_routes; ++i) {
+		struct v4l2_subdev_route *route = &routing->routes[i];
+
+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
+			continue;
+
+		/*
+		 * Each route needs a format on both ends of the route, except for
+		 * source streams which only need one format.
+		 */
+		num_configs += (route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE) ? 1 : 2;
+	}
+
+	if (num_configs) {
+		stream_configs->configs =
+			kvcalloc(num_configs, sizeof(*stream_configs->configs),
+				 GFP_KERNEL);
+
+		if (!stream_configs->configs)
+			return -ENOMEM;
+
+		stream_configs->num_configs = num_configs;
+	}
+
+	/* Fill in the 'pad' and stream' value for each item in the array from the routing table */
+	for (i = 0; i < routing->num_routes; ++i) {
+		struct v4l2_subdev_route *route = &routing->routes[i];
+		u32 idx;
+
+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
+			continue;
+
+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE)) {
+			idx = format_idx++;
+
+			stream_configs->configs[idx].pad = route->sink_pad;
+			stream_configs->configs[idx].stream = route->sink_stream;
+		}
+
+		idx = format_idx++;
+
+		stream_configs->configs[idx].pad = route->source_pad;
+		stream_configs->configs[idx].stream = route->source_stream;
+	}
+
+	return 0;
+}
+
 #ifdef CONFIG_MEDIA_CONTROLLER
 
 int v4l2_subdev_get_fwnode_pad_1_to_1(struct media_entity *entity,
@@ -1046,7 +1143,8 @@ __v4l2_alloc_subdev_state(struct v4l2_subdev *sd,
 
 	state->which = which;
 
-	if (sd->entity.num_pads) {
+	/* Drivers that support streams do not need the legacy pad config */
+	if (!(sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED) && sd->entity.num_pads) {
 		state->pads = kvmalloc_array(sd->entity.num_pads,
 					     sizeof(*state->pads),
 					     GFP_KERNEL | __GFP_ZERO);
@@ -1080,6 +1178,7 @@ void v4l2_free_subdev_state(struct v4l2_subdev_state *state)
 	mutex_destroy(&state->lock);
 
 	kvfree(state->routing.routes);
+	kvfree(state->stream_configs.configs);
 	kvfree(state->pads);
 	kfree(state);
 }
@@ -1183,6 +1282,27 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
 		dst->which = src->which;
 	}
 
-	return 0;
+	return v4l2_init_stream_configs(&state->stream_configs, dst);
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing);
+
+struct v4l2_mbus_framefmt *
+v4l2_state_get_stream_format(struct v4l2_subdev_state *state, unsigned int pad,
+			     u32 stream)
+{
+	struct v4l2_subdev_stream_configs *stream_configs;
+	unsigned int i;
+
+	lockdep_assert_held(&state->lock);
+
+	stream_configs = &state->stream_configs;
+
+	for (i = 0; i < stream_configs->num_configs; ++i) {
+		if (stream_configs->configs[i].pad == pad &&
+		    stream_configs->configs[i].stream == stream)
+			return &stream_configs->configs[i].fmt;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(v4l2_state_get_stream_format);
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 90e9fda6babb..5a2c7e2cb561 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -694,6 +694,37 @@ struct v4l2_subdev_pad_config {
 	struct v4l2_rect try_compose;
 };
 
+/**
+ * struct v4l2_subdev_stream_config - Used for storing stream configuration.
+ *
+ * @pad: pad number
+ * @stream: stream number
+ * @fmt: &struct v4l2_mbus_framefmt
+ * @crop: &struct v4l2_rect to be used for crop
+ * @compose: &struct v4l2_rect to be used for compose
+ *
+ * This structure stores configuration for a stream.
+ */
+struct v4l2_subdev_stream_config {
+	u32 pad;
+	u32 stream;
+
+	struct v4l2_mbus_framefmt fmt;
+	struct v4l2_rect crop;
+	struct v4l2_rect compose;
+};
+
+/**
+ * struct v4l2_subdev_stream_configs - A collection of stream configs.
+ *
+ * @num_configs: number of entries in @config.
+ * @configs: an array of &struct v4l2_subdev_stream_configs.
+ */
+struct v4l2_subdev_stream_configs {
+	u32 num_configs;
+	struct v4l2_subdev_stream_config *configs;
+};
+
 /**
  * struct v4l2_subdev_krouting - subdev routing table
  *
@@ -717,6 +748,7 @@ struct v4l2_subdev_krouting {
  * @which: state type (from enum v4l2_subdev_format_whence)
  * @pads: &struct v4l2_subdev_pad_config array
  * @routing: routing table for the subdev
+ * @stream_configs: stream configurations (only for V4L2_SUBDEV_FL_MULTIPLEXED)
  *
  * This structure only needs to be passed to the pad op if the 'which' field
  * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
@@ -727,6 +759,7 @@ struct v4l2_subdev_state {
 	u32 which;
 	struct v4l2_subdev_pad_config *pads;
 	struct v4l2_subdev_krouting routing;
+	struct v4l2_subdev_stream_configs stream_configs;
 };
 
 /**
@@ -1452,4 +1485,19 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
 			    struct v4l2_subdev_state *state,
 			    struct v4l2_subdev_krouting *routing);
 
+/**
+ * v4l2_state_get_stream_format() - Get pointer to a stream format
+ * @state: subdevice state
+ * @pad: pad id
+ * @stream: stream id
+ *
+ * This returns a pointer to &struct v4l2_mbus_framefmt for the given pad +
+ * stream in the subdev state.
+ *
+ * If the state does not contain the given pad + stream, NULL is returned.
+ */
+struct v4l2_mbus_framefmt *
+v4l2_state_get_stream_format(struct v4l2_subdev_state *state, unsigned int pad,
+			     u32 stream);
+
 #endif
diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
index 3aa623e0e5f9..d00ef85db1c7 100644
--- a/include/uapi/linux/v4l2-subdev.h
+++ b/include/uapi/linux/v4l2-subdev.h
@@ -44,13 +44,15 @@ enum v4l2_subdev_format_whence {
  * @which: format type (from enum v4l2_subdev_format_whence)
  * @pad: pad number, as reported by the media API
  * @format: media bus format (format code and frame size)
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_format {
 	__u32 which;
 	__u32 pad;
 	struct v4l2_mbus_framefmt format;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 /**
@@ -58,13 +60,15 @@ struct v4l2_subdev_format {
  * @which: format type (from enum v4l2_subdev_format_whence)
  * @pad: pad number, as reported by the media API
  * @rect: pad crop rectangle boundaries
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_crop {
 	__u32 which;
 	__u32 pad;
 	struct v4l2_rect rect;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 #define V4L2_SUBDEV_MBUS_CODE_CSC_COLORSPACE	0x00000001
@@ -80,6 +84,7 @@ struct v4l2_subdev_crop {
  * @code: format code (MEDIA_BUS_FMT_ definitions)
  * @which: format type (from enum v4l2_subdev_format_whence)
  * @flags: flags set by the driver, (V4L2_SUBDEV_MBUS_CODE_*)
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_mbus_code_enum {
@@ -88,7 +93,8 @@ struct v4l2_subdev_mbus_code_enum {
 	__u32 code;
 	__u32 which;
 	__u32 flags;
-	__u32 reserved[7];
+	__u32 stream;
+	__u32 reserved[6];
 };
 
 /**
@@ -101,6 +107,7 @@ struct v4l2_subdev_mbus_code_enum {
  * @min_height: minimum frame height, in pixels
  * @max_height: maximum frame height, in pixels
  * @which: format type (from enum v4l2_subdev_format_whence)
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_frame_size_enum {
@@ -112,19 +119,22 @@ struct v4l2_subdev_frame_size_enum {
 	__u32 min_height;
 	__u32 max_height;
 	__u32 which;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 /**
  * struct v4l2_subdev_frame_interval - Pad-level frame rate
  * @pad: pad number, as reported by the media API
  * @interval: frame interval in seconds
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_frame_interval {
 	__u32 pad;
 	struct v4l2_fract interval;
-	__u32 reserved[9];
+	__u32 stream;
+	__u32 reserved[8];
 };
 
 /**
@@ -136,6 +146,7 @@ struct v4l2_subdev_frame_interval {
  * @height: frame height in pixels
  * @interval: frame interval in seconds
  * @which: format type (from enum v4l2_subdev_format_whence)
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_frame_interval_enum {
@@ -146,7 +157,8 @@ struct v4l2_subdev_frame_interval_enum {
 	__u32 height;
 	struct v4l2_fract interval;
 	__u32 which;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 /**
@@ -158,6 +170,7 @@ struct v4l2_subdev_frame_interval_enum {
  *	    defined in v4l2-common.h; V4L2_SEL_TGT_* .
  * @flags: constraint flags, defined in v4l2-common.h; V4L2_SEL_FLAG_*.
  * @r: coordinates of the selection window
+ * @stream: stream number, defined in subdev routing
  * @reserved: for future use, set to zero for now
  *
  * Hardware may use multiple helper windows to process a video stream.
@@ -170,7 +183,8 @@ struct v4l2_subdev_selection {
 	__u32 target;
 	__u32 flags;
 	struct v4l2_rect r;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 /**
-- 
2.25.1

[PATCH v8 32/36] media: subdev: use streams in v4l2_subdev_link_validate()

[Tomi Valkeinen]

Update v4l2_subdev_link_validate() to use routing and streams for
validation.

Instead of just looking at the format on the pad on both ends of the
link, the routing tables are used to collect all the streams going from
the source to the sink over the link, and the streams' formats on both
ends of the link are verified.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 217 +++++++++++++++++++++++---
 1 file changed, 199 insertions(+), 18 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 3df30e7e0fef..e8bfca595e11 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -16,6 +16,7 @@
 #include <linux/videodev2.h>
 #include <linux/export.h>
 #include <linux/version.h>
+#include <linux/sort.h>
 
 #include <media/v4l2-ctrls.h>
 #include <media/v4l2-device.h>
@@ -1047,6 +1048,7 @@ EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate_default);
 
 static int
 v4l2_subdev_link_validate_get_format(struct media_pad *pad,
+				     u32 stream,
 				     struct v4l2_subdev_format *fmt)
 {
 	if (is_media_entity_v4l2_subdev(pad->entity)) {
@@ -1058,6 +1060,7 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
 
 		fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
 		fmt->pad = pad->index;
+		fmt->stream = stream;
 		return v4l2_subdev_call(sd, pad, get_fmt, state, fmt);
 	}
 
@@ -1068,31 +1071,209 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
 	return -EINVAL;
 }
 
-int v4l2_subdev_link_validate(struct media_link *link)
+static int cmp_u32(const void *a, const void *b)
 {
-	struct v4l2_subdev *sink;
-	struct v4l2_subdev_format sink_fmt, source_fmt;
-	int rval;
+	u32 a32 = *(u32 *)a;
+	u32 b32 = *(u32 *)b;
 
-	rval = v4l2_subdev_link_validate_get_format(
-		link->source, &source_fmt);
-	if (rval < 0)
-		return 0;
+	return a32 > b32 ? 1 : (a32 < b32 ? -1 : 0);
+}
+
+static int v4l2_link_validate_get_streams(struct media_link *link,
+					  bool is_source, u32 *out_num_streams,
+					  const u32 **out_streams,
+					  bool *allocated)
+{
+	static const u32 default_streams[] = { 0 };
+	struct v4l2_subdev_krouting *routing;
+	struct v4l2_subdev *subdev;
+	u32 num_streams;
+	u32 *streams;
+	unsigned int i;
+	struct v4l2_subdev_state *state;
+
+	if (is_source)
+		subdev = media_entity_to_v4l2_subdev(link->source->entity);
+	else
+		subdev = media_entity_to_v4l2_subdev(link->sink->entity);
 
-	rval = v4l2_subdev_link_validate_get_format(
-		link->sink, &sink_fmt);
-	if (rval < 0)
+	if (!(subdev->flags & V4L2_SUBDEV_FL_MULTIPLEXED)) {
+		*out_num_streams = 1;
+		*out_streams = default_streams;
+		*allocated = false;
 		return 0;
+	}
 
-	sink = media_entity_to_v4l2_subdev(link->sink->entity);
+	state = v4l2_subdev_lock_active_state(subdev);
 
-	rval = v4l2_subdev_call(sink, pad, link_validate, link,
-				&source_fmt, &sink_fmt);
-	if (rval != -ENOIOCTLCMD)
-		return rval;
+	routing = &state->routing;
+
+	streams = kmalloc_array(routing->num_routes, sizeof(u32), GFP_KERNEL);
+
+	if (!streams) {
+		v4l2_subdev_unlock_state(state);
+		return -ENOMEM;
+	}
+
+	num_streams = 0;
+
+	for (i = 0; i < routing->num_routes; ++i) {
+		struct v4l2_subdev_route *route = &routing->routes[i];
+		int j;
+		u32 route_pad;
+		u32 route_stream;
+		u32 link_pad;
+
+		if (is_source) {
+			route_pad = route->source_pad;
+			route_stream = route->source_stream;
+			link_pad = link->source->index;
+		} else {
+			route_pad = route->sink_pad;
+			route_stream = route->sink_stream;
+			link_pad = link->sink->index;
+		}
+
+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
+			continue;
+
+		if (route_pad != link_pad)
+			continue;
+
+		/* look for duplicates... */
+		for (j = 0; j < num_streams; ++j) {
+			if (streams[j] == route_stream)
+				break;
+		}
+
+		/* ...and drop the stream if we already have it */
+		if (j != num_streams)
+			continue;
+
+		streams[num_streams++] = route_stream;
+	}
+
+	v4l2_subdev_unlock_state(state);
+
+	sort(streams, num_streams, sizeof(u32), &cmp_u32, NULL);
+
+	*out_num_streams = num_streams;
+	*out_streams = streams;
+	*allocated = true;
 
-	return v4l2_subdev_link_validate_default(
-		sink, link, &source_fmt, &sink_fmt);
+	return 0;
+}
+
+int v4l2_subdev_link_validate(struct media_link *link)
+{
+	struct v4l2_subdev *sink_subdev =
+		media_entity_to_v4l2_subdev(link->sink->entity);
+	struct device *dev = sink_subdev->entity.graph_obj.mdev->dev;
+	u32 num_source_streams;
+	const u32 *source_streams;
+	bool source_allocated;
+	u32 num_sink_streams;
+	const u32 *sink_streams;
+	bool sink_allocated;
+	unsigned int sink_idx;
+	unsigned int source_idx;
+	int ret;
+
+	dev_dbg(dev, "validating link \"%s\":%u -> \"%s\":%u\n",
+		link->source->entity->name, link->source->index,
+		link->sink->entity->name, link->sink->index);
+
+	ret = v4l2_link_validate_get_streams(link, true, &num_source_streams,
+					     &source_streams,
+					     &source_allocated);
+	if (ret)
+		return ret;
+
+	ret = v4l2_link_validate_get_streams(link, false, &num_sink_streams,
+					     &sink_streams, &sink_allocated);
+	if (ret)
+		goto free_source;
+
+	/* It is ok to have more source streams than sink streams */
+	if (num_source_streams < num_sink_streams) {
+		dev_err(dev,
+			"Not enough source streams: %d < %d\n",
+			num_source_streams, num_sink_streams);
+		ret = -EINVAL;
+		goto out;
+	}
+
+	/* Validate source and sink stream formats */
+
+	source_idx = 0;
+
+	for (sink_idx = 0; sink_idx < num_sink_streams; ++sink_idx) {
+		struct v4l2_subdev_format sink_fmt, source_fmt;
+		u32 stream;
+
+		stream = sink_streams[sink_idx];
+
+		for (; source_idx < num_source_streams; ++source_idx) {
+			if (source_streams[source_idx] == stream)
+				break;
+		}
+
+		if (source_idx == num_source_streams) {
+			dev_err(dev, "No source stream for sink stream %u\n",
+				stream);
+			ret = -EINVAL;
+			goto out;
+		}
+
+		dev_dbg(dev, "validating stream \"%s\":%u:%u -> \"%s\":%u:%u\n",
+			link->source->entity->name, link->source->index, stream,
+			link->sink->entity->name, link->sink->index, stream);
+
+		ret = v4l2_subdev_link_validate_get_format(link->source, stream,
+							   &source_fmt);
+		if (ret < 0) {
+			dev_dbg(dev, "Failed to get format for \"%s\":%u:%u (but that's ok)\n",
+				link->source->entity->name, link->source->index,
+				stream);
+			ret = 0;
+			continue;
+		}
+
+		ret = v4l2_subdev_link_validate_get_format(link->sink, stream,
+							   &sink_fmt);
+		if (ret < 0) {
+			dev_dbg(dev, "Failed to get format for \"%s\":%u:%u (but that's ok)\n",
+				link->sink->entity->name, link->sink->index,
+				stream);
+			ret = 0;
+			continue;
+		}
+
+		/* TODO: add stream number to link_validate() */
+		ret = v4l2_subdev_call(sink_subdev, pad, link_validate, link,
+				       &source_fmt, &sink_fmt);
+		if (!ret)
+			continue;
+
+		if (ret != -ENOIOCTLCMD)
+			goto out;
+
+		ret = v4l2_subdev_link_validate_default(sink_subdev, link,
+							&source_fmt, &sink_fmt);
+
+		if (ret)
+			goto out;
+	}
+
+out:
+	if (sink_allocated)
+		kfree(sink_streams);
+
+free_source:
+	if (source_allocated)
+		kfree(source_streams);
+
+	return ret;
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
 
-- 
2.25.1

[PATCH v8 33/36] media: subdev: add "opposite" stream helper funcs

[Tomi Valkeinen]

Add two helper functions to make dealing with streams easier:

v4l2_state_find_opposite_end - given a routing table and a pad + stream,
return the pad + stream on the opposite side of the subdev.

v4l2_state_get_opposite_stream_format - return a pointer to the format
on the pad + stream on the opposite side from the given pad + stream.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 42 +++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 31 ++++++++++++++++++++
 2 files changed, 73 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index e8bfca595e11..68db3f23c256 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1487,3 +1487,45 @@ v4l2_state_get_stream_format(struct v4l2_subdev_state *state, unsigned int pad,
 	return NULL;
 }
 EXPORT_SYMBOL_GPL(v4l2_state_get_stream_format);
+
+int v4l2_state_find_opposite_end(struct v4l2_subdev_krouting *routing, u32 pad,
+				 u32 stream, u32 *other_pad, u32 *other_stream)
+{
+	unsigned int i;
+
+	for (i = 0; i < routing->num_routes; ++i) {
+		struct v4l2_subdev_route *route = &routing->routes[i];
+
+		if (route->source_pad == pad &&
+		    route->source_stream == stream) {
+			*other_pad = route->sink_pad;
+			*other_stream = route->sink_stream;
+			return 0;
+		}
+
+		if (route->sink_pad == pad && route->sink_stream == stream) {
+			*other_pad = route->source_pad;
+			*other_stream = route->source_stream;
+			return 0;
+		}
+	}
+
+	return -EINVAL;
+}
+EXPORT_SYMBOL_GPL(v4l2_state_find_opposite_end);
+
+struct v4l2_mbus_framefmt *
+v4l2_state_get_opposite_stream_format(struct v4l2_subdev_state *state, u32 pad,
+				      u32 stream)
+{
+	u32 other_pad, other_stream;
+	int ret;
+
+	ret = v4l2_state_find_opposite_end(&state->routing, pad, stream,
+					   &other_pad, &other_stream);
+	if (ret)
+		return NULL;
+
+	return v4l2_state_get_stream_format(state, other_pad, other_stream);
+}
+EXPORT_SYMBOL_GPL(v4l2_state_get_opposite_stream_format);
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 5a2c7e2cb561..bc69123f71cb 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1500,4 +1500,35 @@ struct v4l2_mbus_framefmt *
 v4l2_state_get_stream_format(struct v4l2_subdev_state *state, unsigned int pad,
 			     u32 stream);
 
+/**
+ * v4l2_state_find_opposite_end() - Find the opposite stream
+ * @routing: routing used to find the opposite side
+ * @pad: pad id
+ * @stream: stream id
+ * @other_pad: pointer used to return the opposite pad
+ * @other_stream: pointer used to return the opposite stream
+ *
+ * This function uses the routing table to find the pad + stream which is
+ * opposite the given pad + stream.
+ *
+ * Returns 0 on success, or -EINVAL if no matching route is found.
+ */
+int v4l2_state_find_opposite_end(struct v4l2_subdev_krouting *routing, u32 pad,
+				 u32 stream, u32 *other_pad, u32 *other_stream);
+
+/**
+ * v4l2_state_get_opposite_stream_format() - Get pointer to opposite stream format
+ * @state: subdevice state
+ * @pad: pad id
+ * @stream: stream id
+ *
+ * This returns a pointer to &struct v4l2_mbus_framefmt for the pad + stream
+ * that is opposite the given pad + stream in the subdev state.
+ *
+ * If the state does not contain the given pad + stream, NULL is returned.
+ */
+struct v4l2_mbus_framefmt *
+v4l2_state_get_opposite_stream_format(struct v4l2_subdev_state *state, u32 pad,
+				      u32 stream);
+
 #endif
-- 
2.25.1

[PATCH v8 34/36] media: subdev: add v4l2_subdev_get_fmt() helper function

[Tomi Valkeinen]

Add v4l2_subdev_get_fmt() helper function which implements
v4l2_subdev_pad_ops.get_fmt using streams. Subdev drivers that do not
need to do anything special in their get_fmt op can use this helper
directly for v4l2_subdev_pad_ops.get_fmt.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 21 +++++++++++++++++++++
 include/media/v4l2-subdev.h           | 16 ++++++++++++++++
 2 files changed, 37 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 68db3f23c256..df3e376e439c 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1529,3 +1529,24 @@ v4l2_state_get_opposite_stream_format(struct v4l2_subdev_state *state, u32 pad,
 	return v4l2_state_get_stream_format(state, other_pad, other_stream);
 }
 EXPORT_SYMBOL_GPL(v4l2_state_get_opposite_stream_format);
+
+int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
+			struct v4l2_subdev_format *format)
+{
+	struct v4l2_mbus_framefmt *fmt;
+
+	v4l2_subdev_lock_state(state);
+
+	fmt = v4l2_state_get_stream_format(state, format->pad, format->stream);
+	if (!fmt) {
+		v4l2_subdev_unlock_state(state);
+		return -EINVAL;
+	}
+
+	format->format = *fmt;
+
+	v4l2_subdev_unlock_state(state);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_get_fmt);
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index bc69123f71cb..c3e859077557 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1530,5 +1530,21 @@ int v4l2_state_find_opposite_end(struct v4l2_subdev_krouting *routing, u32 pad,
 struct v4l2_mbus_framefmt *
 v4l2_state_get_opposite_stream_format(struct v4l2_subdev_state *state, u32 pad,
 				      u32 stream);
+/**
+ * v4l2_subdev_get_fmt() - Fill format based on state
+ * @sd: subdevice
+ * @state: subdevice state
+ * @format: pointer to &struct v4l2_subdev_format
+ *
+ * Fill @format based on the pad and stream given in the @format struct.
+ *
+ * This function can be used by the subdev drivers to implement
+ * v4l2_subdev_pad_ops.get_fmt if the subdev driver does not need to do
+ * anything special in their get_fmt op.
+ *
+ * Returns 0 on success, error value otherwise.
+ */
+int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
+			struct v4l2_subdev_format *format);
 
 #endif
-- 
2.25.1

[PATCH v8 35/36] media: subdev: add v4l2_subdev_set_routing_with_fmt() helper

[Tomi Valkeinen]

v4l2_subdev_set_routing_with_fmt() is the same as
v4l2_subdev_set_routing(), but additionally initializes all the streams
with the given format.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 22 ++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 15 +++++++++++++++
 2 files changed, 37 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index df3e376e439c..83fa87e88bce 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1467,6 +1467,28 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing);
 
+int v4l2_subdev_set_routing_with_fmt(struct v4l2_subdev *sd,
+				     struct v4l2_subdev_state *state,
+				     struct v4l2_subdev_krouting *routing,
+				     const struct v4l2_mbus_framefmt *fmt)
+{
+	struct v4l2_subdev_stream_configs *stream_configs;
+	unsigned int i;
+	int ret;
+
+	ret = v4l2_subdev_set_routing(sd, state, routing);
+	if (ret)
+		return ret;
+
+	stream_configs = &state->stream_configs;
+
+	for (i = 0; i < stream_configs->num_configs; ++i)
+		stream_configs->configs[i].fmt = *fmt;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing_with_fmt);
+
 struct v4l2_mbus_framefmt *
 v4l2_state_get_stream_format(struct v4l2_subdev_state *state, unsigned int pad,
 			     u32 stream)
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index c3e859077557..3f72c500ff6e 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1485,6 +1485,21 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
 			    struct v4l2_subdev_state *state,
 			    struct v4l2_subdev_krouting *routing);
 
+/**
+ * v4l2_subdev_set_routing_with_fmt() - Set given routing and format to subdev state
+ * @sd: The subdevice
+ * @state: The subdevice state
+ * @routing: Routing that will be copied to subdev state
+ * @fmt: Format used to initialize all the streams
+ *
+ * This is the same as v4l2_subdev_set_routing, but additionally initializes
+ * all the streams using the given format.
+ */
+int v4l2_subdev_set_routing_with_fmt(struct v4l2_subdev *sd,
+				     struct v4l2_subdev_state *state,
+				     struct v4l2_subdev_krouting *routing,
+				     const struct v4l2_mbus_framefmt *fmt);
+
 /**
  * v4l2_state_get_stream_format() - Get pointer to a stream format
  * @state: subdevice state
-- 
2.25.1

[PATCH v8 36/36] media: subdev: add v4l2_routing_simple_verify() helper

[Tomi Valkeinen]

Add a helper for verifying routing for the common case of
non-overlapping 1-to-1 streams.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 24 ++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 14 ++++++++++++++
 2 files changed, 38 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 83fa87e88bce..e3a7793157db 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1572,3 +1572,27 @@ int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
 	return 0;
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_get_fmt);
+
+int v4l2_routing_simple_verify(const struct v4l2_subdev_krouting *routing)
+{
+	unsigned int i, j;
+
+	for (i = 0; i < routing->num_routes; ++i) {
+		const struct v4l2_subdev_route *route = &routing->routes[i];
+
+		for (j = i + 1; j < routing->num_routes; ++j) {
+			const struct v4l2_subdev_route *r = &routing->routes[j];
+
+			if (route->sink_pad == r->sink_pad &&
+			    route->sink_stream == r->sink_stream)
+				return -EINVAL;
+
+			if (route->source_pad == r->source_pad &&
+			    route->source_stream == r->source_stream)
+				return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_routing_simple_verify);
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 3f72c500ff6e..476d797ecc68 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1562,4 +1562,18 @@ v4l2_state_get_opposite_stream_format(struct v4l2_subdev_state *state, u32 pad,
 int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
 			struct v4l2_subdev_format *format);
 
+/**
+ * v4l2_routing_simple_verify() - Verify that all streams are non-overlapping
+ * 				  1-to-1 streams
+ * @routing: routing to verify
+ *
+ * This verifies that the given routing contains only non-overlapping 1-to-1
+ * streams. In other words, no two streams have the same source or sink
+ * stream ID on a single pad. This is the most common case of routing
+ * supported by devices.
+ *
+ * Returns 0 on success, error value otherwise.
+ */
+int v4l2_routing_simple_verify(const struct v4l2_subdev_krouting *routing);
+
 #endif
-- 
2.25.1

Re: [PATCH v8 02/36] media: subdev: add active state to struct v4l2_subdev

[Jacopo Mondi]

Hi Tomi,

On Mon, Aug 30, 2021 at 02:00:42PM +0300, Tomi Valkeinen wrote:
> Add a new 'state' field to struct v4l2_subdev to which we can store the
> active state of a subdev. This will place the subdev configuration into
> a known place, allowing us to use the state directly from the v4l2
> framework, thus simplifying the drivers.
>
> We also add v4l2_subdev_alloc_state() and v4l2_subdev_free_state(),
> which need to be used by the drivers that support subdev state in struct
> v4l2_subdev.
>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 21 ++++++++++++++++
>  include/media/v4l2-subdev.h           | 36 +++++++++++++++++++++++++++
>  2 files changed, 57 insertions(+)
>
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 26a34a8e3d37..e1a794f69815 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -943,3 +943,24 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
>  	v4l2_subdev_notify(sd, V4L2_DEVICE_NOTIFY_EVENT, (void *)ev);
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
> +
> +int v4l2_subdev_alloc_state(struct v4l2_subdev *sd)
> +{
> +	struct v4l2_subdev_state *state;
> +
> +	state = v4l2_alloc_subdev_state(sd);
> +	if (IS_ERR(state))
> +		return PTR_ERR(state);
> +
> +	sd->state = state;
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_alloc_state);
> +
> +void v4l2_subdev_free_state(struct v4l2_subdev *sd)
> +{
> +	v4l2_free_subdev_state(sd->state);
> +	sd->state = NULL;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_free_state);
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 8701d2e7d893..ecaf040ead57 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -898,6 +898,8 @@ struct v4l2_subdev_platform_data {
>   * @subdev_notifier: A sub-device notifier implicitly registered for the sub-
>   *		     device using v4l2_async_register_subdev_sensor().
>   * @pdata: common part of subdevice platform data
> + * @state: active state for the subdev (NULL for subdevs tracking the state
> + * 	   internally)
>   *
>   * Each instance of a subdev driver should create this struct, either
>   * stand-alone or embedded in a larger struct.
> @@ -929,6 +931,7 @@ struct v4l2_subdev {
>  	struct v4l2_async_notifier *notifier;
>  	struct v4l2_async_notifier *subdev_notifier;
>  	struct v4l2_subdev_platform_data *pdata;
> +	struct v4l2_subdev_state *state;

Is there anything preventing state from being a struct member and only
allocate the required number of v4l2_subdev_pad_config entries ?

>  };
>
>
> @@ -1217,4 +1220,37 @@ extern const struct v4l2_subdev_ops v4l2_subdev_call_wrappers;
>  void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
>  			      const struct v4l2_event *ev);
>
> +/**
> + * v4l2_subdev_alloc_state() - Allocate active subdev state for subdevice
> + * @sd: The subdev for which the state is allocated
> + *
> + * This will allocate a subdev state and store it to
> + * &struct v4l2_subdev->state.
> + *
> + * Must call v4l2_subdev_free_state() when the state is no longer needed.
> + */
> +int v4l2_subdev_alloc_state(struct v4l2_subdev *sd);
> +
> +/**
> + * v4l2_subdev_free_state() - Free the active subdev state for subdevice
> + * @sd: The subdevice
> + *
> + * This will free the subdev's state and set
> + * &struct v4l2_subdev->state to NULL.
> + */
> +void v4l2_subdev_free_state(struct v4l2_subdev *sd);
> +
> +/**
> + * v4l2_subdev_get_active_state() - Return the active subdev state for subdevice
> + * @sd: The subdevice
> + *
> + * Return the active state for the subdevice, or NULL if the subdev does not
> + * support active state.
> + */
> +static inline struct v4l2_subdev_state *
> +v4l2_subdev_get_active_state(struct v4l2_subdev *sd)
> +{
> +	return sd->state;
> +}

It would also make safer to access sd->state, as if a driver doesn't
allocate a state but calls this function it would get back a NULL
pointer.

Also, the name 'active' suggests there will be a non-active state ?

Thanks
   j

> +
>  #endif
> --
> 2.25.1
>

Re: [PATCH v8 03/36] media: subdev: add 'which' to subdev state

[Jacopo Mondi]

Hi Tomi,

On Mon, Aug 30, 2021 at 02:00:43PM +0300, Tomi Valkeinen wrote:
> The subdev state is passed to functions in the media drivers, and
> usually either V4L2_SUBDEV_FORMAT_ACTIVE or V4L2_SUBDEV_FORMAT_TRY is
> also given to the function in one way or another.
>
> One op where this is not the case is v4l2_subdev_pad_ops.init_cfg. One
> could argue that the initialization of the state should be the same for
> both ACTIVE and TRY cases, but unfortunately that is not the case:
>
> - Some drivers do also other things than just touch the state when
> dealing with ACTIVE, e.g. if there is extra state outside the standard
> subdev state.
> - Some drivers might need to create, say, struct v4l2_subdev_format
> which has 'which' field, and that needs to be filled with either ACTIVE
> or TRY.
>
> Currently init_cfg is only called for TRY case from the v4l2 framework,
> passing the TRY state. Some drivers call their own init_cfg, passing
> NULL as the state, which is used to indicate ACTIVE case.
>
> In the future we want to pass subdev's active state from the v4l2
> framework side, so we need a solution to this.
>
> We could change the init_cfg() to include the TRY/ACTIVE value, which
> would require changing more or less all the drivers. Instead, I have
> added 'which' field to the subdev state itself, filled at state
> allocation time, which only requires changes to the drivers that
> allocate a state themselves.
>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/platform/rcar-vin/rcar-v4l2.c |  2 +-
>  drivers/media/platform/vsp1/vsp1_entity.c   |  2 +-
>  drivers/media/v4l2-core/v4l2-subdev.c       | 10 +++++++---
>  drivers/staging/media/tegra-video/vi.c      |  2 +-
>  include/media/v4l2-subdev.h                 |  7 ++++++-
>  5 files changed, 16 insertions(+), 7 deletions(-)
>
> diff --git a/drivers/media/platform/rcar-vin/rcar-v4l2.c b/drivers/media/platform/rcar-vin/rcar-v4l2.c
> index 5f4fa8c48f68..1de30d5b437f 100644
> --- a/drivers/media/platform/rcar-vin/rcar-v4l2.c
> +++ b/drivers/media/platform/rcar-vin/rcar-v4l2.c
> @@ -252,7 +252,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
>  	u32 width, height;
>  	int ret;
>
> -	sd_state = v4l2_alloc_subdev_state(sd);
> +	sd_state = v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_ACTIVE);

Shouldn't the 'which' parameters be used to decide if either ACTIVE or
TRY have to be used ? this function is also used to set TRY formats,
in example...

Oh, maybe I got how it works, the state's which is not
relevant but the v4l2_subdev_format's which is, as it will be used in
the next patch to decide if the subdev's state of the file-handle's
state should be passed to the ioctl.

>  	if (IS_ERR(sd_state))
>  		return PTR_ERR(sd_state);
>
> diff --git a/drivers/media/platform/vsp1/vsp1_entity.c b/drivers/media/platform/vsp1/vsp1_entity.c
> index e40bca254b8b..63ea5e472c33 100644
> --- a/drivers/media/platform/vsp1/vsp1_entity.c
> +++ b/drivers/media/platform/vsp1/vsp1_entity.c
> @@ -675,7 +675,7 @@ int vsp1_entity_init(struct vsp1_device *vsp1, struct vsp1_entity *entity,
>  	 * Allocate the pad configuration to store formats and selection
>  	 * rectangles.
>  	 */
> -	entity->config = v4l2_alloc_subdev_state(&entity->subdev);
> +	entity->config = v4l2_alloc_subdev_state(&entity->subdev, V4L2_SUBDEV_FORMAT_ACTIVE);
>  	if (IS_ERR(entity->config)) {
>  		media_entity_cleanup(&entity->subdev.entity);
>  		return PTR_ERR(entity->config);
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index e1a794f69815..04ad319fb150 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -28,7 +28,7 @@ static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
>  {
>  	struct v4l2_subdev_state *state;
>
> -	state = v4l2_alloc_subdev_state(sd);
> +	state = v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_TRY);

At the same time I'm not sure I get the purpose of this. Don't
init_cfg() callback implementations deal with try formats themeselves
? I mean, it's not a fixed rule, they can as well initialize their
default 'active' formats, but what matters is that they initialize
their per-fh try states ?

Shouldn't init_cfg receive the fh's state so that it can initialize
it, and just in case they need to, access their subdev's state and
initialize them ? I'm missing what the purpose of the flag is tbh.

Thanks
   j


>  	if (IS_ERR(state))
>  		return PTR_ERR(state);
>
> @@ -870,7 +870,9 @@ int v4l2_subdev_link_validate(struct media_link *link)
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
>
> -struct v4l2_subdev_state *v4l2_alloc_subdev_state(struct v4l2_subdev *sd)
> +struct v4l2_subdev_state *
> +v4l2_alloc_subdev_state(struct v4l2_subdev *sd,
> +			enum v4l2_subdev_format_whence which)
>  {
>  	struct v4l2_subdev_state *state;
>  	int ret;
> @@ -879,6 +881,8 @@ struct v4l2_subdev_state *v4l2_alloc_subdev_state(struct v4l2_subdev *sd)
>  	if (!state)
>  		return ERR_PTR(-ENOMEM);
>
> +	state->which = which;
> +
>  	if (sd->entity.num_pads) {
>  		state->pads = kvmalloc_array(sd->entity.num_pads,
>  					     sizeof(*state->pads),
> @@ -948,7 +952,7 @@ int v4l2_subdev_alloc_state(struct v4l2_subdev *sd)
>  {
>  	struct v4l2_subdev_state *state;
>
> -	state = v4l2_alloc_subdev_state(sd);
> +	state = v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_ACTIVE);
>  	if (IS_ERR(state))
>  		return PTR_ERR(state);
>
> diff --git a/drivers/staging/media/tegra-video/vi.c b/drivers/staging/media/tegra-video/vi.c
> index a94d19e2a67c..691f5e04b0a1 100644
> --- a/drivers/staging/media/tegra-video/vi.c
> +++ b/drivers/staging/media/tegra-video/vi.c
> @@ -507,7 +507,7 @@ static int __tegra_channel_try_format(struct tegra_vi_channel *chan,
>  	if (!subdev)
>  		return -ENODEV;
>
> -	sd_state = v4l2_alloc_subdev_state(subdev);
> +	sd_state = v4l2_alloc_subdev_state(subdev, V4L2_SUBDEV_FORMAT_ACTIVE);
>  	if (IS_ERR(sd_state))
>  		return PTR_ERR(sd_state);
>  	/*
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index ecaf040ead57..5ec78ffda4f5 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -655,6 +655,7 @@ struct v4l2_subdev_pad_config {
>  /**
>   * struct v4l2_subdev_state - Used for storing subdev state information.
>   *
> + * @which: state type (from enum v4l2_subdev_format_whence)
>   * @pads: &struct v4l2_subdev_pad_config array
>   *
>   * This structure only needs to be passed to the pad op if the 'which' field
> @@ -662,6 +663,7 @@ struct v4l2_subdev_pad_config {
>   * %V4L2_SUBDEV_FORMAT_ACTIVE it is safe to pass %NULL.
>   */
>  struct v4l2_subdev_state {
> +	u32 which;
>  	struct v4l2_subdev_pad_config *pads;
>  };
>
> @@ -1141,10 +1143,13 @@ int v4l2_subdev_link_validate(struct media_link *link);
>   * v4l2_alloc_subdev_state - allocate v4l2_subdev_state
>   *
>   * @sd: pointer to &struct v4l2_subdev for which the state is being allocated.
> + * @which: configuration type for the state (from enum v4l2_subdev_format_whence)
>   *
>   * Must call v4l2_free_subdev_state() when state is no longer needed.
>   */
> -struct v4l2_subdev_state *v4l2_alloc_subdev_state(struct v4l2_subdev *sd);
> +struct v4l2_subdev_state *
> +v4l2_alloc_subdev_state(struct v4l2_subdev *sd,
> +			enum v4l2_subdev_format_whence which);
>
>  /**
>   * v4l2_free_subdev_state - free a v4l2_subdev_state
> --
> 2.25.1
>

Re: [PATCH v8 02/36] media: subdev: add active state to struct v4l2_subdev

[Tomi Valkeinen]

Hi,

On 13/09/2021 13:57, Jacopo Mondi wrote:
> Hi Tomi,
> 
> On Mon, Aug 30, 2021 at 02:00:42PM +0300, Tomi Valkeinen wrote:
>> Add a new 'state' field to struct v4l2_subdev to which we can store the
>> active state of a subdev. This will place the subdev configuration into
>> a known place, allowing us to use the state directly from the v4l2
>> framework, thus simplifying the drivers.
>>
>> We also add v4l2_subdev_alloc_state() and v4l2_subdev_free_state(),
>> which need to be used by the drivers that support subdev state in struct
>> v4l2_subdev.
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>>   drivers/media/v4l2-core/v4l2-subdev.c | 21 ++++++++++++++++
>>   include/media/v4l2-subdev.h           | 36 +++++++++++++++++++++++++++
>>   2 files changed, 57 insertions(+)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>> index 26a34a8e3d37..e1a794f69815 100644
>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>> @@ -943,3 +943,24 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
>>   	v4l2_subdev_notify(sd, V4L2_DEVICE_NOTIFY_EVENT, (void *)ev);
>>   }
>>   EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
>> +
>> +int v4l2_subdev_alloc_state(struct v4l2_subdev *sd)
>> +{
>> +	struct v4l2_subdev_state *state;
>> +
>> +	state = v4l2_alloc_subdev_state(sd);
>> +	if (IS_ERR(state))
>> +		return PTR_ERR(state);
>> +
>> +	sd->state = state;
>> +
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_subdev_alloc_state);
>> +
>> +void v4l2_subdev_free_state(struct v4l2_subdev *sd)
>> +{
>> +	v4l2_free_subdev_state(sd->state);
>> +	sd->state = NULL;
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_subdev_free_state);
>> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
>> index 8701d2e7d893..ecaf040ead57 100644
>> --- a/include/media/v4l2-subdev.h
>> +++ b/include/media/v4l2-subdev.h
>> @@ -898,6 +898,8 @@ struct v4l2_subdev_platform_data {
>>    * @subdev_notifier: A sub-device notifier implicitly registered for the sub-
>>    *		     device using v4l2_async_register_subdev_sensor().
>>    * @pdata: common part of subdevice platform data
>> + * @state: active state for the subdev (NULL for subdevs tracking the state
>> + * 	   internally)
>>    *
>>    * Each instance of a subdev driver should create this struct, either
>>    * stand-alone or embedded in a larger struct.
>> @@ -929,6 +931,7 @@ struct v4l2_subdev {
>>   	struct v4l2_async_notifier *notifier;
>>   	struct v4l2_async_notifier *subdev_notifier;
>>   	struct v4l2_subdev_platform_data *pdata;
>> +	struct v4l2_subdev_state *state;
> 
> Is there anything preventing state from being a struct member and only
> allocate the required number of v4l2_subdev_pad_config entries ?

Perhaps nothing strictly prevents it, but having the state pointer != 
NULL tells us that the driver has created the state. One annoyance was 
that I didn't find a good place to implicitly create the state based on 
a subdev flag. Instead the subdev driver needs to call 
v4l2_subdev_alloc_state() explicitly. So having the state as a pointer 
makes it clear that the state has been initialized.

>>   };
>>
>>
>> @@ -1217,4 +1220,37 @@ extern const struct v4l2_subdev_ops v4l2_subdev_call_wrappers;
>>   void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
>>   			      const struct v4l2_event *ev);
>>
>> +/**
>> + * v4l2_subdev_alloc_state() - Allocate active subdev state for subdevice
>> + * @sd: The subdev for which the state is allocated
>> + *
>> + * This will allocate a subdev state and store it to
>> + * &struct v4l2_subdev->state.
>> + *
>> + * Must call v4l2_subdev_free_state() when the state is no longer needed.
>> + */
>> +int v4l2_subdev_alloc_state(struct v4l2_subdev *sd);
>> +
>> +/**
>> + * v4l2_subdev_free_state() - Free the active subdev state for subdevice
>> + * @sd: The subdevice
>> + *
>> + * This will free the subdev's state and set
>> + * &struct v4l2_subdev->state to NULL.
>> + */
>> +void v4l2_subdev_free_state(struct v4l2_subdev *sd);
>> +
>> +/**
>> + * v4l2_subdev_get_active_state() - Return the active subdev state for subdevice
>> + * @sd: The subdevice
>> + *
>> + * Return the active state for the subdevice, or NULL if the subdev does not
>> + * support active state.
>> + */
>> +static inline struct v4l2_subdev_state *
>> +v4l2_subdev_get_active_state(struct v4l2_subdev *sd)
>> +{
>> +	return sd->state;
>> +}
> 
> It would also make safer to access sd->state, as if a driver doesn't
> allocate a state but calls this function it would get back a NULL
> pointer.

Would WARN_ON(!sd->state) be ok?

> Also, the name 'active' suggests there will be a non-active state ?

It's the V4L2_SUBDEV_FORMAT_ACTIVE vs V4L2_SUBDEV_FORMAT_TRY. For TRY we 
don't need getters, as the TRY state is passed directly to the relevant 
ops. But the drivers need to get the ACTIVE state, e.g. when starting 
the streaming.

  Tomi

Re: [PATCH v8 03/36] media: subdev: add 'which' to subdev state

[Tomi Valkeinen]

On 13/09/2021 14:41, Jacopo Mondi wrote:
> Hi Tomi,
> 
> On Mon, Aug 30, 2021 at 02:00:43PM +0300, Tomi Valkeinen wrote:
>> The subdev state is passed to functions in the media drivers, and
>> usually either V4L2_SUBDEV_FORMAT_ACTIVE or V4L2_SUBDEV_FORMAT_TRY is
>> also given to the function in one way or another.
>>
>> One op where this is not the case is v4l2_subdev_pad_ops.init_cfg. One
>> could argue that the initialization of the state should be the same for
>> both ACTIVE and TRY cases, but unfortunately that is not the case:
>>
>> - Some drivers do also other things than just touch the state when
>> dealing with ACTIVE, e.g. if there is extra state outside the standard
>> subdev state.
>> - Some drivers might need to create, say, struct v4l2_subdev_format
>> which has 'which' field, and that needs to be filled with either ACTIVE
>> or TRY.
>>
>> Currently init_cfg is only called for TRY case from the v4l2 framework,
>> passing the TRY state. Some drivers call their own init_cfg, passing
>> NULL as the state, which is used to indicate ACTIVE case.
>>
>> In the future we want to pass subdev's active state from the v4l2
>> framework side, so we need a solution to this.
>>
>> We could change the init_cfg() to include the TRY/ACTIVE value, which
>> would require changing more or less all the drivers. Instead, I have
>> added 'which' field to the subdev state itself, filled at state
>> allocation time, which only requires changes to the drivers that
>> allocate a state themselves.
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>>   drivers/media/platform/rcar-vin/rcar-v4l2.c |  2 +-
>>   drivers/media/platform/vsp1/vsp1_entity.c   |  2 +-
>>   drivers/media/v4l2-core/v4l2-subdev.c       | 10 +++++++---
>>   drivers/staging/media/tegra-video/vi.c      |  2 +-
>>   include/media/v4l2-subdev.h                 |  7 ++++++-
>>   5 files changed, 16 insertions(+), 7 deletions(-)
>>
>> diff --git a/drivers/media/platform/rcar-vin/rcar-v4l2.c b/drivers/media/platform/rcar-vin/rcar-v4l2.c
>> index 5f4fa8c48f68..1de30d5b437f 100644
>> --- a/drivers/media/platform/rcar-vin/rcar-v4l2.c
>> +++ b/drivers/media/platform/rcar-vin/rcar-v4l2.c
>> @@ -252,7 +252,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
>>   	u32 width, height;
>>   	int ret;
>>
>> -	sd_state = v4l2_alloc_subdev_state(sd);
>> +	sd_state = v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_ACTIVE);
> 
> Shouldn't the 'which' parameters be used to decide if either ACTIVE or
> TRY have to be used ? this function is also used to set TRY formats,
> in example...
> 
> Oh, maybe I got how it works, the state's which is not
> relevant but the v4l2_subdev_format's which is, as it will be used in
> the next patch to decide if the subdev's state of the file-handle's
> state should be passed to the ioctl.

Yes. It's messy, but it's how it worked before also.

The drivers can't really allocate TRY state as it must come from the 
core, based on the filehandle. Now as I say that, makes me wonder why 
even expose the option to drivers. Maybe v4l2_alloc_subdev_state() 
should take just the sd parameter, and always allocate ACTIVE state, and 
the v4l2 core can use another way to create the TRY state.

>>   	if (IS_ERR(sd_state))
>>   		return PTR_ERR(sd_state);
>>
>> diff --git a/drivers/media/platform/vsp1/vsp1_entity.c b/drivers/media/platform/vsp1/vsp1_entity.c
>> index e40bca254b8b..63ea5e472c33 100644
>> --- a/drivers/media/platform/vsp1/vsp1_entity.c
>> +++ b/drivers/media/platform/vsp1/vsp1_entity.c
>> @@ -675,7 +675,7 @@ int vsp1_entity_init(struct vsp1_device *vsp1, struct vsp1_entity *entity,
>>   	 * Allocate the pad configuration to store formats and selection
>>   	 * rectangles.
>>   	 */
>> -	entity->config = v4l2_alloc_subdev_state(&entity->subdev);
>> +	entity->config = v4l2_alloc_subdev_state(&entity->subdev, V4L2_SUBDEV_FORMAT_ACTIVE);
>>   	if (IS_ERR(entity->config)) {
>>   		media_entity_cleanup(&entity->subdev.entity);
>>   		return PTR_ERR(entity->config);
>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>> index e1a794f69815..04ad319fb150 100644
>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>> @@ -28,7 +28,7 @@ static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
>>   {
>>   	struct v4l2_subdev_state *state;
>>
>> -	state = v4l2_alloc_subdev_state(sd);
>> +	state = v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_TRY);
> 
> At the same time I'm not sure I get the purpose of this. Don't
> init_cfg() callback implementations deal with try formats themeselves
> ? I mean, it's not a fixed rule, they can as well initialize their
> default 'active' formats, but what matters is that they initialize
> their per-fh try states ?

That is what they do currently. init_cfg() only deals with TRY state, as 
that's the only "state" (i.e. pad_config) there used to be from v4l2 
core's perspective.

> Shouldn't init_cfg receive the fh's state so that it can initialize
> it, and just in case they need to, access their subdev's state and
> initialize them ? I'm missing what the purpose of the flag is tbh.

Now we have (a possibility to have) state for both TRY and ACTIVE on the 
v4l2 core side. The active state has to be initialized also, and a 
logical way to do that is to use the init_cfg().

So now, for drivers that support the new active state, init_cfg() can 
get either TRY or ACTIVE state. And if you want to call, say, the 
driver's set_routing() to setup the routing in the state, you have to 
set the 'which' in the routing struct to a value. So somehow init_cfg 
needs to know if it's initializing an ACTIVE or TRY state.

  Tomi

Re: [PATCH v8 03/36] media: subdev: add 'which' to subdev state

[Jacopo Mondi]

Hi Tomi,

On Mon, Sep 13, 2021 at 03:17:01PM +0300, Tomi Valkeinen wrote:
> On 13/09/2021 14:41, Jacopo Mondi wrote:
> > Hi Tomi,
> >
> > On Mon, Aug 30, 2021 at 02:00:43PM +0300, Tomi Valkeinen wrote:
> > > The subdev state is passed to functions in the media drivers, and
> > > usually either V4L2_SUBDEV_FORMAT_ACTIVE or V4L2_SUBDEV_FORMAT_TRY is
> > > also given to the function in one way or another.
> > >
> > > One op where this is not the case is v4l2_subdev_pad_ops.init_cfg. One
> > > could argue that the initialization of the state should be the same for
> > > both ACTIVE and TRY cases, but unfortunately that is not the case:
> > >
> > > - Some drivers do also other things than just touch the state when
> > > dealing with ACTIVE, e.g. if there is extra state outside the standard
> > > subdev state.
> > > - Some drivers might need to create, say, struct v4l2_subdev_format
> > > which has 'which' field, and that needs to be filled with either ACTIVE
> > > or TRY.
> > >
> > > Currently init_cfg is only called for TRY case from the v4l2 framework,
> > > passing the TRY state. Some drivers call their own init_cfg, passing
> > > NULL as the state, which is used to indicate ACTIVE case.
> > >
> > > In the future we want to pass subdev's active state from the v4l2
> > > framework side, so we need a solution to this.
> > >
> > > We could change the init_cfg() to include the TRY/ACTIVE value, which
> > > would require changing more or less all the drivers. Instead, I have
> > > added 'which' field to the subdev state itself, filled at state
> > > allocation time, which only requires changes to the drivers that
> > > allocate a state themselves.
> > >
> > > Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > > ---
> > >   drivers/media/platform/rcar-vin/rcar-v4l2.c |  2 +-
> > >   drivers/media/platform/vsp1/vsp1_entity.c   |  2 +-
> > >   drivers/media/v4l2-core/v4l2-subdev.c       | 10 +++++++---
> > >   drivers/staging/media/tegra-video/vi.c      |  2 +-
> > >   include/media/v4l2-subdev.h                 |  7 ++++++-
> > >   5 files changed, 16 insertions(+), 7 deletions(-)
> > >
> > > diff --git a/drivers/media/platform/rcar-vin/rcar-v4l2.c b/drivers/media/platform/rcar-vin/rcar-v4l2.c
> > > index 5f4fa8c48f68..1de30d5b437f 100644
> > > --- a/drivers/media/platform/rcar-vin/rcar-v4l2.c
> > > +++ b/drivers/media/platform/rcar-vin/rcar-v4l2.c
> > > @@ -252,7 +252,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
> > >   	u32 width, height;
> > >   	int ret;
> > >
> > > -	sd_state = v4l2_alloc_subdev_state(sd);
> > > +	sd_state = v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_ACTIVE);
> >
> > Shouldn't the 'which' parameters be used to decide if either ACTIVE or
> > TRY have to be used ? this function is also used to set TRY formats,
> > in example...
> >
> > Oh, maybe I got how it works, the state's which is not
> > relevant but the v4l2_subdev_format's which is, as it will be used in
> > the next patch to decide if the subdev's state of the file-handle's
> > state should be passed to the ioctl.
>
> Yes. It's messy, but it's how it worked before also.
>
> The drivers can't really allocate TRY state as it must come from the core,
> based on the filehandle. Now as I say that, makes me wonder why even expose
> the option to drivers. Maybe v4l2_alloc_subdev_state() should take just the
> sd parameter, and always allocate ACTIVE state, and the v4l2 core can use
> another way to create the TRY state.
>

init_cfg() as well as other operations used to received an
array of per fh's pad_configs, and the sd pointer itself. The fh's pad
configs were allocated by the core, as well as the core now allocates
the per-fh's state.

Before the introduction of 'state', if the 'which' flags was set to
TRY then information were stored/retrieved/initialized in the per-fh
pad_config, otherwise the active configuration (usually stored in the
driver main structure) was used.

So we had a clear separation of per-fh information and the active
state. The core didn't care afaict, and passed in both, then driver had
to deal with them doing the right thing by inspecting the 'which' flag.

The typical pattern was:

        static int subdev_ops(sd, pad_cfg, which)
        {
                if (which == TRY)
                        /* Operate on config stored in pad_cfg */

                else
                        /*
                         * Operate on config stored in subdev (and
                         * applied to HW)
                         */
        }

Or am I overlooking some cases or do you agree with my understanding
so far ?

Now we have a 'state' that holds the array of pad_configs and along
the continuation of the series will end up holding per-pad
configurations.

We now also have one 'state' per file-handle, and one
per-subdev. As I see this, it would be natual for drivers to receive
one state without knowing where it comes from. In the next patch you
instrument the core to do exactly this: inspect the which flag and
pass in the 'right' state. Ofc drivers need to have access to 'which'
to know if they have to apply settings to the HW or not.

Looking ahead in your series I see these structures:

        struct v4l2_subdev_pad_config {
                struct v4l2_mbus_framefmt try_fmt;
                struct v4l2_rect try_crop;
                struct v4l2_rect try_compose;
        };

        struct v4l2_subdev_stream_config {
                u32 pad;
                u32 stream;

                struct v4l2_mbus_framefmt fmt;
                struct v4l2_rect crop;
                struct v4l2_rect compose;
        };

        struct v4l2_subdev_stream_configs {
                u32 num_configs;
                struct v4l2_subdev_stream_config *configs;
        };

All of them part of state:

struct v4l2_subdev_state {
	struct mutex lock;
	u32 which;
	struct v4l2_subdev_pad_config *pads;
	struct v4l2_subdev_krouting routing;
	struct v4l2_subdev_stream_configs stream_configs;
};

So 'state' will hold 'TRY' information (only used for 'state'
instances allocated in the fh) and 'ACTIVE' ones (used for states
allocated in the sd).

Looking at 'v4l2_subdev_pad_config' and 'v4l2_subdev_stream_config' they
seem to describe more or less the same things: fmt, crop and compose
(per pad-stream in case of stream_config). I wonder if those shouldn't
be unified so that:

1) Drivers receive one state: the core passes in the 'correct' one
(per-fh or per-sd) as you do in next patch
2) The 'which' information is not stored in the state but it's only
'contextual' (as in a parameter to the subdev operation) so that
drivers inspect it to know if they have to apply settings to hw or not
3) v4l2_subdev_pad_config can be re-used and expanded, to maintain per-pad
configurations regardless if they're ACTIVE or TRY, as this only depends
on where the state is stored.

As I immagine it a subdev pad operation could look like:

        static int subdev_op(sd, pad, state, which, ...)
        {
                /* Doesn't matter if state is per-fh or the sd one. */
                state->pads[pad].fmt = ....;

                if (which == TRY)
                        return;

                /* Apply to the HW. */
        }

Does it make any sense to you ? I might have missed some reason why
this is not possible.

> > >   	if (IS_ERR(sd_state))
> > >   		return PTR_ERR(sd_state);
> > >
> > > diff --git a/drivers/media/platform/vsp1/vsp1_entity.c b/drivers/media/platform/vsp1/vsp1_entity.c
> > > index e40bca254b8b..63ea5e472c33 100644
> > > --- a/drivers/media/platform/vsp1/vsp1_entity.c
> > > +++ b/drivers/media/platform/vsp1/vsp1_entity.c
> > > @@ -675,7 +675,7 @@ int vsp1_entity_init(struct vsp1_device *vsp1, struct vsp1_entity *entity,
> > >   	 * Allocate the pad configuration to store formats and selection
> > >   	 * rectangles.
> > >   	 */
> > > -	entity->config = v4l2_alloc_subdev_state(&entity->subdev);
> > > +	entity->config = v4l2_alloc_subdev_state(&entity->subdev, V4L2_SUBDEV_FORMAT_ACTIVE);
> > >   	if (IS_ERR(entity->config)) {
> > >   		media_entity_cleanup(&entity->subdev.entity);
> > >   		return PTR_ERR(entity->config);
> > > diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> > > index e1a794f69815..04ad319fb150 100644
> > > --- a/drivers/media/v4l2-core/v4l2-subdev.c
> > > +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> > > @@ -28,7 +28,7 @@ static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
> > >   {
> > >   	struct v4l2_subdev_state *state;
> > >
> > > -	state = v4l2_alloc_subdev_state(sd);
> > > +	state = v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_TRY);
> >
> > At the same time I'm not sure I get the purpose of this. Don't
> > init_cfg() callback implementations deal with try formats themeselves
> > ? I mean, it's not a fixed rule, they can as well initialize their
> > default 'active' formats, but what matters is that they initialize
> > their per-fh try states ?
>
> That is what they do currently. init_cfg() only deals with TRY state, as
> that's the only "state" (i.e. pad_config) there used to be from v4l2 core's
> perspective.
>
> > Shouldn't init_cfg receive the fh's state so that it can initialize
> > it, and just in case they need to, access their subdev's state and
> > initialize them ? I'm missing what the purpose of the flag is tbh.
>
> Now we have (a possibility to have) state for both TRY and ACTIVE on the
> v4l2 core side. The active state has to be initialized also, and a logical
> way to do that is to use the init_cfg().

The 'ACTIVE' state is stored in the subdev, to which init_cfg() has
access, and it receives the 'TRY' state as a parameter.

It is possible to access both states and initialize them properly if
I'm not mistaken.

>
> So now, for drivers that support the new active state, init_cfg() can get
> either TRY or ACTIVE state. And if you want to call, say, the driver's
> set_routing() to setup the routing in the state, you have to set the 'which'
> in the routing struct to a value. So somehow init_cfg needs to know if it's
> initializing an ACTIVE or TRY state.

I'm not sure I got this part. set_routing() as other ops will receive
a state and 'which'. If my proposal above makes sensem where the state
comes from (aka it's TRY or ACTIVE) it's not relevant for the driver.

Ofc init_cfg() should have initialized both states opportunely, but
that's the case today as well.

Thanks
  j

>
>  Tomi

Re: [PATCH v8 03/36] media: subdev: add 'which' to subdev state

[Tomi Valkeinen]

On 13/09/2021 16:38, Jacopo Mondi wrote:
> Hi Tomi,
> 
> On Mon, Sep 13, 2021 at 03:17:01PM +0300, Tomi Valkeinen wrote:
>> On 13/09/2021 14:41, Jacopo Mondi wrote:
>>> Hi Tomi,
>>>
>>> On Mon, Aug 30, 2021 at 02:00:43PM +0300, Tomi Valkeinen wrote:
>>>> The subdev state is passed to functions in the media drivers, and
>>>> usually either V4L2_SUBDEV_FORMAT_ACTIVE or V4L2_SUBDEV_FORMAT_TRY is
>>>> also given to the function in one way or another.
>>>>
>>>> One op where this is not the case is v4l2_subdev_pad_ops.init_cfg. One
>>>> could argue that the initialization of the state should be the same for
>>>> both ACTIVE and TRY cases, but unfortunately that is not the case:
>>>>
>>>> - Some drivers do also other things than just touch the state when
>>>> dealing with ACTIVE, e.g. if there is extra state outside the standard
>>>> subdev state.
>>>> - Some drivers might need to create, say, struct v4l2_subdev_format
>>>> which has 'which' field, and that needs to be filled with either ACTIVE
>>>> or TRY.
>>>>
>>>> Currently init_cfg is only called for TRY case from the v4l2 framework,
>>>> passing the TRY state. Some drivers call their own init_cfg, passing
>>>> NULL as the state, which is used to indicate ACTIVE case.
>>>>
>>>> In the future we want to pass subdev's active state from the v4l2
>>>> framework side, so we need a solution to this.
>>>>
>>>> We could change the init_cfg() to include the TRY/ACTIVE value, which
>>>> would require changing more or less all the drivers. Instead, I have
>>>> added 'which' field to the subdev state itself, filled at state
>>>> allocation time, which only requires changes to the drivers that
>>>> allocate a state themselves.
>>>>
>>>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>>>> ---
>>>>    drivers/media/platform/rcar-vin/rcar-v4l2.c |  2 +-
>>>>    drivers/media/platform/vsp1/vsp1_entity.c   |  2 +-
>>>>    drivers/media/v4l2-core/v4l2-subdev.c       | 10 +++++++---
>>>>    drivers/staging/media/tegra-video/vi.c      |  2 +-
>>>>    include/media/v4l2-subdev.h                 |  7 ++++++-
>>>>    5 files changed, 16 insertions(+), 7 deletions(-)
>>>>
>>>> diff --git a/drivers/media/platform/rcar-vin/rcar-v4l2.c b/drivers/media/platform/rcar-vin/rcar-v4l2.c
>>>> index 5f4fa8c48f68..1de30d5b437f 100644
>>>> --- a/drivers/media/platform/rcar-vin/rcar-v4l2.c
>>>> +++ b/drivers/media/platform/rcar-vin/rcar-v4l2.c
>>>> @@ -252,7 +252,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
>>>>    	u32 width, height;
>>>>    	int ret;
>>>>
>>>> -	sd_state = v4l2_alloc_subdev_state(sd);
>>>> +	sd_state = v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_ACTIVE);
>>>
>>> Shouldn't the 'which' parameters be used to decide if either ACTIVE or
>>> TRY have to be used ? this function is also used to set TRY formats,
>>> in example...
>>>
>>> Oh, maybe I got how it works, the state's which is not
>>> relevant but the v4l2_subdev_format's which is, as it will be used in
>>> the next patch to decide if the subdev's state of the file-handle's
>>> state should be passed to the ioctl.
>>
>> Yes. It's messy, but it's how it worked before also.
>>
>> The drivers can't really allocate TRY state as it must come from the core,
>> based on the filehandle. Now as I say that, makes me wonder why even expose
>> the option to drivers. Maybe v4l2_alloc_subdev_state() should take just the
>> sd parameter, and always allocate ACTIVE state, and the v4l2 core can use
>> another way to create the TRY state.
>>
> 
> init_cfg() as well as other operations used to received an
> array of per fh's pad_configs, and the sd pointer itself. The fh's pad
> configs were allocated by the core, as well as the core now allocates
> the per-fh's state.
> 
> Before the introduction of 'state', if the 'which' flags was set to
> TRY then information were stored/retrieved/initialized in the per-fh
> pad_config, otherwise the active configuration (usually stored in the
> driver main structure) was used.
> 
> So we had a clear separation of per-fh information and the active
> state. The core didn't care afaict, and passed in both, then driver had
> to deal with them doing the right thing by inspecting the 'which' flag.
> 
> The typical pattern was:
> 
>          static int subdev_ops(sd, pad_cfg, which)
>          {
>                  if (which == TRY)
>                          /* Operate on config stored in pad_cfg */
> 
>                  else
>                          /*
>                           * Operate on config stored in subdev (and
>                           * applied to HW)
>                           */
>          }
> 
> Or am I overlooking some cases or do you agree with my understanding
> so far ?

More or less, yes. I think there are (used to be) three kinds of ops:

- Ops that get pad_cfg and 'which' in an op specific struct. E.g. 
set_fmt. The pad_cfg is TRY pad_config, even if 'which' == ACTIVE.

- Ops that don't get pad_cfg, like s_stream. 'which' is implicitly ACTIVE.

- init_cfg which gets pad_cfg, but no which (as 'which' is always 
implicitly TRY)

So pad_cfg was TRY state. Drivers could use pad_configs internally to 
track ACTIVE state, but the core had no knowledge about this.

> Now we have a 'state' that holds the array of pad_configs and along
> the continuation of the series will end up holding per-pad
> configurations.
> 
> We now also have one 'state' per file-handle, and one
> per-subdev. As I see this, it would be natual for drivers to receive
> one state without knowing where it comes from. In the next patch you

Note that only subdev's that explicitly support the new state code, and 
allocate the state, have the subdev active state. Which means only the 
drivers in my work branch.

The "old" drivers work like they used to: they get the state 
(essentially repackaged pad_cfg) for TRY cases, NULL otherwise.

And yes, it would be natural to just get a state, but the subdev drivers 
need to know if the context is TRY/ACTIVE. As you can see from the 
bullet list above, the driver knows this in all the other places except 
init_cfg.

> instrument the core to do exactly this: inspect the which flag and
> pass in the 'right' state. Ofc drivers need to have access to 'which'
> to know if they have to apply settings to the HW or not.
> 
> Looking ahead in your series I see these structures:
> 
>          struct v4l2_subdev_pad_config {
>                  struct v4l2_mbus_framefmt try_fmt;
>                  struct v4l2_rect try_crop;
>                  struct v4l2_rect try_compose;
>          };
> 
>          struct v4l2_subdev_stream_config {
>                  u32 pad;
>                  u32 stream;
> 
>                  struct v4l2_mbus_framefmt fmt;
>                  struct v4l2_rect crop;
>                  struct v4l2_rect compose;
>          };
> 
>          struct v4l2_subdev_stream_configs {
>                  u32 num_configs;
>                  struct v4l2_subdev_stream_config *configs;
>          };
> 
> All of them part of state:
> 
> struct v4l2_subdev_state {
> 	struct mutex lock;
> 	u32 which;
> 	struct v4l2_subdev_pad_config *pads;
> 	struct v4l2_subdev_krouting routing;
> 	struct v4l2_subdev_stream_configs stream_configs;
> };
> 
> So 'state' will hold 'TRY' information (only used for 'state'
> instances allocated in the fh) and 'ACTIVE' ones (used for states
> allocated in the sd).

Right.

> Looking at 'v4l2_subdev_pad_config' and 'v4l2_subdev_stream_config' they
> seem to describe more or less the same things: fmt, crop and compose
> (per pad-stream in case of stream_config). I wonder if those shouldn't
> be unified so that:
> 
> 1) Drivers receive one state: the core passes in the 'correct' one
> (per-fh or per-sd) as you do in next patch

Yes. But note that "old" drivers don't have active state.

> 2) The 'which' information is not stored in the state but it's only
> 'contextual' (as in a parameter to the subdev operation) so that
> drivers inspect it to know if they have to apply settings to hw or not

Yes, except we have init_cfg...

> 3) v4l2_subdev_pad_config can be re-used and expanded, to maintain per-pad
> configurations regardless if they're ACTIVE or TRY, as this only depends
> on where the state is stored.

pad_config is a static array of per-pad configs. stream_configs is a 
dynamic per-stream config. stream_configs is a super-set of pad-configs, 
so we could drop pad_configs, but it would require changing all the 
drivers in non-trivial ways.

v4l2_subdev_pad_config is not used or even allocated by the "new" 
drivers. And routing & stream_configs are not used by the "old" drivers.

> As I immagine it a subdev pad operation could look like:
> 
>          static int subdev_op(sd, pad, state, which, ...)
>          {
>                  /* Doesn't matter if state is per-fh or the sd one. */
>                  state->pads[pad].fmt = ....;
> 
>                  if (which == TRY)
>                          return;
> 
>                  /* Apply to the HW. */
>          }
> 
> Does it make any sense to you ? I might have missed some reason why
> this is not possible.

It makes sense, but there are the buts =). I've tried to explain these 
in the commit messages, but it's kind of confusing.

One but I haven't mentioned in the emails is that when subdev drivers 
call ops in other subdev drivers they pass NULL in the state. This is 
fine for the "old" drivers, as they expect a state only for TRY case. 
However, the "new" drivers unfortunately expect to get a state on both 
TRY and ACTIVE cases, and the only sensible way I figured out to handle 
this was the v4l2_subdev_validate_state() function (patch 6).

So, all this could be much neater, but would require modifying all 
subdev drivers in non-trivial ways. I think this is something that can 
be done slowly in the future.

>>>>    	if (IS_ERR(sd_state))
>>>>    		return PTR_ERR(sd_state);
>>>>
>>>> diff --git a/drivers/media/platform/vsp1/vsp1_entity.c b/drivers/media/platform/vsp1/vsp1_entity.c
>>>> index e40bca254b8b..63ea5e472c33 100644
>>>> --- a/drivers/media/platform/vsp1/vsp1_entity.c
>>>> +++ b/drivers/media/platform/vsp1/vsp1_entity.c
>>>> @@ -675,7 +675,7 @@ int vsp1_entity_init(struct vsp1_device *vsp1, struct vsp1_entity *entity,
>>>>    	 * Allocate the pad configuration to store formats and selection
>>>>    	 * rectangles.
>>>>    	 */
>>>> -	entity->config = v4l2_alloc_subdev_state(&entity->subdev);
>>>> +	entity->config = v4l2_alloc_subdev_state(&entity->subdev, V4L2_SUBDEV_FORMAT_ACTIVE);
>>>>    	if (IS_ERR(entity->config)) {
>>>>    		media_entity_cleanup(&entity->subdev.entity);
>>>>    		return PTR_ERR(entity->config);
>>>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>>>> index e1a794f69815..04ad319fb150 100644
>>>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>>>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>>>> @@ -28,7 +28,7 @@ static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
>>>>    {
>>>>    	struct v4l2_subdev_state *state;
>>>>
>>>> -	state = v4l2_alloc_subdev_state(sd);
>>>> +	state = v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_TRY);
>>>
>>> At the same time I'm not sure I get the purpose of this. Don't
>>> init_cfg() callback implementations deal with try formats themeselves
>>> ? I mean, it's not a fixed rule, they can as well initialize their
>>> default 'active' formats, but what matters is that they initialize
>>> their per-fh try states ?
>>
>> That is what they do currently. init_cfg() only deals with TRY state, as
>> that's the only "state" (i.e. pad_config) there used to be from v4l2 core's
>> perspective.
>>
>>> Shouldn't init_cfg receive the fh's state so that it can initialize
>>> it, and just in case they need to, access their subdev's state and
>>> initialize them ? I'm missing what the purpose of the flag is tbh.
>>
>> Now we have (a possibility to have) state for both TRY and ACTIVE on the
>> v4l2 core side. The active state has to be initialized also, and a logical
>> way to do that is to use the init_cfg().
> 
> The 'ACTIVE' state is stored in the subdev, to which init_cfg() has
> access, and it receives the 'TRY' state as a parameter.

No, init_cfg gets either ACTIVE or TRY state, whichever is being 
allocated. For "old" drivers, ACTIVE state is never allocated so they 
don't get init_cfg calls for ACTIVE at all.

Aaand while writing that, I realized that some drivers manually do 
allocate ACTIVE state temporarily, which would cause init_cfg with 
ACTIVE state to be called. I wonder if they explode... Need to check.

> It is possible to access both states and initialize them properly if
> I'm not mistaken.
> 
>>
>> So now, for drivers that support the new active state, init_cfg() can get
>> either TRY or ACTIVE state. And if you want to call, say, the driver's
>> set_routing() to setup the routing in the state, you have to set the 'which'
>> in the routing struct to a value. So somehow init_cfg needs to know if it's
>> initializing an ACTIVE or TRY state.
> 
> I'm not sure I got this part. set_routing() as other ops will receive
> a state and 'which'. If my proposal above makes sensem where the state

Yes, but if it's init_cfg calling set_routing, init_cfg has to figure 
out the 'which' from somewhere.

E.g. init_cfg from ub913 driver:

static int ub913_init_cfg(struct v4l2_subdev *sd,
			  struct v4l2_subdev_state *state)
{
	u32 which = state->which;

	struct v4l2_subdev_route routes[] = {
		{
			.sink_pad = 0,
			.sink_stream = 0,
			.source_pad = 1,
			.source_stream = 0,
			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
		},
	};

	struct v4l2_subdev_krouting routing = {
		.which = which,
		.num_routes = ARRAY_SIZE(routes),
		.routes = routes,
	};

	return ub913_set_routing(sd, state, &routing);
}

It uses set_routing to setup a default routing (and set_routing in turn 
also initializes the formats), but set_routing needs 'which'.

  Tomi

Re: [PATCH v8 02/36] media: subdev: add active state to struct v4l2_subdev

[Jacopo Mondi]

Hi Tomi,

On Mon, Aug 30, 2021 at 02:00:42PM +0300, Tomi Valkeinen wrote:
> Add a new 'state' field to struct v4l2_subdev to which we can store the
> active state of a subdev. This will place the subdev configuration into
> a known place, allowing us to use the state directly from the v4l2
> framework, thus simplifying the drivers.
>
> We also add v4l2_subdev_alloc_state() and v4l2_subdev_free_state(),
> which need to be used by the drivers that support subdev state in struct
> v4l2_subdev.
>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 21 ++++++++++++++++
>  include/media/v4l2-subdev.h           | 36 +++++++++++++++++++++++++++
>  2 files changed, 57 insertions(+)
>
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 26a34a8e3d37..e1a794f69815 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -943,3 +943,24 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
>  	v4l2_subdev_notify(sd, V4L2_DEVICE_NOTIFY_EVENT, (void *)ev);
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
> +
> +int v4l2_subdev_alloc_state(struct v4l2_subdev *sd)
> +{
> +	struct v4l2_subdev_state *state;
> +
> +	state = v4l2_alloc_subdev_state(sd);

So, I think this is one source of confusion about init_cfg.

v4l2_alloc_subdev_state() calls init_cfg() and 'state-aware' driver
are now supposed to allocate their state by calling
v4l2_subdev_alloc_state(), in the same way as the core does for the
file-handle ones.

This will lead to init_cfg to be called for the 'active' (ie owned by
the subdev) state, and then you need to add context to the state (by
adding a 'which' field) to know what state you're dealing with.

According to the init_cfg() documentation

 * @init_cfg: initialize the pad config to default values

the op has to be called in order to initialize the per-file-handle
context, not the active one.

I would rather just embed 'struct v4l2_subdev_state' in 'struct
v4l2_subdev', have the core going through the
'v4l2_subdev_alloc_state()' to initialize the per-fh state, but have
drivers initialize their own state at probe time. If they need for
some reason to access their 'active' state at init_cfg() time, they
caan fish it from their subdev.

If I'm not mistaken this will remove the need to have a which filed in
the state, as I think the 'context' should be inferred from the
'which' argument embedded in the ops-specific structures, and not held
in the state itself.

Thanks
   j


> +	if (IS_ERR(state))
> +		return PTR_ERR(state);
> +
> +	sd->state = state;
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_alloc_state);
> +
> +void v4l2_subdev_free_state(struct v4l2_subdev *sd)
> +{
> +	v4l2_free_subdev_state(sd->state);
> +	sd->state = NULL;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_free_state);
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 8701d2e7d893..ecaf040ead57 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -898,6 +898,8 @@ struct v4l2_subdev_platform_data {
>   * @subdev_notifier: A sub-device notifier implicitly registered for the sub-
>   *		     device using v4l2_async_register_subdev_sensor().
>   * @pdata: common part of subdevice platform data
> + * @state: active state for the subdev (NULL for subdevs tracking the state
> + * 	   internally)
>   *
>   * Each instance of a subdev driver should create this struct, either
>   * stand-alone or embedded in a larger struct.
> @@ -929,6 +931,7 @@ struct v4l2_subdev {
>  	struct v4l2_async_notifier *notifier;
>  	struct v4l2_async_notifier *subdev_notifier;
>  	struct v4l2_subdev_platform_data *pdata;
> +	struct v4l2_subdev_state *state;
>  };
>
>
> @@ -1217,4 +1220,37 @@ extern const struct v4l2_subdev_ops v4l2_subdev_call_wrappers;
>  void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
>  			      const struct v4l2_event *ev);
>
> +/**
> + * v4l2_subdev_alloc_state() - Allocate active subdev state for subdevice
> + * @sd: The subdev for which the state is allocated
> + *
> + * This will allocate a subdev state and store it to
> + * &struct v4l2_subdev->state.
> + *
> + * Must call v4l2_subdev_free_state() when the state is no longer needed.
> + */
> +int v4l2_subdev_alloc_state(struct v4l2_subdev *sd);
> +
> +/**
> + * v4l2_subdev_free_state() - Free the active subdev state for subdevice
> + * @sd: The subdevice
> + *
> + * This will free the subdev's state and set
> + * &struct v4l2_subdev->state to NULL.
> + */
> +void v4l2_subdev_free_state(struct v4l2_subdev *sd);
> +
> +/**
> + * v4l2_subdev_get_active_state() - Return the active subdev state for subdevice
> + * @sd: The subdevice
> + *
> + * Return the active state for the subdevice, or NULL if the subdev does not
> + * support active state.
> + */
> +static inline struct v4l2_subdev_state *
> +v4l2_subdev_get_active_state(struct v4l2_subdev *sd)
> +{
> +	return sd->state;
> +}
> +
>  #endif
> --
> 2.25.1
>

Re: [PATCH v8 04/36] media: subdev: pass also the active state to subdevs from ioctls

[Jacopo Mondi]

Hi Tomi

On Mon, Aug 30, 2021 at 02:00:44PM +0300, Tomi Valkeinen wrote:
> At the moment when a subdev op is called, the TRY subdev state
> (subdev_fh->state) is passed as a parameter even for ACTIVE case, or
> alternatively a NULL can be passed for ACTIVE case. This used to make
> sense, as the ACTIVE state was handled internally by the subdev drivers.
>
> We now have a state for ACTIVE case in a standard place, and can pass
> that alto to the drivers. This patch changes the subdev ioctls to either
> pass the TRY or ACTIVE state to the subdev.
>
> Unfortunately many drivers call ops from other subdevs, and implicitly
> pass NULL as the state, so this is just a partial solution. A coccinelle
> spatch could perhaps be created which fixes the drivers' subdev calls.
>
> For all current upstream drivers this doesn't matter, as they do not
> expect to get a valid state for ACTIVE case. But future drivers which
> support multiplexed streaming and routing will depend on getting a state
> for both active and try cases, and the simplest way to avoid this
> problem is to introduce a helper function, used by the new drivers,
> which makes sure the driver has either the TRY or ACTIVE state. This
> helper will be introduced in a follow-up patch.
>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 73 +++++++++++++++++++++++----
>  1 file changed, 63 insertions(+), 10 deletions(-)
>
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 04ad319fb150..b3637cddca58 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -353,6 +353,53 @@ const struct v4l2_subdev_ops v4l2_subdev_call_wrappers = {
>  EXPORT_SYMBOL(v4l2_subdev_call_wrappers);
>
>  #if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
> +
> +static struct v4l2_subdev_state *
> +subdev_ioctl_get_state(struct v4l2_subdev *sd, struct v4l2_subdev_fh *subdev_fh,
> +		       unsigned int cmd, void *arg)
> +{
> +	u32 which;
> +
> +	switch (cmd) {
> +	default:
> +		return NULL;
> +
> +	case VIDIOC_SUBDEV_G_FMT:
> +	case VIDIOC_SUBDEV_S_FMT: {
> +		which = ((struct v4l2_subdev_format *)arg)->which;
> +		break;
> +	}
> +	case VIDIOC_SUBDEV_G_CROP:
> +	case VIDIOC_SUBDEV_S_CROP: {
> +		which = ((struct v4l2_subdev_crop *)arg)->which;
> +		break;
> +	}
> +	case VIDIOC_SUBDEV_ENUM_MBUS_CODE: {
> +		which = ((struct v4l2_subdev_mbus_code_enum *)arg)->which;
> +		break;
> +	}
> +	case VIDIOC_SUBDEV_ENUM_FRAME_SIZE: {
> +		which = ((struct v4l2_subdev_frame_size_enum *)arg)->which;
> +		break;
> +	}
> +
> +	case VIDIOC_SUBDEV_ENUM_FRAME_INTERVAL: {
> +		which = ((struct v4l2_subdev_frame_interval_enum *)arg)->which;
> +		break;
> +	}
> +
> +	case VIDIOC_SUBDEV_G_SELECTION:
> +	case VIDIOC_SUBDEV_S_SELECTION: {
> +		which = ((struct v4l2_subdev_selection *)arg)->which;
> +		break;
> +	}
> +	}
> +
> +	return which == V4L2_SUBDEV_FORMAT_TRY ?
> +			     subdev_fh->state :
> +			     v4l2_subdev_get_active_state(sd);

Why this additional indirection layer ?

v4l2_subdev_get_active_state(struct v4l2_subdev *sd)
{
    return sd->state;
}

I understand you want to have the core to fish the 'right' state for
the drivers, but this then requires to protect against bridge drivers
calling an op through v4l2_subdev_call() with a NULL state by using
one more indirection like

	state = v4l2_subdev_validate_state(sd, state);

        static inline struct v4l2_subdev_state *
        v4l2_subdev_validate_state(struct v4l2_subdev *sd,
                                   struct v4l2_subdev_state *state)
        {
                return state ? state : sd->state;
        }

Which I very much don't like as it implicitly changes what state the
driver receives to work-around a design flaw (the fact that even if
the core tries to, there's no gurantee state is valid).

If feel like it would be much simpler if:

1) The core passes in a state which always come from the fh (the
   try_state) when it do_ioctl()

2) Drivers use their 'active' states embedded in the subdev or the
   'try' state passed in as parameter and decide
   which one to use based on the context. It's a pattern we have
   everywere already when using the per-fh try formats

	switch (which) {
	case V4L2_SUBDEV_FORMAT_TRY:
		return v4l2_subdev_get_try_format(&sd, sd_state, pad);
	case V4L2_SUBDEV_FORMAT_ACTIVE:
		return &sd->fmt;
	default:
		return NULL;
	}

I liked the idea to have the core pass in a state without the driver
having to care where it comes from, but it requires too many
indirections and implicities like the above shown
v4l2_subdev_validate_state().

One middle-ground could be to have the core pass in the 'correct' state as it
does in your series, and default it to sd->state if a bridge driver
calls an op through v4l2_subdev_call() with a NULL state, as the
format is implicitly ACTIVE in that case.

This ofc requires the state to be embedded (ie it's always there) and
that state-aware drivers to have properly initialized it, but that's a
given.

Nonetheless, this considerations do not defeat the purpose of having a
'state', as currently we have

struct v4l2_subdev_state {
        struct v4l2_subdev_krouting; /* Use for TRY and ACTIVE */
        struct v4l2_stream_configs; /* Use for ACTIVE */
        struct v4l2_pad_configs; /* Used for TRY */
};

and v4l2_stream_configs is a super-set of v4l2_pad_configs

If we could get to

struct v4l2_subdev_state {
        struct v4l2_subdev_krouting; /* Use for TRY and ACTIVE */
        struct v4l2_stream_configs; /* Use for TRY and ACTIVE */
};

This could turn out to be pretty neat, as it allows 'new' drivers to
maintain their current formats and routings in a subdev 'state'
instead of scattering those information in the driver-wide structure
as they currently do for formats, crops and whatnot. This can ofc go
on top.


> +}
> +
>  static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  {
>  	struct video_device *vdev = video_devdata(file);
> @@ -360,8 +407,11 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  	struct v4l2_fh *vfh = file->private_data;
>  	struct v4l2_subdev_fh *subdev_fh = to_v4l2_subdev_fh(vfh);
>  	bool ro_subdev = test_bit(V4L2_FL_SUBDEV_RO_DEVNODE, &vdev->flags);
> +	struct v4l2_subdev_state *state;
>  	int rval;
>
> +	state = subdev_ioctl_get_state(sd, subdev_fh, cmd, arg);
> +
>  	switch (cmd) {
>  	case VIDIOC_SUBDEV_QUERYCAP: {
>  		struct v4l2_subdev_capability *cap = arg;
> @@ -484,7 +534,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>
>  		memset(format->reserved, 0, sizeof(format->reserved));
>  		memset(format->format.reserved, 0, sizeof(format->format.reserved));
> -		return v4l2_subdev_call(sd, pad, get_fmt, subdev_fh->state, format);
> +		return v4l2_subdev_call(sd, pad, get_fmt, state, format);
>  	}
>
>  	case VIDIOC_SUBDEV_S_FMT: {
> @@ -495,7 +545,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>
>  		memset(format->reserved, 0, sizeof(format->reserved));
>  		memset(format->format.reserved, 0, sizeof(format->format.reserved));
> -		return v4l2_subdev_call(sd, pad, set_fmt, subdev_fh->state, format);
> +		return v4l2_subdev_call(sd, pad, set_fmt, state, format);
>  	}
>
>  	case VIDIOC_SUBDEV_G_CROP: {
> @@ -509,7 +559,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  		sel.target = V4L2_SEL_TGT_CROP;
>
>  		rval = v4l2_subdev_call(
> -			sd, pad, get_selection, subdev_fh->state, &sel);
> +			sd, pad, get_selection, state, &sel);
>
>  		crop->rect = sel.r;
>
> @@ -531,7 +581,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  		sel.r = crop->rect;
>
>  		rval = v4l2_subdev_call(
> -			sd, pad, set_selection, subdev_fh->state, &sel);
> +			sd, pad, set_selection, state, &sel);
>
>  		crop->rect = sel.r;
>
> @@ -542,7 +592,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  		struct v4l2_subdev_mbus_code_enum *code = arg;
>
>  		memset(code->reserved, 0, sizeof(code->reserved));
> -		return v4l2_subdev_call(sd, pad, enum_mbus_code, subdev_fh->state,
> +		return v4l2_subdev_call(sd, pad, enum_mbus_code, state,
>  					code);
>  	}
>
> @@ -550,7 +600,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  		struct v4l2_subdev_frame_size_enum *fse = arg;
>
>  		memset(fse->reserved, 0, sizeof(fse->reserved));
> -		return v4l2_subdev_call(sd, pad, enum_frame_size, subdev_fh->state,
> +		return v4l2_subdev_call(sd, pad, enum_frame_size, state,
>  					fse);
>  	}
>
> @@ -575,7 +625,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  		struct v4l2_subdev_frame_interval_enum *fie = arg;
>
>  		memset(fie->reserved, 0, sizeof(fie->reserved));
> -		return v4l2_subdev_call(sd, pad, enum_frame_interval, subdev_fh->state,
> +		return v4l2_subdev_call(sd, pad, enum_frame_interval, state,
>  					fie);
>  	}
>
> @@ -584,7 +634,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>
>  		memset(sel->reserved, 0, sizeof(sel->reserved));
>  		return v4l2_subdev_call(
> -			sd, pad, get_selection, subdev_fh->state, sel);
> +			sd, pad, get_selection, state, sel);
>  	}
>
>  	case VIDIOC_SUBDEV_S_SELECTION: {
> @@ -595,7 +645,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>
>  		memset(sel->reserved, 0, sizeof(sel->reserved));
>  		return v4l2_subdev_call(
> -			sd, pad, set_selection, subdev_fh->state, sel);
> +			sd, pad, set_selection, state, sel);
>  	}
>
>  	case VIDIOC_G_EDID: {
> @@ -829,10 +879,13 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
>  	if (is_media_entity_v4l2_subdev(pad->entity)) {
>  		struct v4l2_subdev *sd =
>  			media_entity_to_v4l2_subdev(pad->entity);
> +		struct v4l2_subdev_state *state;
> +
> +		state = v4l2_subdev_get_active_state(sd);
>
>  		fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
>  		fmt->pad = pad->index;
> -		return v4l2_subdev_call(sd, pad, get_fmt, NULL, fmt);
> +		return v4l2_subdev_call(sd, pad, get_fmt, state, fmt);
>  	}
>
>  	WARN(pad->entity->function != MEDIA_ENT_F_IO_V4L,
> --
> 2.25.1
>

Re: [PATCH v8 28/36] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Jacopo Mondi]

Hi Tomi,

On Mon, Aug 30, 2021 at 02:01:08PM +0300, Tomi Valkeinen wrote:
> From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
>
> Add support for subdev internal routing. A route is defined as a single
> stream from a sink pad to a source pad.
>
> The userspace can configure the routing via two new ioctls,
> VIDIOC_SUBDEV_G_ROUTING and VIDIOC_SUBDEV_S_ROUTING, and subdevs can
> implement the functionality with v4l2_subdev_pad_ops.set_routing().
>
> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Signed-off-by: Michal Simek <michal.simek@xilinx.com>
>
> - Add sink and source streams for multiplexed links
> - Copy the argument back in case of an error. This is needed to let the
>   caller know the number of routes.
>
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
>
> - Expand and refine documentation.
> - Make the 'routes' pointer a __u64 __user pointer so that a compat32
>   version of the ioctl is not required.
> - Add struct v4l2_subdev_krouting to be used for subdevice operations.
>
> Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
>
> - Fix typecasing warnings
> - Check sink & source pad types
> - Add 'which' field
> - Add V4L2_SUBDEV_ROUTE_FL_SOURCE
> - Routing to subdev state
> - Dropped get_routing subdev op
>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/v4l2-core/v4l2-ioctl.c  | 25 ++++++++-
>  drivers/media/v4l2-core/v4l2-subdev.c | 75 +++++++++++++++++++++++++++
>  include/media/v4l2-subdev.h           | 24 +++++++++
>  include/uapi/linux/v4l2-subdev.h      | 57 ++++++++++++++++++++
>  4 files changed, 180 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index 05d5db3d85e5..8e9315ffcb99 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -16,6 +16,7 @@
>  #include <linux/kernel.h>
>  #include <linux/version.h>
>
> +#include <linux/v4l2-subdev.h>
>  #include <linux/videodev2.h>
>
>  #include <media/v4l2-common.h>
> @@ -3065,6 +3066,21 @@ static int check_array_args(unsigned int cmd, void *parg, size_t *array_size,
>  		ret = 1;
>  		break;
>  	}
> +
> +	case VIDIOC_SUBDEV_G_ROUTING:
> +	case VIDIOC_SUBDEV_S_ROUTING: {
> +		struct v4l2_subdev_routing *routing = parg;
> +
> +		if (routing->num_routes > 256)
> +			return -EINVAL;
> +
> +		*user_ptr = u64_to_user_ptr(routing->routes);
> +		*kernel_ptr = (void **)&routing->routes;
> +		*array_size = sizeof(struct v4l2_subdev_route)
> +			    * routing->num_routes;
> +		ret = 1;
> +		break;
> +	}
>  	}
>
>  	return ret;
> @@ -3328,8 +3344,15 @@ video_usercopy(struct file *file, unsigned int orig_cmd, unsigned long arg,
>  	/*
>  	 * Some ioctls can return an error, but still have valid
>  	 * results that must be returned.
> +	 *
> +	 * FIXME: subdev IOCTLS are partially handled here and partially in
> +	 * v4l2-subdev.c and the 'always_copy' flag can only be set for IOCTLS
> +	 * defined here as part of the 'v4l2_ioctls' array. As
> +	 * VIDIOC_SUBDEV_G_ROUTING needs to return results to applications even
> +	 * in case of failure, but it is not defined here as part of the
> +	 * 'v4l2_ioctls' array, insert an ad-hoc check to address that.
>  	 */
> -	if (err < 0 && !always_copy)
> +	if (err < 0 && !always_copy && cmd != VIDIOC_SUBDEV_G_ROUTING)
>  		goto out;
>
>  out_array_args:
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index b1e65488210d..0e1f325b3159 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -395,6 +395,12 @@ subdev_ioctl_get_state(struct v4l2_subdev *sd, struct v4l2_subdev_fh *subdev_fh,
>  		which = ((struct v4l2_subdev_selection *)arg)->which;
>  		break;
>  	}
> +
> +	case VIDIOC_SUBDEV_G_ROUTING:
> +	case VIDIOC_SUBDEV_S_ROUTING: {
> +		which = ((struct v4l2_subdev_routing *)arg)->which;
> +		break;
> +	}
>  	}
>
>  	return which == V4L2_SUBDEV_FORMAT_TRY ?
> @@ -711,6 +717,74 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  	case VIDIOC_SUBDEV_QUERYSTD:
>  		return v4l2_subdev_call(sd, video, querystd, arg);
>
> +	case VIDIOC_SUBDEV_G_ROUTING: {
> +		struct v4l2_subdev_routing *routing = arg;
> +		struct v4l2_subdev_krouting *krouting;
> +
> +		if (!(sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED))
> +			return -ENOIOCTLCMD;
> +
> +		memset(routing->reserved, 0, sizeof(routing->reserved));
> +
> +		krouting = &state->routing;
> +
> +		if (routing->num_routes < krouting->num_routes) {
> +			routing->num_routes = krouting->num_routes;
> +			return -ENOSPC;
> +		}
> +
> +		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
> +		       krouting->routes,
> +		       krouting->num_routes * sizeof(*krouting->routes));
> +		routing->num_routes = krouting->num_routes;
> +
> +		return 0;
> +	}
> +
> +	case VIDIOC_SUBDEV_S_ROUTING: {
> +		struct v4l2_subdev_routing *routing = arg;
> +		struct v4l2_subdev_route *routes =
> +			(struct v4l2_subdev_route *)(uintptr_t)routing->routes;
> +		struct v4l2_subdev_krouting krouting = {};
> +		unsigned int i;
> +
> +		if (!(sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED))
> +			return -ENOIOCTLCMD;
> +
> +		if (routing->which != V4L2_SUBDEV_FORMAT_TRY && ro_subdev)
> +			return -EPERM;
> +
> +		memset(routing->reserved, 0, sizeof(routing->reserved));
> +
> +		for (i = 0; i < routing->num_routes; ++i) {
> +			const struct v4l2_subdev_route *route = &routes[i];
> +			const struct media_pad *pads = sd->entity.pads;
> +
> +			/* Do not check sink pad for source routes */
> +			if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE)) {
> +				if (route->sink_pad >= sd->entity.num_pads)
> +					return -EINVAL;
> +
> +				if (!(pads[route->sink_pad].flags &
> +				      MEDIA_PAD_FL_SINK))
> +					return -EINVAL;
> +			}
> +
> +			if (route->source_pad >= sd->entity.num_pads)
> +				return -EINVAL;
> +
> +			if (!(pads[route->source_pad].flags &
> +			      MEDIA_PAD_FL_SOURCE))
> +				return -EINVAL;
> +		}
> +
> +		krouting.which = routing->which;
> +		krouting.num_routes = routing->num_routes;
> +		krouting.routes = routes;
> +
> +		return v4l2_subdev_call(sd, pad, set_routing, state, &krouting);
> +	}
> +
>  	default:
>  		return v4l2_subdev_call(sd, core, ioctl, cmd, arg);
>  	}
> @@ -975,6 +1049,7 @@ void v4l2_free_subdev_state(struct v4l2_subdev_state *state)
>
>  	mutex_destroy(&state->lock);
>
> +	kvfree(state->routing.routes);
>  	kvfree(state->pads);
>  	kfree(state);
>  }
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 356901d8a948..cd6aad21ae0c 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -694,12 +694,29 @@ struct v4l2_subdev_pad_config {
>  	struct v4l2_rect try_compose;
>  };
>
> +/**
> + * struct v4l2_subdev_krouting - subdev routing table
> + *
> + * @which: format type (from enum v4l2_subdev_format_whence)
> + * @routes: &struct v4l2_subdev_route
> + * @num_routes: number of routes
> + *
> + * This structure is used to translate arguments received from
> + * VIDIOC_SUBDEV_G/S_ROUTING() ioctl to subdev device drivers operations.
> + */
> +struct v4l2_subdev_krouting {
> +	u32 which;
> +	struct v4l2_subdev_route *routes;
> +	unsigned int num_routes;
> +};
> +
>  /**
>   * struct v4l2_subdev_state - Used for storing subdev state information.
>   *
>   * @lock: mutex for the state
>   * @which: state type (from enum v4l2_subdev_format_whence)
>   * @pads: &struct v4l2_subdev_pad_config array
> + * @routing: routing table for the subdev
>   *
>   * This structure only needs to be passed to the pad op if the 'which' field
>   * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
> @@ -709,6 +726,7 @@ struct v4l2_subdev_state {
>  	struct mutex lock;
>  	u32 which;
>  	struct v4l2_subdev_pad_config *pads;
> +	struct v4l2_subdev_krouting routing;
>  };
>
>  /**
> @@ -772,6 +790,9 @@ struct v4l2_subdev_state {
>   *		     applied to the hardware. The operation shall fail if the
>   *		     pad index it has been called on is not valid or in case of
>   *		     unrecoverable failures.
> + *
> + * @set_routing: enable or disable data connection routes described in the
> + *		 subdevice routing table.
>   */
>  struct v4l2_subdev_pad_ops {
>  	int (*init_cfg)(struct v4l2_subdev *sd,
> @@ -816,6 +837,9 @@ struct v4l2_subdev_pad_ops {
>  			       struct v4l2_mbus_config *config);
>  	int (*set_mbus_config)(struct v4l2_subdev *sd, unsigned int pad,
>  			       struct v4l2_mbus_config *config);
> +	int (*set_routing)(struct v4l2_subdev *sd,
> +			   struct v4l2_subdev_state *state,
> +			   struct v4l2_subdev_krouting *route);
>  };
>
>  /**
> diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
> index 658106f5b5dc..3aa623e0e5f9 100644
> --- a/include/uapi/linux/v4l2-subdev.h
> +++ b/include/uapi/linux/v4l2-subdev.h
> @@ -188,6 +188,61 @@ struct v4l2_subdev_capability {
>  /* The v4l2 sub-device video device node is registered in read-only mode. */
>  #define V4L2_SUBDEV_CAP_RO_SUBDEV		0x00000001
>
> +/*
> + * Is the route active? An active route will start when streaming is enabled
> + * on a video node.
> + */
> +#define V4L2_SUBDEV_ROUTE_FL_ACTIVE		BIT(0)

Am I wrong or BIT() is not available to userspace ?

../../include/linux/v4l2-subdev.h:209:38: error: ‘BIT’ was not declared in this scope
  209 | #define V4L2_SUBDEV_ROUTE_FL_ACTIVE  BIT(0)

Header was exported with headers_install.

> +
> +/*
> + * Is the route immutable, i.e. can it be activated and inactivated?
> + * Set by the driver.
> + */
> +#define V4L2_SUBDEV_ROUTE_FL_IMMUTABLE		BIT(1)
> +
> +/*
> + * Is the route a source endpoint? A source endpoint route refers to a stream
> + * generated internally by the subdevice (usually a sensor), and thus there
> + * is no sink-side endpoint for the route. The sink_pad and sink_stream
> + * fields are unused.
> + * Set by the driver.
> + */
> +#define V4L2_SUBDEV_ROUTE_FL_SOURCE		BIT(2)
> +
> +/**
> + * struct v4l2_subdev_route - A route inside a subdev
> + *
> + * @sink_pad: the sink pad index
> + * @sink_stream: the sink stream identifier
> + * @source_pad: the source pad index
> + * @source_stream: the source stream identifier
> + * @flags: route flags V4L2_SUBDEV_ROUTE_FL_*
> + * @reserved: drivers and applications must zero this array
> + */
> +struct v4l2_subdev_route {
> +	__u32 sink_pad;
> +	__u32 sink_stream;
> +	__u32 source_pad;
> +	__u32 source_stream;
> +	__u32 flags;
> +	__u32 reserved[5];
> +};
> +
> +/**
> + * struct v4l2_subdev_routing - Subdev routing information
> + *
> + * @which: configuration type (from enum v4l2_subdev_format_whence)
> + * @routes: pointer to the routes array
> + * @num_routes: the total number of routes in the routes array
> + * @reserved: drivers and applications must zero this array
> + */
> +struct v4l2_subdev_routing {
> +	__u32 which;
> +	__u64 routes;
> +	__u32 num_routes;
> +	__u32 reserved[5];
> +};
> +
>  /* Backwards compatibility define --- to be removed */
>  #define v4l2_subdev_edid v4l2_edid
>
> @@ -203,6 +258,8 @@ struct v4l2_subdev_capability {
>  #define VIDIOC_SUBDEV_S_CROP			_IOWR('V', 60, struct v4l2_subdev_crop)
>  #define VIDIOC_SUBDEV_G_SELECTION		_IOWR('V', 61, struct v4l2_subdev_selection)
>  #define VIDIOC_SUBDEV_S_SELECTION		_IOWR('V', 62, struct v4l2_subdev_selection)
> +#define VIDIOC_SUBDEV_G_ROUTING			_IOWR('V', 38, struct v4l2_subdev_routing)
> +#define VIDIOC_SUBDEV_S_ROUTING			_IOWR('V', 39, struct v4l2_subdev_routing)
>  /* The following ioctls are identical to the ioctls in videodev2.h */
>  #define VIDIOC_SUBDEV_G_STD			_IOR('V', 23, v4l2_std_id)
>  #define VIDIOC_SUBDEV_S_STD			_IOW('V', 24, v4l2_std_id)
> --
> 2.25.1
>

Re: [PATCH v8 02/36] media: subdev: add active state to struct v4l2_subdev

[Tomi Valkeinen]

Hi,

On 15/09/2021 12:44, Jacopo Mondi wrote:
> Hi Tomi,
> 
> On Mon, Aug 30, 2021 at 02:00:42PM +0300, Tomi Valkeinen wrote:
>> Add a new 'state' field to struct v4l2_subdev to which we can store the
>> active state of a subdev. This will place the subdev configuration into
>> a known place, allowing us to use the state directly from the v4l2
>> framework, thus simplifying the drivers.
>>
>> We also add v4l2_subdev_alloc_state() and v4l2_subdev_free_state(),
>> which need to be used by the drivers that support subdev state in struct
>> v4l2_subdev.
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>>   drivers/media/v4l2-core/v4l2-subdev.c | 21 ++++++++++++++++
>>   include/media/v4l2-subdev.h           | 36 +++++++++++++++++++++++++++
>>   2 files changed, 57 insertions(+)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>> index 26a34a8e3d37..e1a794f69815 100644
>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>> @@ -943,3 +943,24 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
>>   	v4l2_subdev_notify(sd, V4L2_DEVICE_NOTIFY_EVENT, (void *)ev);
>>   }
>>   EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
>> +
>> +int v4l2_subdev_alloc_state(struct v4l2_subdev *sd)
>> +{
>> +	struct v4l2_subdev_state *state;
>> +
>> +	state = v4l2_alloc_subdev_state(sd);
> 
> So, I think this is one source of confusion about init_cfg.
> 
> v4l2_alloc_subdev_state() calls init_cfg() and 'state-aware' driver
> are now supposed to allocate their state by calling
> v4l2_subdev_alloc_state(), in the same way as the core does for the
> file-handle ones.
> 
> This will lead to init_cfg to be called for the 'active' (ie owned by
> the subdev) state, and then you need to add context to the state (by
> adding a 'which' field) to know what state you're dealing with.
> 
> According to the init_cfg() documentation
> 
>   * @init_cfg: initialize the pad config to default values
> 
> the op has to be called in order to initialize the per-file-handle
> context, not the active one.

I have missed updating the documentation there =).

> I would rather just embed 'struct v4l2_subdev_state' in 'struct
> v4l2_subdev', have the core going through the

Why would embedding the state change anything?

> 'v4l2_subdev_alloc_state()' to initialize the per-fh state, but have
> drivers initialize their own state at probe time. If they need for
> some reason to access their 'active' state at init_cfg() time, they
> caan fish it from their subdev.
> 
> If I'm not mistaken this will remove the need to have a which filed in
> the state, as I think the 'context' should be inferred from the
> 'which' argument embedded in the ops-specific structures, and not held
> in the state itself.

I'll comment on these in the next reply, as your second mail covered the 
same topics.

  Tomi

Re: [PATCH v8 04/36] media: subdev: pass also the active state to subdevs from ioctls

[Tomi Valkeinen]

Hi,

On 15/09/2021 13:17, Jacopo Mondi wrote:
> Hi Tomi
> 
> On Mon, Aug 30, 2021 at 02:00:44PM +0300, Tomi Valkeinen wrote:
>> At the moment when a subdev op is called, the TRY subdev state
>> (subdev_fh->state) is passed as a parameter even for ACTIVE case, or
>> alternatively a NULL can be passed for ACTIVE case. This used to make
>> sense, as the ACTIVE state was handled internally by the subdev drivers.
>>
>> We now have a state for ACTIVE case in a standard place, and can pass
>> that alto to the drivers. This patch changes the subdev ioctls to either
>> pass the TRY or ACTIVE state to the subdev.
>>
>> Unfortunately many drivers call ops from other subdevs, and implicitly
>> pass NULL as the state, so this is just a partial solution. A coccinelle
>> spatch could perhaps be created which fixes the drivers' subdev calls.
>>
>> For all current upstream drivers this doesn't matter, as they do not
>> expect to get a valid state for ACTIVE case. But future drivers which
>> support multiplexed streaming and routing will depend on getting a state
>> for both active and try cases, and the simplest way to avoid this
>> problem is to introduce a helper function, used by the new drivers,
>> which makes sure the driver has either the TRY or ACTIVE state. This
>> helper will be introduced in a follow-up patch.
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>>   drivers/media/v4l2-core/v4l2-subdev.c | 73 +++++++++++++++++++++++----
>>   1 file changed, 63 insertions(+), 10 deletions(-)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>> index 04ad319fb150..b3637cddca58 100644
>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>> @@ -353,6 +353,53 @@ const struct v4l2_subdev_ops v4l2_subdev_call_wrappers = {
>>   EXPORT_SYMBOL(v4l2_subdev_call_wrappers);
>>
>>   #if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
>> +
>> +static struct v4l2_subdev_state *
>> +subdev_ioctl_get_state(struct v4l2_subdev *sd, struct v4l2_subdev_fh *subdev_fh,
>> +		       unsigned int cmd, void *arg)
>> +{
>> +	u32 which;
>> +
>> +	switch (cmd) {
>> +	default:
>> +		return NULL;
>> +
>> +	case VIDIOC_SUBDEV_G_FMT:
>> +	case VIDIOC_SUBDEV_S_FMT: {
>> +		which = ((struct v4l2_subdev_format *)arg)->which;
>> +		break;
>> +	}
>> +	case VIDIOC_SUBDEV_G_CROP:
>> +	case VIDIOC_SUBDEV_S_CROP: {
>> +		which = ((struct v4l2_subdev_crop *)arg)->which;
>> +		break;
>> +	}
>> +	case VIDIOC_SUBDEV_ENUM_MBUS_CODE: {
>> +		which = ((struct v4l2_subdev_mbus_code_enum *)arg)->which;
>> +		break;
>> +	}
>> +	case VIDIOC_SUBDEV_ENUM_FRAME_SIZE: {
>> +		which = ((struct v4l2_subdev_frame_size_enum *)arg)->which;
>> +		break;
>> +	}
>> +
>> +	case VIDIOC_SUBDEV_ENUM_FRAME_INTERVAL: {
>> +		which = ((struct v4l2_subdev_frame_interval_enum *)arg)->which;
>> +		break;
>> +	}
>> +
>> +	case VIDIOC_SUBDEV_G_SELECTION:
>> +	case VIDIOC_SUBDEV_S_SELECTION: {
>> +		which = ((struct v4l2_subdev_selection *)arg)->which;
>> +		break;
>> +	}
>> +	}
>> +
>> +	return which == V4L2_SUBDEV_FORMAT_TRY ?
>> +			     subdev_fh->state :
>> +			     v4l2_subdev_get_active_state(sd);
> 
> Why this additional indirection layer ?
> 
> v4l2_subdev_get_active_state(struct v4l2_subdev *sd)
> {
>      return sd->state;
> }

I wanted to hide all direct accesses to the state to make it easier to 
figure out how and where the state is accessed.

> I understand you want to have the core to fish the 'right' state for
> the drivers, but this then requires to protect against bridge drivers
> calling an op through v4l2_subdev_call() with a NULL state by using
> one more indirection like
> 
> 	state = v4l2_subdev_validate_state(sd, state);
> 
>          static inline struct v4l2_subdev_state *
>          v4l2_subdev_validate_state(struct v4l2_subdev *sd,
>                                     struct v4l2_subdev_state *state)
>          {
>                  return state ? state : sd->state;
>          }
> 
> Which I very much don't like as it implicitly changes what state the
> driver receives to work-around a design flaw (the fact that even if
> the core tries to, there's no gurantee state is valid).

I don't like it either. My idea was that in the future the subdevs would 
always get the correct state. In other words, all the subdev drivers 
calling ops in other subdevs would be changed to pass the state 
correctly. Thus the v4l2_subdev_validate_state() is a helper for the 
transition period, which can easily be dropped when the drivers work 
correctly.

> If feel like it would be much simpler if:
> 
> 1) The core passes in a state which always come from the fh (the
>     try_state) when it do_ioctl()
> 
> 2) Drivers use their 'active' states embedded in the subdev or the
>     'try' state passed in as parameter and decide
>     which one to use based on the context. It's a pattern we have
>     everywere already when using the per-fh try formats
> 
> 	switch (which) {
> 	case V4L2_SUBDEV_FORMAT_TRY:
> 		return v4l2_subdev_get_try_format(&sd, sd_state, pad);
> 	case V4L2_SUBDEV_FORMAT_ACTIVE:
> 		return &sd->fmt;
> 	default:
> 		return NULL;
> 	}

This is possible, of course. We could do this if we decide we don't want 
the subdev drivers to pass the state properly in the future.

However, if, in my series, I currently call this in a subdev driver:

state = v4l2_subdev_validate_state(sd, state);

With the change you suggest I'd just do (possibly with a helper):

state = which == V4L2_SUBDEV_FORMAT_TRY ? state : sd->state;

Is it any better?

> I liked the idea to have the core pass in a state without the driver
> having to care where it comes from, but it requires too many
> indirections and implicities like the above shown
> v4l2_subdev_validate_state().
> 
> One middle-ground could be to have the core pass in the 'correct' state as it
> does in your series, and default it to sd->state if a bridge driver
> calls an op through v4l2_subdev_call() with a NULL state, as the
> format is implicitly ACTIVE in that case.

If you mean changing all the bridge drivers so that they would give the 
state properly, yes, that was my plan (I think I mentioned it in a 
commit desc, perhaps). It's not a trivial change, though, as 
v4l2_subdev_call() cannot handle this at the moment.

I believe it should be doable with coccinelle. Maybe add a new macro, 
v4l2_subdev_call_state() or such, which gives the active state in the 
second parameter (looks like all the ops have the state as the second 
param). Then use coccinelle to find all the v4l2_subdev_call uses which 
call ops that get a state, verify that the current caller uses NULL as 
the state, and change v4l2_subdev_call to v4l2_subdev_call_state.

> This ofc requires the state to be embedded (ie it's always there) and
> that state-aware drivers to have properly initialized it, but that's a
> given.

Why does the state need to be embedded? If the subdev driver is not 
state-aware, it does not expect to get a state except for the TRY case. 
Passing NULL for those drivers should be fine.

> Nonetheless, this considerations do not defeat the purpose of having a
> 'state', as currently we have
> 
> struct v4l2_subdev_state {
>          struct v4l2_subdev_krouting; /* Use for TRY and ACTIVE */
>          struct v4l2_stream_configs; /* Use for ACTIVE */

stream_configs is used for TRY also.

>          struct v4l2_pad_configs; /* Used for TRY */

Probably no point in this, but this _could_ also be used for ACTIVE. We 
could have state aware drivers that don't use routing or streams, and 
use just a plain old pad_configs array. This would allow moving the 
ACTIVE pad_configs from the driver to the core.

But, as you suggest, probably a better direction is to try to get rid of 
pad_configs instead.

> };
> 
> and v4l2_stream_configs is a super-set of v4l2_pad_configs
> 
> If we could get to
> 
> struct v4l2_subdev_state {
>          struct v4l2_subdev_krouting; /* Use for TRY and ACTIVE */
>          struct v4l2_stream_configs; /* Use for TRY and ACTIVE */
> };
> 
> This could turn out to be pretty neat, as it allows 'new' drivers to
> maintain their current formats and routings in a subdev 'state'
> instead of scattering those information in the driver-wide structure
> as they currently do for formats, crops and whatnot. This can ofc go
> on top.

Yes, that's the long term plan, but it's a huge change. And when I say 
plan, I don't mean I'm planning to change all the current drivers, I'm 
just saying my series is designed so that it allows these to be done in 
the future.

  Tomi

Re: [PATCH v8 02/36] media: subdev: add active state to struct v4l2_subdev

[Tomi Valkeinen]

On 16/09/2021 09:17, Tomi Valkeinen wrote:
> Hi,
> 
> On 15/09/2021 12:44, Jacopo Mondi wrote:
>> Hi Tomi,
>>
>> On Mon, Aug 30, 2021 at 02:00:42PM +0300, Tomi Valkeinen wrote:
>>> Add a new 'state' field to struct v4l2_subdev to which we can store the
>>> active state of a subdev. This will place the subdev configuration into
>>> a known place, allowing us to use the state directly from the v4l2
>>> framework, thus simplifying the drivers.
>>>
>>> We also add v4l2_subdev_alloc_state() and v4l2_subdev_free_state(),
>>> which need to be used by the drivers that support subdev state in struct
>>> v4l2_subdev.
>>>
>>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>>> ---
>>>   drivers/media/v4l2-core/v4l2-subdev.c | 21 ++++++++++++++++
>>>   include/media/v4l2-subdev.h           | 36 +++++++++++++++++++++++++++
>>>   2 files changed, 57 insertions(+)
>>>
>>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c 
>>> b/drivers/media/v4l2-core/v4l2-subdev.c
>>> index 26a34a8e3d37..e1a794f69815 100644
>>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>>> @@ -943,3 +943,24 @@ void v4l2_subdev_notify_event(struct v4l2_subdev 
>>> *sd,
>>>       v4l2_subdev_notify(sd, V4L2_DEVICE_NOTIFY_EVENT, (void *)ev);
>>>   }
>>>   EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
>>> +
>>> +int v4l2_subdev_alloc_state(struct v4l2_subdev *sd)
>>> +{
>>> +    struct v4l2_subdev_state *state;
>>> +
>>> +    state = v4l2_alloc_subdev_state(sd);

Replying to this again, as the second email didn't actually cover all 
the topics...

>> So, I think this is one source of confusion about init_cfg.
>>
>> v4l2_alloc_subdev_state() calls init_cfg() and 'state-aware' driver
>> are now supposed to allocate their state by calling
>> v4l2_subdev_alloc_state(), in the same way as the core does for the
>> file-handle ones.
>>
>> This will lead to init_cfg to be called for the 'active' (ie owned by
>> the subdev) state, and then you need to add context to the state (by
>> adding a 'which' field) to know what state you're dealing with.
>>
>> According to the init_cfg() documentation
>>
>>   * @init_cfg: initialize the pad config to default values
>>
>> the op has to be called in order to initialize the per-file-handle
>> context, not the active one.
> 
> I have missed updating the documentation there =).

The documentation above doesn't imply per-file-handle context or TRY 
case, afaics. It just says "initialize state to default". Unless "pad 
config" always means TRY, which I think it doesn't as the drivers have 
internally been using pad configs.

But it's true that so far init_cfg has only been called for TRY case, 
and perhaps that's enough of a reason to keep it so.

>> I would rather just embed 'struct v4l2_subdev_state' in 'struct
>> v4l2_subdev', have the core going through the
> 
> Why would embedding the state change anything?
> 
>> 'v4l2_subdev_alloc_state()' to initialize the per-fh state, but have
>> drivers initialize their own state at probe time. If they need for
>> some reason to access their 'active' state at init_cfg() time, they
>> caan fish it from their subdev.
>>
>> If I'm not mistaken this will remove the need to have a which filed in
>> the state, as I think the 'context' should be inferred from the
>> 'which' argument embedded in the ops-specific structures, and not held
>> in the state itself.

It's true that the state's which field is mainly (probably only) needed 
for handling the init_cfg. It could be solved in other ways too:

- New subdev op to initialize active state
- New subdev op which gets 'which' as a parameter, to initialize both 
states (state-aware drivers wouldn't need to implement the old init_cfg)
- Coccinelle to change init_cfg to get the which as a parameter

Without doing any deep thinking, the middle one sounds best to me.

  Tomi

Re: [PATCH v8 28/36] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Tomi Valkeinen]

Hi,

On 15/09/2021 19:10, Jacopo Mondi wrote:

>> +#define V4L2_SUBDEV_ROUTE_FL_ACTIVE		BIT(0)
> 
> Am I wrong or BIT() is not available to userspace ?
> 
> ../../include/linux/v4l2-subdev.h:209:38: error: ‘BIT’ was not declared in this scope
>    209 | #define V4L2_SUBDEV_ROUTE_FL_ACTIVE  BIT(0)
> 
> Header was exported with headers_install.

Yes, I think you are correct. I see only two other BIT() uses in uapi 
headers, and those might be errors too.

  Tomi

Re: [PATCH v8 04/36] media: subdev: pass also the active state to subdevs from ioctls

[Jacopo Mondi]

Hi Tomi,

On Thu, Sep 16, 2021 at 09:44:23AM +0300, Tomi Valkeinen wrote:
> Hi,
>
> On 15/09/2021 13:17, Jacopo Mondi wrote:
> > Hi Tomi
> >
> > On Mon, Aug 30, 2021 at 02:00:44PM +0300, Tomi Valkeinen wrote:
> > > At the moment when a subdev op is called, the TRY subdev state
> > > (subdev_fh->state) is passed as a parameter even for ACTIVE case, or
> > > alternatively a NULL can be passed for ACTIVE case. This used to make
> > > sense, as the ACTIVE state was handled internally by the subdev drivers.
> > >
> > > We now have a state for ACTIVE case in a standard place, and can pass
> > > that alto to the drivers. This patch changes the subdev ioctls to either
> > > pass the TRY or ACTIVE state to the subdev.
> > >
> > > Unfortunately many drivers call ops from other subdevs, and implicitly
> > > pass NULL as the state, so this is just a partial solution. A coccinelle
> > > spatch could perhaps be created which fixes the drivers' subdev calls.
> > >
> > > For all current upstream drivers this doesn't matter, as they do not
> > > expect to get a valid state for ACTIVE case. But future drivers which
> > > support multiplexed streaming and routing will depend on getting a state
> > > for both active and try cases, and the simplest way to avoid this
> > > problem is to introduce a helper function, used by the new drivers,
> > > which makes sure the driver has either the TRY or ACTIVE state. This
> > > helper will be introduced in a follow-up patch.
> > >
> > > Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > > ---
> > >   drivers/media/v4l2-core/v4l2-subdev.c | 73 +++++++++++++++++++++++----
> > >   1 file changed, 63 insertions(+), 10 deletions(-)
> > >
> > > diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> > > index 04ad319fb150..b3637cddca58 100644
> > > --- a/drivers/media/v4l2-core/v4l2-subdev.c
> > > +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> > > @@ -353,6 +353,53 @@ const struct v4l2_subdev_ops v4l2_subdev_call_wrappers = {
> > >   EXPORT_SYMBOL(v4l2_subdev_call_wrappers);
> > >
> > >   #if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
> > > +
> > > +static struct v4l2_subdev_state *
> > > +subdev_ioctl_get_state(struct v4l2_subdev *sd, struct v4l2_subdev_fh *subdev_fh,
> > > +		       unsigned int cmd, void *arg)
> > > +{
> > > +	u32 which;
> > > +
> > > +	switch (cmd) {
> > > +	default:
> > > +		return NULL;
> > > +
> > > +	case VIDIOC_SUBDEV_G_FMT:
> > > +	case VIDIOC_SUBDEV_S_FMT: {
> > > +		which = ((struct v4l2_subdev_format *)arg)->which;
> > > +		break;
> > > +	}
> > > +	case VIDIOC_SUBDEV_G_CROP:
> > > +	case VIDIOC_SUBDEV_S_CROP: {
> > > +		which = ((struct v4l2_subdev_crop *)arg)->which;
> > > +		break;
> > > +	}
> > > +	case VIDIOC_SUBDEV_ENUM_MBUS_CODE: {
> > > +		which = ((struct v4l2_subdev_mbus_code_enum *)arg)->which;
> > > +		break;
> > > +	}
> > > +	case VIDIOC_SUBDEV_ENUM_FRAME_SIZE: {
> > > +		which = ((struct v4l2_subdev_frame_size_enum *)arg)->which;
> > > +		break;
> > > +	}
> > > +
> > > +	case VIDIOC_SUBDEV_ENUM_FRAME_INTERVAL: {
> > > +		which = ((struct v4l2_subdev_frame_interval_enum *)arg)->which;
> > > +		break;
> > > +	}
> > > +
> > > +	case VIDIOC_SUBDEV_G_SELECTION:
> > > +	case VIDIOC_SUBDEV_S_SELECTION: {
> > > +		which = ((struct v4l2_subdev_selection *)arg)->which;
> > > +		break;
> > > +	}
> > > +	}
> > > +
> > > +	return which == V4L2_SUBDEV_FORMAT_TRY ?
> > > +			     subdev_fh->state :
> > > +			     v4l2_subdev_get_active_state(sd);
> >
> > Why this additional indirection layer ?
> >
> > v4l2_subdev_get_active_state(struct v4l2_subdev *sd)
> > {
> >      return sd->state;
> > }
>
> I wanted to hide all direct accesses to the state to make it easier to
> figure out how and where the state is accessed.
>
> > I understand you want to have the core to fish the 'right' state for
> > the drivers, but this then requires to protect against bridge drivers
> > calling an op through v4l2_subdev_call() with a NULL state by using
> > one more indirection like
> >
> > 	state = v4l2_subdev_validate_state(sd, state);
> >
> >          static inline struct v4l2_subdev_state *
> >          v4l2_subdev_validate_state(struct v4l2_subdev *sd,
> >                                     struct v4l2_subdev_state *state)
> >          {
> >                  return state ? state : sd->state;
> >          }
> >
> > Which I very much don't like as it implicitly changes what state the
> > driver receives to work-around a design flaw (the fact that even if
> > the core tries to, there's no gurantee state is valid).
>
> I don't like it either. My idea was that in the future the subdevs would
> always get the correct state. In other words, all the subdev drivers calling
> ops in other subdevs would be changed to pass the state correctly. Thus the
> v4l2_subdev_validate_state() is a helper for the transition period, which
> can easily be dropped when the drivers work correctly.

Most of the drivers that call v4l2_subdev_call() with a NULL state are
bridge drivers operating in the ACTIVE use case. Even if we get to a
point where we remove all calls passing in a NULL state, what are the
bridges expected to provide as a state to the subdev they call
operations on ? The subdev's state as well ? something like

        v4l2_subdev_call(sd, pad, set_fmt, sd->state, ...)

With your current dynamicaly allocated state, sd->state could very well
be NULL.

I still think this could be way simpler if we assume that the state
received as parameter is the file-handle's one (like it was for
pad_configs) and in the active case we let driver use their own
sd->state.

>
> > If feel like it would be much simpler if:
> >
> > 1) The core passes in a state which always come from the fh (the
> >     try_state) when it do_ioctl()
> >
> > 2) Drivers use their 'active' states embedded in the subdev or the
> >     'try' state passed in as parameter and decide
> >     which one to use based on the context. It's a pattern we have
> >     everywere already when using the per-fh try formats
> >
> > 	switch (which) {
> > 	case V4L2_SUBDEV_FORMAT_TRY:
> > 		return v4l2_subdev_get_try_format(&sd, sd_state, pad);
> > 	case V4L2_SUBDEV_FORMAT_ACTIVE:
> > 		return &sd->fmt;
> > 	default:
> > 		return NULL;
> > 	}
>
> This is possible, of course. We could do this if we decide we don't want the
> subdev drivers to pass the state properly in the future.
>
> However, if, in my series, I currently call this in a subdev driver:
>
> state = v4l2_subdev_validate_state(sd, state);
>
> With the change you suggest I'd just do (possibly with a helper):
>
> state = which == V4L2_SUBDEV_FORMAT_TRY ? state : sd->state;
>
> Is it any better?
>
> > I liked the idea to have the core pass in a state without the driver
> > having to care where it comes from, but it requires too many
> > indirections and implicities like the above shown
> > v4l2_subdev_validate_state().
> >
> > One middle-ground could be to have the core pass in the 'correct' state as it
> > does in your series, and default it to sd->state if a bridge driver
> > calls an op through v4l2_subdev_call() with a NULL state, as the
> > format is implicitly ACTIVE in that case.
>
> If you mean changing all the bridge drivers so that they would give the
> state properly, yes, that was my plan (I think I mentioned it in a commit
> desc, perhaps). It's not a trivial change, though, as v4l2_subdev_call()
> cannot handle this at the moment.

Unfortunately this cannot be done automatically in v4l2_subdev_call(),
at least not easily.

>
> I believe it should be doable with coccinelle. Maybe add a new macro,
> v4l2_subdev_call_state() or such, which gives the active state in the second
> parameter (looks like all the ops have the state as the second param). Then
> use coccinelle to find all the v4l2_subdev_call uses which call ops that get
> a state, verify that the current caller uses NULL as the state, and change
> v4l2_subdev_call to v4l2_subdev_call_state.


Even if we beautify it, I think bridge drivers passing as parameter to
a subdev operation a subdev attribute, like in the above shown

        v4l2_subdev_call(sd, pad, set_fmt, sd->state, ...)

is unecessary and a possible source of confusion, with the subdev
driver having to infer where the state comes from and the possibility
of it being NULL anyway if the bridge operates with a non-state aware
subdev which has not allocated a state (which is harmelss now, as they
won't be interested in 'state').

It could be made easier if we clearly say drivers "if it's TRY, expect
a state, if is ACTIVE use your own one (if you want to)". This seems
impossible to get wrong to me for subdev drivers.

>
> > This ofc requires the state to be embedded (ie it's always there) and
> > that state-aware drivers to have properly initialized it, but that's a
> > given.
>
> Why does the state need to be embedded? If the subdev driver is not
> state-aware, it does not expect to get a state except for the TRY case.
> Passing NULL for those drivers should be fine.
>

It doesn't -need- to be, I just think it avoids allocation and
releasing at run-time and offers a place where to store subdev-wide
configurations to all drivers as an opt-in feature.

Of course we pay a little price in the size of the subdev, but it's
all in-kernel stuff and going forward the state could very wel just
become the standard 'subdev_config'

        struct v4l2_subdev {
                ....

                struct v4l2_subdev_config {
                        struct v4l2_subdev_routing routes;
                        struct v4l2_subdev_streams streams;
                } config;
        };

But yeah, allocated or embedded is tangential and I defer this call to
maintainers which know better than me for sure.

> > Nonetheless, this considerations do not defeat the purpose of having a
> > 'state', as currently we have
> >
> > struct v4l2_subdev_state {
> >          struct v4l2_subdev_krouting; /* Use for TRY and ACTIVE */
> >          struct v4l2_stream_configs; /* Use for ACTIVE */
>
> stream_configs is used for TRY also.
>
> >          struct v4l2_pad_configs; /* Used for TRY */
>
> Probably no point in this, but this _could_ also be used for ACTIVE. We
> could have state aware drivers that don't use routing or streams, and use
> just a plain old pad_configs array. This would allow moving the ACTIVE
> pad_configs from the driver to the core.

That would be nice, but it would be better is stream_configs could be
used for pad-only drivers (it's just about assuming stream = 0 for all
of them, right ?). But yes, my point is about trying to centralize the
subdev configuration in one place. But that's probably for later
indeed.

Thanks
  j

>
> But, as you suggest, probably a better direction is to try to get rid of
> pad_configs instead.
>
> > };
> >
> > and v4l2_stream_configs is a super-set of v4l2_pad_configs
> >
> > If we could get to
> >
> > struct v4l2_subdev_state {
> >          struct v4l2_subdev_krouting; /* Use for TRY and ACTIVE */
> >          struct v4l2_stream_configs; /* Use for TRY and ACTIVE */
> > };
> >
> > This could turn out to be pretty neat, as it allows 'new' drivers to
> > maintain their current formats and routings in a subdev 'state'
> > instead of scattering those information in the driver-wide structure
> > as they currently do for formats, crops and whatnot. This can ofc go
> > on top.
>
> Yes, that's the long term plan, but it's a huge change. And when I say plan,
> I don't mean I'm planning to change all the current drivers, I'm just saying
> my series is designed so that it allows these to be done in the future.
>
>  Tomi

Re: [PATCH v8 02/36] media: subdev: add active state to struct v4l2_subdev

[Jacopo Mondi]

Hi Tomi,

On Thu, Sep 16, 2021 at 09:52:42AM +0300, Tomi Valkeinen wrote:
> On 16/09/2021 09:17, Tomi Valkeinen wrote:
> > Hi,
> >
> > On 15/09/2021 12:44, Jacopo Mondi wrote:
> > > Hi Tomi,
> > >
> > > On Mon, Aug 30, 2021 at 02:00:42PM +0300, Tomi Valkeinen wrote:
> > > > Add a new 'state' field to struct v4l2_subdev to which we can store the
> > > > active state of a subdev. This will place the subdev configuration into
> > > > a known place, allowing us to use the state directly from the v4l2
> > > > framework, thus simplifying the drivers.
> > > >
> > > > We also add v4l2_subdev_alloc_state() and v4l2_subdev_free_state(),
> > > > which need to be used by the drivers that support subdev state in struct
> > > > v4l2_subdev.
> > > >
> > > > Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > > > ---
> > > >   drivers/media/v4l2-core/v4l2-subdev.c | 21 ++++++++++++++++
> > > >   include/media/v4l2-subdev.h           | 36 +++++++++++++++++++++++++++
> > > >   2 files changed, 57 insertions(+)
> > > >
> > > > diff --git a/drivers/media/v4l2-core/v4l2-subdev.c
> > > > b/drivers/media/v4l2-core/v4l2-subdev.c
> > > > index 26a34a8e3d37..e1a794f69815 100644
> > > > --- a/drivers/media/v4l2-core/v4l2-subdev.c
> > > > +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> > > > @@ -943,3 +943,24 @@ void v4l2_subdev_notify_event(struct
> > > > v4l2_subdev *sd,
> > > >       v4l2_subdev_notify(sd, V4L2_DEVICE_NOTIFY_EVENT, (void *)ev);
> > > >   }
> > > >   EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
> > > > +
> > > > +int v4l2_subdev_alloc_state(struct v4l2_subdev *sd)
> > > > +{
> > > > +    struct v4l2_subdev_state *state;
> > > > +
> > > > +    state = v4l2_alloc_subdev_state(sd);
>
> Replying to this again, as the second email didn't actually cover all the
> topics...
>
> > > So, I think this is one source of confusion about init_cfg.
> > >
> > > v4l2_alloc_subdev_state() calls init_cfg() and 'state-aware' driver
> > > are now supposed to allocate their state by calling
> > > v4l2_subdev_alloc_state(), in the same way as the core does for the
> > > file-handle ones.
> > >
> > > This will lead to init_cfg to be called for the 'active' (ie owned by
> > > the subdev) state, and then you need to add context to the state (by
> > > adding a 'which' field) to know what state you're dealing with.
> > >
> > > According to the init_cfg() documentation
> > >
> > >   * @init_cfg: initialize the pad config to default values
> > >
> > > the op has to be called in order to initialize the per-file-handle
> > > context, not the active one.
> >
> > I have missed updating the documentation there =).
>
> The documentation above doesn't imply per-file-handle context or TRY case,
> afaics. It just says "initialize state to default". Unless "pad config"
> always means TRY, which I think it doesn't as the drivers have internally
> been using pad configs.

If they do, they would have the 'active' pad_configs  allocated or
embedded somewhere in their driver structures, they would not receive
it as parameter. Or have I missed where the core is capable of fishing
the 'right' pad_configs ? I think the same should happen for state.

>
> But it's true that so far init_cfg has only been called for TRY case, and
> perhaps that's enough of a reason to keep it so.
>
> > > I would rather just embed 'struct v4l2_subdev_state' in 'struct
> > > v4l2_subdev', have the core going through the
> >
> > Why would embedding the state change anything?
> >
> > > 'v4l2_subdev_alloc_state()' to initialize the per-fh state, but have
> > > drivers initialize their own state at probe time. If they need for
> > > some reason to access their 'active' state at init_cfg() time, they
> > > caan fish it from their subdev.
> > >
> > > If I'm not mistaken this will remove the need to have a which filed in
> > > the state, as I think the 'context' should be inferred from the
> > > 'which' argument embedded in the ops-specific structures, and not held
> > > in the state itself.
>
> It's true that the state's which field is mainly (probably only) needed for
> handling the init_cfg. It could be solved in other ways too:
>
> - New subdev op to initialize active state
> - New subdev op which gets 'which' as a parameter, to initialize both states
> (state-aware drivers wouldn't need to implement the old init_cfg)
> - Coccinelle to change init_cfg to get the which as a parameter
>
> Without doing any deep thinking, the middle one sounds best to me.

Isn't it simpler if you just don't call init_cfg for the 'active'
state ? Driver will initialize them at probe time and that's it, then
you can remove 'which' from the state (and from routing tables too if I'm
not mistaken).

Thanks
  j

>
>  Tomi

Re: [PATCH v8 04/36] media: subdev: pass also the active state to subdevs from ioctls

[Tomi Valkeinen]

Hi,

On 16/09/2021 11:02, Jacopo Mondi wrote:

>> I don't like it either. My idea was that in the future the subdevs would
>> always get the correct state. In other words, all the subdev drivers calling
>> ops in other subdevs would be changed to pass the state correctly. Thus the
>> v4l2_subdev_validate_state() is a helper for the transition period, which
>> can easily be dropped when the drivers work correctly.
> 
> Most of the drivers that call v4l2_subdev_call() with a NULL state are
> bridge drivers operating in the ACTIVE use case. Even if we get to a
> point where we remove all calls passing in a NULL state, what are the
> bridges expected to provide as a state to the subdev they call
> operations on ? The subdev's state as well ? something like
> 
>          v4l2_subdev_call(sd, pad, set_fmt, sd->state, ...)

Yes. Although we should hide it, so that when calling ops that support 
state, the subdev drivers do:

v4l2_subdev_call_state(sd, pad, set_fmt, ...)

and v4l2_subdev_call_state macro (maybe needs a better name...) uses 
sd->state as the second parameter to the op.

> With your current dynamicaly allocated state, sd->state could very well
> be NULL.

Yes, that sounds logical to me. The subdev drivers don't have active 
state, and th driver code doesn't use it, so they get NULL.

> I still think this could be way simpler if we assume that the state
> received as parameter is the file-handle's one (like it was for
> pad_configs) and in the active case we let driver use their own
> sd->state.

I'm kind of okay-ish with that too.

It doesn't feel logical to me, and afaik the drivers should not touch 
the file-handle's state when dealing with active case so passing it is 
kind of wrong, but I agree that it is how things have been.

I don't think it's any simpler, though. This change wouldn't affect the 
old drivers, and the new drivers would just use another helper instead 
of v4l2_subdev_validate_state. And if we change the v4l2_subdev_call() 
call as discussed above, the new drivers can drop the 
v4l2_subdev_validate_state().

So I would argue that the new approach is (will be) simpler, but it's 
different than what we have now.

>>> If feel like it would be much simpler if:
>>>
>>> 1) The core passes in a state which always come from the fh (the
>>>      try_state) when it do_ioctl()
>>>
>>> 2) Drivers use their 'active' states embedded in the subdev or the
>>>      'try' state passed in as parameter and decide
>>>      which one to use based on the context. It's a pattern we have
>>>      everywere already when using the per-fh try formats
>>>
>>> 	switch (which) {
>>> 	case V4L2_SUBDEV_FORMAT_TRY:
>>> 		return v4l2_subdev_get_try_format(&sd, sd_state, pad);
>>> 	case V4L2_SUBDEV_FORMAT_ACTIVE:
>>> 		return &sd->fmt;
>>> 	default:
>>> 		return NULL;
>>> 	}
>>
>> This is possible, of course. We could do this if we decide we don't want the
>> subdev drivers to pass the state properly in the future.
>>
>> However, if, in my series, I currently call this in a subdev driver:
>>
>> state = v4l2_subdev_validate_state(sd, state);
>>
>> With the change you suggest I'd just do (possibly with a helper):
>>
>> state = which == V4L2_SUBDEV_FORMAT_TRY ? state : sd->state;
>>
>> Is it any better?
>>
>>> I liked the idea to have the core pass in a state without the driver
>>> having to care where it comes from, but it requires too many
>>> indirections and implicities like the above shown
>>> v4l2_subdev_validate_state().
>>>
>>> One middle-ground could be to have the core pass in the 'correct' state as it
>>> does in your series, and default it to sd->state if a bridge driver
>>> calls an op through v4l2_subdev_call() with a NULL state, as the
>>> format is implicitly ACTIVE in that case.
>>
>> If you mean changing all the bridge drivers so that they would give the
>> state properly, yes, that was my plan (I think I mentioned it in a commit
>> desc, perhaps). It's not a trivial change, though, as v4l2_subdev_call()
>> cannot handle this at the moment.
> 
> Unfortunately this cannot be done automatically in v4l2_subdev_call(),
> at least not easily.
> 
>>
>> I believe it should be doable with coccinelle. Maybe add a new macro,
>> v4l2_subdev_call_state() or such, which gives the active state in the second
>> parameter (looks like all the ops have the state as the second param). Then
>> use coccinelle to find all the v4l2_subdev_call uses which call ops that get
>> a state, verify that the current caller uses NULL as the state, and change
>> v4l2_subdev_call to v4l2_subdev_call_state.
> 
> 
> Even if we beautify it, I think bridge drivers passing as parameter to
> a subdev operation a subdev attribute, like in the above shown
> 
>          v4l2_subdev_call(sd, pad, set_fmt, sd->state, ...)
> 
> is unecessary and a possible source of confusion, with the subdev
> driver having to infer where the state comes from and the possibility

Why do the drivers need to infer where the state comes from? Except for 
the init_cfg case, but that can be fixed other ways

> of it being NULL anyway if the bridge operates with a non-state aware
> subdev which has not allocated a state (which is harmelss now, as they
> won't be interested in 'state').

Yes, it can be NULL, but it can be NULL already now, and as you say, 
it's harmless.

> It could be made easier if we clearly say drivers "if it's TRY, expect
> a state, if is ACTIVE use your own one (if you want to)". This seems
> impossible to get wrong to me for subdev drivers.

We can write such a clear statement for this new approach also.

>>> This ofc requires the state to be embedded (ie it's always there) and
>>> that state-aware drivers to have properly initialized it, but that's a
>>> given.
>>
>> Why does the state need to be embedded? If the subdev driver is not
>> state-aware, it does not expect to get a state except for the TRY case.
>> Passing NULL for those drivers should be fine.
>>
> 
> It doesn't -need- to be, I just think it avoids allocation and
> releasing at run-time and offers a place where to store subdev-wide
> configurations to all drivers as an opt-in feature.

They do have that option already, they just need to manually allocate 
the state. If we embed the state, the subdev drivers need to manually 
initialize the state. It doesn't really change much, except now we have 
a clear indication (sd->state != NULL) that the driver is state aware. 
And also, 99% of the drivers don't need the state, which might have some 
memory use impact.

And the reason for the subdev drivers having to manually allocate/init 
the state is that there's no place in core to do that. Maybe the various 
v4l2_*_register_subdev might do it, but it wasn't clear to me if it 
would work in practice or not.

So at the moment you have to call the v4l2_subdev_alloc_state() after 
media_entity_pads_init() but before registering the subdev (or possibly 
before registering an async notifier).

> Of course we pay a little price in the size of the subdev, but it's
> all in-kernel stuff and going forward the state could very wel just
> become the standard 'subdev_config'
> 
>          struct v4l2_subdev {
>                  ....
> 
>                  struct v4l2_subdev_config {
>                          struct v4l2_subdev_routing routes;
>                          struct v4l2_subdev_streams streams;
>                  } config;
>          };
> 
> But yeah, allocated or embedded is tangential and I defer this call to
> maintainers which know better than me for sure.

With the wrapper functions, subdev drivers never touch the sd->state 
directly, and thus changing it from allocated to embedded in the future 
should be trivial.

>>> Nonetheless, this considerations do not defeat the purpose of having a
>>> 'state', as currently we have
>>>
>>> struct v4l2_subdev_state {
>>>           struct v4l2_subdev_krouting; /* Use for TRY and ACTIVE */
>>>           struct v4l2_stream_configs; /* Use for ACTIVE */
>>
>> stream_configs is used for TRY also.
>>
>>>           struct v4l2_pad_configs; /* Used for TRY */
>>
>> Probably no point in this, but this _could_ also be used for ACTIVE. We
>> could have state aware drivers that don't use routing or streams, and use
>> just a plain old pad_configs array. This would allow moving the ACTIVE
>> pad_configs from the driver to the core.
> 
> That would be nice, but it would be better is stream_configs could be
> used for pad-only drivers (it's just about assuming stream = 0 for all
> of them, right ?). But yes, my point is about trying to centralize the
> subdev configuration in one place. But that's probably for later
> indeed.

The stream configs require routing to be set first, as routing defines 
the number of stream configs. There are probably ways to hide the 
routing part for simple drivers that don't really need routing but would 
still want to use stream configs.

  Tomi

Re: [PATCH v8 02/36] media: subdev: add active state to struct v4l2_subdev

[Tomi Valkeinen]

On 16/09/2021 11:08, Jacopo Mondi wrote:
> Hi Tomi,
> 
> On Thu, Sep 16, 2021 at 09:52:42AM +0300, Tomi Valkeinen wrote:
>> On 16/09/2021 09:17, Tomi Valkeinen wrote:
>>> Hi,
>>>
>>> On 15/09/2021 12:44, Jacopo Mondi wrote:
>>>> Hi Tomi,
>>>>
>>>> On Mon, Aug 30, 2021 at 02:00:42PM +0300, Tomi Valkeinen wrote:
>>>>> Add a new 'state' field to struct v4l2_subdev to which we can store the
>>>>> active state of a subdev. This will place the subdev configuration into
>>>>> a known place, allowing us to use the state directly from the v4l2
>>>>> framework, thus simplifying the drivers.
>>>>>
>>>>> We also add v4l2_subdev_alloc_state() and v4l2_subdev_free_state(),
>>>>> which need to be used by the drivers that support subdev state in struct
>>>>> v4l2_subdev.
>>>>>
>>>>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>>>>> ---
>>>>>    drivers/media/v4l2-core/v4l2-subdev.c | 21 ++++++++++++++++
>>>>>    include/media/v4l2-subdev.h           | 36 +++++++++++++++++++++++++++
>>>>>    2 files changed, 57 insertions(+)
>>>>>
>>>>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c
>>>>> b/drivers/media/v4l2-core/v4l2-subdev.c
>>>>> index 26a34a8e3d37..e1a794f69815 100644
>>>>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>>>>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>>>>> @@ -943,3 +943,24 @@ void v4l2_subdev_notify_event(struct
>>>>> v4l2_subdev *sd,
>>>>>        v4l2_subdev_notify(sd, V4L2_DEVICE_NOTIFY_EVENT, (void *)ev);
>>>>>    }
>>>>>    EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
>>>>> +
>>>>> +int v4l2_subdev_alloc_state(struct v4l2_subdev *sd)
>>>>> +{
>>>>> +    struct v4l2_subdev_state *state;
>>>>> +
>>>>> +    state = v4l2_alloc_subdev_state(sd);
>>
>> Replying to this again, as the second email didn't actually cover all the
>> topics...
>>
>>>> So, I think this is one source of confusion about init_cfg.
>>>>
>>>> v4l2_alloc_subdev_state() calls init_cfg() and 'state-aware' driver
>>>> are now supposed to allocate their state by calling
>>>> v4l2_subdev_alloc_state(), in the same way as the core does for the
>>>> file-handle ones.
>>>>
>>>> This will lead to init_cfg to be called for the 'active' (ie owned by
>>>> the subdev) state, and then you need to add context to the state (by
>>>> adding a 'which' field) to know what state you're dealing with.
>>>>
>>>> According to the init_cfg() documentation
>>>>
>>>>    * @init_cfg: initialize the pad config to default values
>>>>
>>>> the op has to be called in order to initialize the per-file-handle
>>>> context, not the active one.
>>>
>>> I have missed updating the documentation there =).
>>
>> The documentation above doesn't imply per-file-handle context or TRY case,
>> afaics. It just says "initialize state to default". Unless "pad config"
>> always means TRY, which I think it doesn't as the drivers have internally
>> been using pad configs.
> 
> If they do, they would have the 'active' pad_configs  allocated or
> embedded somewhere in their driver structures, they would not receive
> it as parameter. Or have I missed where the core is capable of fishing
> the 'right' pad_configs ? I think the same should happen for state.

Yes, that is correct. I was just saying that the documentation doesn't 
say that init_cfg is only used for the TRY case, even if that has been 
the case in practice.

>> But it's true that so far init_cfg has only been called for TRY case, and
>> perhaps that's enough of a reason to keep it so.
>>
>>>> I would rather just embed 'struct v4l2_subdev_state' in 'struct
>>>> v4l2_subdev', have the core going through the
>>>
>>> Why would embedding the state change anything?
>>>
>>>> 'v4l2_subdev_alloc_state()' to initialize the per-fh state, but have
>>>> drivers initialize their own state at probe time. If they need for
>>>> some reason to access their 'active' state at init_cfg() time, they
>>>> caan fish it from their subdev.
>>>>
>>>> If I'm not mistaken this will remove the need to have a which filed in
>>>> the state, as I think the 'context' should be inferred from the
>>>> 'which' argument embedded in the ops-specific structures, and not held
>>>> in the state itself.
>>
>> It's true that the state's which field is mainly (probably only) needed for
>> handling the init_cfg. It could be solved in other ways too:
>>
>> - New subdev op to initialize active state
>> - New subdev op which gets 'which' as a parameter, to initialize both states
>> (state-aware drivers wouldn't need to implement the old init_cfg)
>> - Coccinelle to change init_cfg to get the which as a parameter
>>
>> Without doing any deep thinking, the middle one sounds best to me.
> 
> Isn't it simpler if you just don't call init_cfg for the 'active'
> state ? Driver will initialize them at probe time and that's it, then
> you can remove 'which' from the state (and from routing tables too if I'm
> not mistaken).

Routing needs the 'which' similarly to other config structs, like 
v4l2_subdev_format.

I did a quick implementation for the second case, which allows me to 
remove 'which' from the state.

As for "simpler"... Both ways have pros and cons. I'm my mind this new 
way is simpler.

Afaics, in the end (i.e. after doing the v4l2_subdev_call change), the 
only bigger difference is how the state-aware drivers implement their 
ops. With my approach, they get the correct state and that's it. With 
your approach, they need to use a helper function (or do it manually) to 
get the state based on 'which'.

  Tomi

Re: [PATCH v8 03/36] media: subdev: add 'which' to subdev state

[Jacopo Mondi]

Hi Tomi,

On Mon, Sep 13, 2021 at 05:26:45PM +0300, Tomi Valkeinen wrote:
> On 13/09/2021 16:38, Jacopo Mondi wrote:
> > Hi Tomi,
> >
> > On Mon, Sep 13, 2021 at 03:17:01PM +0300, Tomi Valkeinen wrote:
> > > On 13/09/2021 14:41, Jacopo Mondi wrote:
> > > > Hi Tomi,
> > > >
> > > > On Mon, Aug 30, 2021 at 02:00:43PM +0300, Tomi Valkeinen wrote:
> > > > > The subdev state is passed to functions in the media drivers, and
> > > > > usually either V4L2_SUBDEV_FORMAT_ACTIVE or V4L2_SUBDEV_FORMAT_TRY is
> > > > > also given to the function in one way or another.
> > > > >
> > > > > One op where this is not the case is v4l2_subdev_pad_ops.init_cfg. One
> > > > > could argue that the initialization of the state should be the same for
> > > > > both ACTIVE and TRY cases, but unfortunately that is not the case:
> > > > >
> > > > > - Some drivers do also other things than just touch the state when
> > > > > dealing with ACTIVE, e.g. if there is extra state outside the standard
> > > > > subdev state.
> > > > > - Some drivers might need to create, say, struct v4l2_subdev_format
> > > > > which has 'which' field, and that needs to be filled with either ACTIVE
> > > > > or TRY.
> > > > >
> > > > > Currently init_cfg is only called for TRY case from the v4l2 framework,
> > > > > passing the TRY state. Some drivers call their own init_cfg, passing
> > > > > NULL as the state, which is used to indicate ACTIVE case.
> > > > >
> > > > > In the future we want to pass subdev's active state from the v4l2
> > > > > framework side, so we need a solution to this.
> > > > >
> > > > > We could change the init_cfg() to include the TRY/ACTIVE value, which
> > > > > would require changing more or less all the drivers. Instead, I have
> > > > > added 'which' field to the subdev state itself, filled at state
> > > > > allocation time, which only requires changes to the drivers that
> > > > > allocate a state themselves.
> > > > >
> > > > > Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > > > > ---
> > > > >    drivers/media/platform/rcar-vin/rcar-v4l2.c |  2 +-
> > > > >    drivers/media/platform/vsp1/vsp1_entity.c   |  2 +-
> > > > >    drivers/media/v4l2-core/v4l2-subdev.c       | 10 +++++++---
> > > > >    drivers/staging/media/tegra-video/vi.c      |  2 +-
> > > > >    include/media/v4l2-subdev.h                 |  7 ++++++-
> > > > >    5 files changed, 16 insertions(+), 7 deletions(-)
> > > > >
> > > > > diff --git a/drivers/media/platform/rcar-vin/rcar-v4l2.c b/drivers/media/platform/rcar-vin/rcar-v4l2.c
> > > > > index 5f4fa8c48f68..1de30d5b437f 100644
> > > > > --- a/drivers/media/platform/rcar-vin/rcar-v4l2.c
> > > > > +++ b/drivers/media/platform/rcar-vin/rcar-v4l2.c
> > > > > @@ -252,7 +252,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
> > > > >    	u32 width, height;
> > > > >    	int ret;
> > > > >
> > > > > -	sd_state = v4l2_alloc_subdev_state(sd);
> > > > > +	sd_state = v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_ACTIVE);
> > > >
> > > > Shouldn't the 'which' parameters be used to decide if either ACTIVE or
> > > > TRY have to be used ? this function is also used to set TRY formats,
> > > > in example...
> > > >
> > > > Oh, maybe I got how it works, the state's which is not
> > > > relevant but the v4l2_subdev_format's which is, as it will be used in
> > > > the next patch to decide if the subdev's state of the file-handle's
> > > > state should be passed to the ioctl.
> > >
> > > Yes. It's messy, but it's how it worked before also.
> > >
> > > The drivers can't really allocate TRY state as it must come from the core,
> > > based on the filehandle. Now as I say that, makes me wonder why even expose
> > > the option to drivers. Maybe v4l2_alloc_subdev_state() should take just the
> > > sd parameter, and always allocate ACTIVE state, and the v4l2 core can use
> > > another way to create the TRY state.
> > >
> >
> > init_cfg() as well as other operations used to received an
> > array of per fh's pad_configs, and the sd pointer itself. The fh's pad
> > configs were allocated by the core, as well as the core now allocates
> > the per-fh's state.
> >
> > Before the introduction of 'state', if the 'which' flags was set to
> > TRY then information were stored/retrieved/initialized in the per-fh
> > pad_config, otherwise the active configuration (usually stored in the
> > driver main structure) was used.
> >
> > So we had a clear separation of per-fh information and the active
> > state. The core didn't care afaict, and passed in both, then driver had
> > to deal with them doing the right thing by inspecting the 'which' flag.
> >
> > The typical pattern was:
> >
> >          static int subdev_ops(sd, pad_cfg, which)
> >          {
> >                  if (which == TRY)
> >                          /* Operate on config stored in pad_cfg */
> >
> >                  else
> >                          /*
> >                           * Operate on config stored in subdev (and
> >                           * applied to HW)
> >                           */
> >          }
> >
> > Or am I overlooking some cases or do you agree with my understanding
> > so far ?
>
> More or less, yes. I think there are (used to be) three kinds of ops:
>
> - Ops that get pad_cfg and 'which' in an op specific struct. E.g. set_fmt.
> The pad_cfg is TRY pad_config, even if 'which' == ACTIVE.
>
> - Ops that don't get pad_cfg, like s_stream. 'which' is implicitly ACTIVE.

Also note that operations like s_stream do not take a state as
parameter. The driver has to fetch it from the subdev anyway
(this in reply to the idea of having the active state as parameter vs
retrieving it from the subdev if ACTIVE)

While porting the R-Car drivers on top of this series I found myself
in the need to (in the s_stream call chain)

static int rcsi2_start_receiver(struct rcar_csi2 *priv)
{
	const struct v4l2_subdev_state *state = priv->subdev.state;
	const struct v4l2_subdev_stream_configs *configs = &state->stream_configs;

        ...

	/*
	 * Configure field handling inspecting the formats of the
	 * single sink pad streams.
	 */
	for (i = 0; i < configs->num_configs; ++i) {
		const struct v4l2_subdev_stream_config *config = configs->configs;
		if (config->pad != RCAR_CSI2_SINK)
			continue;

		if (config->fmt.field != V4L2_FIELD_ALTERNATE)
			continue;

		fld |= FLD_DET_SEL(1);
		fld |= FLD_FLD_EN(config->stream);

		/* PAL vs NTSC. */
		if (config->fmt.height == 240)
			fld |= FLD_FLD_NUM(0);
		else
			fld |= FLD_FLD_NUM(1);
	}

        ...

}

Am I doing it wrong, or is this a case for the subdev to have to
directly access sd->state ?

>
> - init_cfg which gets pad_cfg, but no which (as 'which' is always implicitly
> TRY)
>
> So pad_cfg was TRY state. Drivers could use pad_configs internally to track
> ACTIVE state, but the core had no knowledge about this.
>
> > Now we have a 'state' that holds the array of pad_configs and along
> > the continuation of the series will end up holding per-pad
> > configurations.
> >
> > We now also have one 'state' per file-handle, and one
> > per-subdev. As I see this, it would be natual for drivers to receive
> > one state without knowing where it comes from. In the next patch you
>
> Note that only subdev's that explicitly support the new state code, and
> allocate the state, have the subdev active state. Which means only the
> drivers in my work branch.
>
> The "old" drivers work like they used to: they get the state (essentially
> repackaged pad_cfg) for TRY cases, NULL otherwise.
>
> And yes, it would be natural to just get a state, but the subdev drivers
> need to know if the context is TRY/ACTIVE. As you can see from the bullet
> list above, the driver knows this in all the other places except init_cfg.
>
> > instrument the core to do exactly this: inspect the which flag and
> > pass in the 'right' state. Ofc drivers need to have access to 'which'
> > to know if they have to apply settings to the HW or not.
> >
> > Looking ahead in your series I see these structures:
> >
> >          struct v4l2_subdev_pad_config {
> >                  struct v4l2_mbus_framefmt try_fmt;
> >                  struct v4l2_rect try_crop;
> >                  struct v4l2_rect try_compose;
> >          };
> >
> >          struct v4l2_subdev_stream_config {
> >                  u32 pad;
> >                  u32 stream;
> >
> >                  struct v4l2_mbus_framefmt fmt;
> >                  struct v4l2_rect crop;
> >                  struct v4l2_rect compose;
> >          };
> >
> >          struct v4l2_subdev_stream_configs {
> >                  u32 num_configs;
> >                  struct v4l2_subdev_stream_config *configs;
> >          };
> >
> > All of them part of state:
> >
> > struct v4l2_subdev_state {
> > 	struct mutex lock;
> > 	u32 which;
> > 	struct v4l2_subdev_pad_config *pads;
> > 	struct v4l2_subdev_krouting routing;
> > 	struct v4l2_subdev_stream_configs stream_configs;
> > };
> >
> > So 'state' will hold 'TRY' information (only used for 'state'
> > instances allocated in the fh) and 'ACTIVE' ones (used for states
> > allocated in the sd).
>
> Right.
>
> > Looking at 'v4l2_subdev_pad_config' and 'v4l2_subdev_stream_config' they
> > seem to describe more or less the same things: fmt, crop and compose
> > (per pad-stream in case of stream_config). I wonder if those shouldn't
> > be unified so that:
> >
> > 1) Drivers receive one state: the core passes in the 'correct' one
> > (per-fh or per-sd) as you do in next patch
>
> Yes. But note that "old" drivers don't have active state.
>
> > 2) The 'which' information is not stored in the state but it's only
> > 'contextual' (as in a parameter to the subdev operation) so that
> > drivers inspect it to know if they have to apply settings to hw or not
>
> Yes, except we have init_cfg...
>
> > 3) v4l2_subdev_pad_config can be re-used and expanded, to maintain per-pad
> > configurations regardless if they're ACTIVE or TRY, as this only depends
> > on where the state is stored.
>
> pad_config is a static array of per-pad configs. stream_configs is a dynamic
> per-stream config. stream_configs is a super-set of pad-configs, so we could
> drop pad_configs, but it would require changing all the drivers in
> non-trivial ways.
>
> v4l2_subdev_pad_config is not used or even allocated by the "new" drivers.
> And routing & stream_configs are not used by the "old" drivers.
>
> > As I immagine it a subdev pad operation could look like:
> >
> >          static int subdev_op(sd, pad, state, which, ...)
> >          {
> >                  /* Doesn't matter if state is per-fh or the sd one. */
> >                  state->pads[pad].fmt = ....;
> >
> >                  if (which == TRY)
> >                          return;
> >
> >                  /* Apply to the HW. */
> >          }
> >
> > Does it make any sense to you ? I might have missed some reason why
> > this is not possible.
>
> It makes sense, but there are the buts =). I've tried to explain these in
> the commit messages, but it's kind of confusing.
>
> One but I haven't mentioned in the emails is that when subdev drivers call
> ops in other subdev drivers they pass NULL in the state. This is fine for
> the "old" drivers, as they expect a state only for TRY case. However, the
> "new" drivers unfortunately expect to get a state on both TRY and ACTIVE
> cases, and the only sensible way I figured out to handle this was the
> v4l2_subdev_validate_state() function (patch 6).
>
> So, all this could be much neater, but would require modifying all subdev
> drivers in non-trivial ways. I think this is something that can be done
> slowly in the future.
>
> > > > >    	if (IS_ERR(sd_state))
> > > > >    		return PTR_ERR(sd_state);
> > > > >
> > > > > diff --git a/drivers/media/platform/vsp1/vsp1_entity.c b/drivers/media/platform/vsp1/vsp1_entity.c
> > > > > index e40bca254b8b..63ea5e472c33 100644
> > > > > --- a/drivers/media/platform/vsp1/vsp1_entity.c
> > > > > +++ b/drivers/media/platform/vsp1/vsp1_entity.c
> > > > > @@ -675,7 +675,7 @@ int vsp1_entity_init(struct vsp1_device *vsp1, struct vsp1_entity *entity,
> > > > >    	 * Allocate the pad configuration to store formats and selection
> > > > >    	 * rectangles.
> > > > >    	 */
> > > > > -	entity->config = v4l2_alloc_subdev_state(&entity->subdev);
> > > > > +	entity->config = v4l2_alloc_subdev_state(&entity->subdev, V4L2_SUBDEV_FORMAT_ACTIVE);
> > > > >    	if (IS_ERR(entity->config)) {
> > > > >    		media_entity_cleanup(&entity->subdev.entity);
> > > > >    		return PTR_ERR(entity->config);
> > > > > diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> > > > > index e1a794f69815..04ad319fb150 100644
> > > > > --- a/drivers/media/v4l2-core/v4l2-subdev.c
> > > > > +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> > > > > @@ -28,7 +28,7 @@ static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
> > > > >    {
> > > > >    	struct v4l2_subdev_state *state;
> > > > >
> > > > > -	state = v4l2_alloc_subdev_state(sd);
> > > > > +	state = v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_TRY);
> > > >
> > > > At the same time I'm not sure I get the purpose of this. Don't
> > > > init_cfg() callback implementations deal with try formats themeselves
> > > > ? I mean, it's not a fixed rule, they can as well initialize their
> > > > default 'active' formats, but what matters is that they initialize
> > > > their per-fh try states ?
> > >
> > > That is what they do currently. init_cfg() only deals with TRY state, as
> > > that's the only "state" (i.e. pad_config) there used to be from v4l2 core's
> > > perspective.
> > >
> > > > Shouldn't init_cfg receive the fh's state so that it can initialize
> > > > it, and just in case they need to, access their subdev's state and
> > > > initialize them ? I'm missing what the purpose of the flag is tbh.
> > >
> > > Now we have (a possibility to have) state for both TRY and ACTIVE on the
> > > v4l2 core side. The active state has to be initialized also, and a logical
> > > way to do that is to use the init_cfg().
> >
> > The 'ACTIVE' state is stored in the subdev, to which init_cfg() has
> > access, and it receives the 'TRY' state as a parameter.
>
> No, init_cfg gets either ACTIVE or TRY state, whichever is being allocated.
> For "old" drivers, ACTIVE state is never allocated so they don't get
> init_cfg calls for ACTIVE at all.
>
> Aaand while writing that, I realized that some drivers manually do allocate
> ACTIVE state temporarily, which would cause init_cfg with ACTIVE state to be
> called. I wonder if they explode... Need to check.
>
> > It is possible to access both states and initialize them properly if
> > I'm not mistaken.
> >
> > >
> > > So now, for drivers that support the new active state, init_cfg() can get
> > > either TRY or ACTIVE state. And if you want to call, say, the driver's
> > > set_routing() to setup the routing in the state, you have to set the 'which'
> > > in the routing struct to a value. So somehow init_cfg needs to know if it's
> > > initializing an ACTIVE or TRY state.
> >
> > I'm not sure I got this part. set_routing() as other ops will receive
> > a state and 'which'. If my proposal above makes sensem where the state
>
> Yes, but if it's init_cfg calling set_routing, init_cfg has to figure out
> the 'which' from somewhere.
>
> E.g. init_cfg from ub913 driver:
>
> static int ub913_init_cfg(struct v4l2_subdev *sd,
> 			  struct v4l2_subdev_state *state)
> {
> 	u32 which = state->which;
>
> 	struct v4l2_subdev_route routes[] = {
> 		{
> 			.sink_pad = 0,
> 			.sink_stream = 0,
> 			.source_pad = 1,
> 			.source_stream = 0,
> 			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> 		},
> 	};
>
> 	struct v4l2_subdev_krouting routing = {
> 		.which = which,
> 		.num_routes = ARRAY_SIZE(routes),
> 		.routes = routes,
> 	};
>
> 	return ub913_set_routing(sd, state, &routing);
> }
>
> It uses set_routing to setup a default routing (and set_routing in turn also
> initializes the formats), but set_routing needs 'which'.
>
>  Tomi

Re: [PATCH v8 03/36] media: subdev: add 'which' to subdev state

[Tomi Valkeinen]

Hi,

On 16/09/2021 16:07, Jacopo Mondi wrote:

> Also note that operations like s_stream do not take a state as
> parameter. The driver has to fetch it from the subdev anyway
> (this in reply to the idea of having the active state as parameter vs
> retrieving it from the subdev if ACTIVE)
> 
> While porting the R-Car drivers on top of this series I found myself
> in the need to (in the s_stream call chain)
> 
> static int rcsi2_start_receiver(struct rcar_csi2 *priv)
> {
> 	const struct v4l2_subdev_state *state = priv->subdev.state;
> 	const struct v4l2_subdev_stream_configs *configs = &state->stream_configs;
> 
>          ...
> 
> 	/*
> 	 * Configure field handling inspecting the formats of the
> 	 * single sink pad streams.
> 	 */
> 	for (i = 0; i < configs->num_configs; ++i) {
> 		const struct v4l2_subdev_stream_config *config = configs->configs;
> 		if (config->pad != RCAR_CSI2_SINK)
> 			continue;
> 
> 		if (config->fmt.field != V4L2_FIELD_ALTERNATE)
> 			continue;
> 
> 		fld |= FLD_DET_SEL(1);
> 		fld |= FLD_FLD_EN(config->stream);
> 
> 		/* PAL vs NTSC. */
> 		if (config->fmt.height == 240)
> 			fld |= FLD_FLD_NUM(0);
> 		else
> 			fld |= FLD_FLD_NUM(1);
> 	}
> 
>          ...
> 
> }
> 
> Am I doing it wrong, or is this a case for the subdev to have to
> directly access sd->state ?

In s_stream path you should:

	state = v4l2_subdev_lock_active_state(sd);

	<do the work with the state>

	v4l2_subdev_unlock_state(state);

If you already have the state, e.g. in set_fmt:

	state = v4l2_subdev_validate_and_lock_state(sd, state);

	<do the work with the state>

	v4l2_subdev_unlock_state(state);

Accessing the stream_configs directly is fine but not that nice. I did 
think about some helpers, perhaps for_each_stream_config(), but I didn't 
add that as I didn't have the need.

There's v4l2_state_get_stream_format() which can be used in many cases, 
but we probably need something else if you need to iterate over all the 
configs.

  Tomi

Re: [PATCH v8 00/36] v4l: subdev internal routing and streams

[Tomi Valkeinen]

Hi all,

On 30/08/2021 14:00, Tomi Valkeinen wrote:
> Hi,
> 
> This is v8 of the multiplexed streams series. v7 can be found from:
> 
> https://lore.kernel.org/linux-media/20210524104408.599645-1-tomi.valkeinen@ideasonboard.com/
> 
> The main change in this version is the implementation and use of
> centralized active state for subdevs.
> 
> I have pushed my work branch to:
> 
> git://git.kernel.org/pub/scm/linux/kernel/git/tomba/linux.git multistream/work-v8
> 
> which contains the patches in this series, along with subdev drivers
> using multiplexed streams.
> 
> Both this series and the branch above are based on top of today's
> git://linuxtv.org/media_tree.git master.
> 
> The documentation still needs improving, but I hope the docs in this
> series, and the drivers in the work branch, are enough to give the
> reviewers enough information to do a review.
> 
> As can be guessed from the work branch, I have been testing this series
> with TI's FPDLink setup. I have also done a "backwards compatibility"
> test by dropping all multiplexed streams patches from the CAL driver
> (the CSI-2 RX on the TI SoC), and using the FPDLink drivers with
> single-stream configuration.

We've had good discussions with Jacopo about this series.

I chose the approaches in this series based on what I think the API 
should be, even if the API has behaved differently before. And I think 
I'm also leaning forward a bit, in the sense that the full benefit of 
the API can only be had after more changes to the core and subdev 
drivers (changes which may or may not happen).

If I understood Jacopo correctly, his comments were essentially that my 
approach is different than the current one, and as the current drivers 
anyway do things the old way, this is very confusing. Basically I create 
two different kinds of subdev drivers: the old and new ones, which 
manage state differently.

I want to summarize two particular topics:

1) Active state & subdev ops

In upstream we have v4l2_subdev_state which contains only the pad_config 
array. This state is "try" state, it's allocated per file-handle, and 
passed to the subdev drivers when executing subdev ioctls in try-mode 
(which == V4L2_SUBDEV_FORMAT_TRY). This try-state is sometimes also 
passed to the subdev drivers when executing in active-mode 
(V4L2_SUBDEV_FORMAT_ACTIVE), but the drivers are supposed to ignore it.

There is also an active-state, but it's driver-specific and 
driver-internal. The drivers check the 'which' value, and either use the 
passed try-state, or the internal state.

What I did in this series aims to have both try- and active-states in 
v4l2 core, and passing the correct state to subdevs so that they don't 
(necessarily) need any internal state. There are some issues with it, 
which have been discussed, but I believe those issues can be fixed.

The subdev drivers need to be written to use this new active-state, so 
it doesn't affect the current drivers.

The question is, do we want to go that way? We could as well keep the 
current behavior of subdev drivers only getting the try-state as a 
parameter, and the drivers digging out the active state manually. This 
active state could either be internal to the driver, or it could be in 
the base struct v4l2_subdev (see also topic 2).

2) Shared subdev active-state

The try-state is specific to a file-handle, and afaics have no real 
race-issues as it's not really shared. Although I guess in theory an 
application could call subdev ioctls from multiple threads using the 
same fd.

In upstream the subdev drivers' internal state is managed fully by the 
subdev drivers. The drivers are expected to handle necessary locking in 
their subdev ops and interrupt handlers. If, say, v4l2 core needs to get 
a format from the subdev, it calls a subdev op to get it.

In my series I aimed to a shared active-state. The state is located in a 
known place, struct v4l2_subdev, and can be accessed without the subdev 
driver's help. This requires locking, which I have implemented.

At the moment the only real benefit with this is reading the routing 
table while doing pipeline validation: Instead of having to dynamically 
allocate memory and call the subdev op to create a copy of the routing 
table (for each subdev, possibly multiple times), the validator can just 
lock the state, and use it. And, in fact, there is no get_routing subdev 
op at all.

But this means that the subdev drivers that support this new 
active-state have to handle locking for the active state, and the 
"mindset" is different than previously.

So the question is, do we want to go that way? We could as well mandate 
that the active-state can only be accessed via subdev's ops (and add the 
get-routing, of course), and the subdev manages the locking internally.

  Tomi

Re: [PATCH v8 01/36] media: subdev: rename subdev-state alloc & free

[Laurent Pinchart]

Hi Tomi,

Thank you for the patch.

On Mon, Aug 30, 2021 at 02:00:41PM +0300, Tomi Valkeinen wrote:
> The recently added v4l2_subdev_alloc_state() and
> v4l2_subdev_free_state() were somewhat badly named. They allocate/free
> the state that can be used for a subdev, but they don't change the
> subdev itself in any way.
> 
> In the future we want to have the subdev state available easily for all
> subdevs, and we want to store the state to subdev struct. Thus we will
> be needing a new function which allocates the state and stores it into
> the subdev struct.
> 
> This patch renames v4l2_subdev_alloc/free_state functions to
> v4l2_alloc/free_subdev_state, so that we can later use
> v4l2_subdev_alloc/free_state for the versions which work on the subdev
> struct.

With have video_device_alloc() and media_request_alloc(), should we use
v4l2_subdev_state_alloc() and v4l2_subdev_state_free() to be consistent
?

With or without this (but preferably with ;-)),

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Note that regardless of the name, if we have both
v4l2_subdev_alloc_state() (to allocate the state and store it in the
subdev structure) and v4l2_subdev_alloc_state() (to allocate the state),
it could be confusing for driver authors. Let's discuss this further in
the patch series when the problem arises (if it does).

> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/platform/rcar-vin/rcar-v4l2.c |  4 ++--
>  drivers/media/platform/vsp1/vsp1_entity.c   |  4 ++--
>  drivers/media/v4l2-core/v4l2-subdev.c       | 12 ++++++------
>  drivers/staging/media/tegra-video/vi.c      |  4 ++--
>  include/media/v4l2-subdev.h                 | 10 +++++-----
>  5 files changed, 17 insertions(+), 17 deletions(-)
> 
> diff --git a/drivers/media/platform/rcar-vin/rcar-v4l2.c b/drivers/media/platform/rcar-vin/rcar-v4l2.c
> index 0d141155f0e3..5f4fa8c48f68 100644
> --- a/drivers/media/platform/rcar-vin/rcar-v4l2.c
> +++ b/drivers/media/platform/rcar-vin/rcar-v4l2.c
> @@ -252,7 +252,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
>  	u32 width, height;
>  	int ret;
>  
> -	sd_state = v4l2_subdev_alloc_state(sd);
> +	sd_state = v4l2_alloc_subdev_state(sd);
>  	if (IS_ERR(sd_state))
>  		return PTR_ERR(sd_state);
>  
> @@ -288,7 +288,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
>  
>  	rvin_format_align(vin, pix);
>  done:
> -	v4l2_subdev_free_state(sd_state);
> +	v4l2_free_subdev_state(sd_state);
>  
>  	return ret;
>  }
> diff --git a/drivers/media/platform/vsp1/vsp1_entity.c b/drivers/media/platform/vsp1/vsp1_entity.c
> index 823c15facd1b..e40bca254b8b 100644
> --- a/drivers/media/platform/vsp1/vsp1_entity.c
> +++ b/drivers/media/platform/vsp1/vsp1_entity.c
> @@ -675,7 +675,7 @@ int vsp1_entity_init(struct vsp1_device *vsp1, struct vsp1_entity *entity,
>  	 * Allocate the pad configuration to store formats and selection
>  	 * rectangles.
>  	 */
> -	entity->config = v4l2_subdev_alloc_state(&entity->subdev);
> +	entity->config = v4l2_alloc_subdev_state(&entity->subdev);
>  	if (IS_ERR(entity->config)) {
>  		media_entity_cleanup(&entity->subdev.entity);
>  		return PTR_ERR(entity->config);
> @@ -690,6 +690,6 @@ void vsp1_entity_destroy(struct vsp1_entity *entity)
>  		entity->ops->destroy(entity);
>  	if (entity->subdev.ctrl_handler)
>  		v4l2_ctrl_handler_free(entity->subdev.ctrl_handler);
> -	v4l2_subdev_free_state(entity->config);
> +	v4l2_free_subdev_state(entity->config);
>  	media_entity_cleanup(&entity->subdev.entity);
>  }
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 5d27a27cc2f2..26a34a8e3d37 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -28,7 +28,7 @@ static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
>  {
>  	struct v4l2_subdev_state *state;
>  
> -	state = v4l2_subdev_alloc_state(sd);
> +	state = v4l2_alloc_subdev_state(sd);
>  	if (IS_ERR(state))
>  		return PTR_ERR(state);
>  
> @@ -39,7 +39,7 @@ static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
>  
>  static void subdev_fh_free(struct v4l2_subdev_fh *fh)
>  {
> -	v4l2_subdev_free_state(fh->state);
> +	v4l2_free_subdev_state(fh->state);
>  	fh->state = NULL;
>  }
>  
> @@ -870,7 +870,7 @@ int v4l2_subdev_link_validate(struct media_link *link)
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
>  
> -struct v4l2_subdev_state *v4l2_subdev_alloc_state(struct v4l2_subdev *sd)
> +struct v4l2_subdev_state *v4l2_alloc_subdev_state(struct v4l2_subdev *sd)
>  {
>  	struct v4l2_subdev_state *state;
>  	int ret;
> @@ -903,9 +903,9 @@ struct v4l2_subdev_state *v4l2_subdev_alloc_state(struct v4l2_subdev *sd)
>  
>  	return ERR_PTR(ret);
>  }
> -EXPORT_SYMBOL_GPL(v4l2_subdev_alloc_state);
> +EXPORT_SYMBOL_GPL(v4l2_alloc_subdev_state);
>  
> -void v4l2_subdev_free_state(struct v4l2_subdev_state *state)
> +void v4l2_free_subdev_state(struct v4l2_subdev_state *state)
>  {
>  	if (!state)
>  		return;
> @@ -913,7 +913,7 @@ void v4l2_subdev_free_state(struct v4l2_subdev_state *state)
>  	kvfree(state->pads);
>  	kfree(state);
>  }
> -EXPORT_SYMBOL_GPL(v4l2_subdev_free_state);
> +EXPORT_SYMBOL_GPL(v4l2_free_subdev_state);
>  
>  #endif /* CONFIG_MEDIA_CONTROLLER */
>  
> diff --git a/drivers/staging/media/tegra-video/vi.c b/drivers/staging/media/tegra-video/vi.c
> index d321790b07d9..a94d19e2a67c 100644
> --- a/drivers/staging/media/tegra-video/vi.c
> +++ b/drivers/staging/media/tegra-video/vi.c
> @@ -507,7 +507,7 @@ static int __tegra_channel_try_format(struct tegra_vi_channel *chan,
>  	if (!subdev)
>  		return -ENODEV;
>  
> -	sd_state = v4l2_subdev_alloc_state(subdev);
> +	sd_state = v4l2_alloc_subdev_state(subdev);
>  	if (IS_ERR(sd_state))
>  		return PTR_ERR(sd_state);
>  	/*
> @@ -558,7 +558,7 @@ static int __tegra_channel_try_format(struct tegra_vi_channel *chan,
>  	v4l2_fill_pix_format(pix, &fmt.format);
>  	tegra_channel_fmt_align(chan, pix, fmtinfo->bpp);
>  
> -	v4l2_subdev_free_state(sd_state);
> +	v4l2_free_subdev_state(sd_state);
>  
>  	return 0;
>  }
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 95ec18c2f49c..8701d2e7d893 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -1135,20 +1135,20 @@ int v4l2_subdev_link_validate_default(struct v4l2_subdev *sd,
>  int v4l2_subdev_link_validate(struct media_link *link);
>  
>  /**
> - * v4l2_subdev_alloc_state - allocate v4l2_subdev_state
> + * v4l2_alloc_subdev_state - allocate v4l2_subdev_state
>   *
>   * @sd: pointer to &struct v4l2_subdev for which the state is being allocated.
>   *
> - * Must call v4l2_subdev_free_state() when state is no longer needed.
> + * Must call v4l2_free_subdev_state() when state is no longer needed.
>   */
> -struct v4l2_subdev_state *v4l2_subdev_alloc_state(struct v4l2_subdev *sd);
> +struct v4l2_subdev_state *v4l2_alloc_subdev_state(struct v4l2_subdev *sd);
>  
>  /**
> - * v4l2_subdev_free_state - free a v4l2_subdev_state
> + * v4l2_free_subdev_state - free a v4l2_subdev_state
>   *
>   * @state: v4l2_subdev_state to be freed.
>   */
> -void v4l2_subdev_free_state(struct v4l2_subdev_state *state);
> +void v4l2_free_subdev_state(struct v4l2_subdev_state *state);
>  
>  #endif /* CONFIG_MEDIA_CONTROLLER */
>  

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 02/36] media: subdev: add active state to struct v4l2_subdev

[Laurent Pinchart]

Hello,

On Thu, Sep 16, 2021 at 12:36:33PM +0300, Tomi Valkeinen wrote:
> On 16/09/2021 11:08, Jacopo Mondi wrote:
> > On Thu, Sep 16, 2021 at 09:52:42AM +0300, Tomi Valkeinen wrote:
> >> On 16/09/2021 09:17, Tomi Valkeinen wrote:
> >>> On 15/09/2021 12:44, Jacopo Mondi wrote:
> >>>> On Mon, Aug 30, 2021 at 02:00:42PM +0300, Tomi Valkeinen wrote:
> >>>>> Add a new 'state' field to struct v4l2_subdev to which we can store the
> >>>>> active state of a subdev. This will place the subdev configuration into
> >>>>> a known place, allowing us to use the state directly from the v4l2
> >>>>> framework, thus simplifying the drivers.
> >>>>>
> >>>>> We also add v4l2_subdev_alloc_state() and v4l2_subdev_free_state(),
> >>>>> which need to be used by the drivers that support subdev state in struct
> >>>>> v4l2_subdev.
> >>>>>
> >>>>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> >>>>> ---
> >>>>>    drivers/media/v4l2-core/v4l2-subdev.c | 21 ++++++++++++++++
> >>>>>    include/media/v4l2-subdev.h           | 36 +++++++++++++++++++++++++++
> >>>>>    2 files changed, 57 insertions(+)
> >>>>>
> >>>>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c
> >>>>> b/drivers/media/v4l2-core/v4l2-subdev.c
> >>>>> index 26a34a8e3d37..e1a794f69815 100644
> >>>>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> >>>>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> >>>>> @@ -943,3 +943,24 @@ void v4l2_subdev_notify_event(struct
> >>>>> v4l2_subdev *sd,
> >>>>>        v4l2_subdev_notify(sd, V4L2_DEVICE_NOTIFY_EVENT, (void *)ev);
> >>>>>    }
> >>>>>    EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
> >>>>> +
> >>>>> +int v4l2_subdev_alloc_state(struct v4l2_subdev *sd)
> >>>>> +{
> >>>>> +    struct v4l2_subdev_state *state;
> >>>>> +
> >>>>> +    state = v4l2_alloc_subdev_state(sd);
> >>
> >> Replying to this again, as the second email didn't actually cover all the
> >> topics...
> >>
> >>>> So, I think this is one source of confusion about init_cfg.
> >>>>
> >>>> v4l2_alloc_subdev_state() calls init_cfg() and 'state-aware' driver
> >>>> are now supposed to allocate their state by calling
> >>>> v4l2_subdev_alloc_state(), in the same way as the core does for the
> >>>> file-handle ones.
> >>>>
> >>>> This will lead to init_cfg to be called for the 'active' (ie owned by
> >>>> the subdev) state, and then you need to add context to the state (by
> >>>> adding a 'which' field) to know what state you're dealing with.
> >>>>
> >>>> According to the init_cfg() documentation
> >>>>
> >>>>    * @init_cfg: initialize the pad config to default values
> >>>>
> >>>> the op has to be called in order to initialize the per-file-handle
> >>>> context, not the active one.
> >>>
> >>> I have missed updating the documentation there =).
> >>
> >> The documentation above doesn't imply per-file-handle context or TRY case,
> >> afaics. It just says "initialize state to default". Unless "pad config"
> >> always means TRY, which I think it doesn't as the drivers have internally
> >> been using pad configs.
> > 
> > If they do, they would have the 'active' pad_configs  allocated or
> > embedded somewhere in their driver structures, they would not receive
> > it as parameter. Or have I missed where the core is capable of fishing
> > the 'right' pad_configs ? I think the same should happen for state.
> 
> Yes, that is correct. I was just saying that the documentation doesn't 
> say that init_cfg is only used for the TRY case, even if that has been 
> the case in practice.
> 
> >> But it's true that so far init_cfg has only been called for TRY case, and
> >> perhaps that's enough of a reason to keep it so.
> >>
> >>>> I would rather just embed 'struct v4l2_subdev_state' in 'struct
> >>>> v4l2_subdev', have the core going through the
> >>>
> >>> Why would embedding the state change anything?
> >>>
> >>>> 'v4l2_subdev_alloc_state()' to initialize the per-fh state, but have
> >>>> drivers initialize their own state at probe time. If they need for
> >>>> some reason to access their 'active' state at init_cfg() time, they
> >>>> caan fish it from their subdev.
> >>>>
> >>>> If I'm not mistaken this will remove the need to have a which filed in
> >>>> the state, as I think the 'context' should be inferred from the
> >>>> 'which' argument embedded in the ops-specific structures, and not held
> >>>> in the state itself.
> >>
> >> It's true that the state's which field is mainly (probably only) needed for
> >> handling the init_cfg. It could be solved in other ways too:
> >>
> >> - New subdev op to initialize active state
> >> - New subdev op which gets 'which' as a parameter, to initialize both states
> >> (state-aware drivers wouldn't need to implement the old init_cfg)
> >> - Coccinelle to change init_cfg to get the which as a parameter
> >>
> >> Without doing any deep thinking, the middle one sounds best to me.
> > 
> > Isn't it simpler if you just don't call init_cfg for the 'active'
> > state ? Driver will initialize them at probe time and that's it, then
> > you can remove 'which' from the state (and from routing tables too if I'm
> > not mistaken).
> 
> Routing needs the 'which' similarly to other config structs, like 
> v4l2_subdev_format.
> 
> I did a quick implementation for the second case, which allows me to 
> remove 'which' from the state.
> 
> As for "simpler"... Both ways have pros and cons. I'm my mind this new 
> way is simpler.
> 
> Afaics, in the end (i.e. after doing the v4l2_subdev_call change), the 
> only bigger difference is how the state-aware drivers implement their 
> ops. With my approach, they get the correct state and that's it. With 
> your approach, they need to use a helper function (or do it manually) to 
> get the state based on 'which'.

I'd like to propose a third approach (or at least I believe it's a third
one). I agree with Jacopo that the state structure should not have a
which field, that's a layering violation. The state is a state,
regardless of whether it holds TRY or ACTIVE data. What are the current
blockers that would prevent that ?

However, I think .init_cfg() should be used to initialize *all* states,
both TRY and ACTIVE. That will simplify driver implementation (but
centralizing the initialization in a single place) and ensure that the
default value for the ACTIVE state always matches the default value for
the TRY state, which I think is important.

When it comes to calling v4l2_subdev_alloc_state() in drivers, I also
agree with Jacopo, I'm not extremely fond of it. This should be
automatic, drivers shouldn't have to care about allocating the ACTIVE
state. However, I also agree with Tomi that there's no real place to do
this today. I think we need to restructure subdev initialization, which
currently has very few rules. This could take the form, for instance, of

1. calling v4l2_subdev_init() as an early subdev init function
2. setting fields of v4l2_subdev manually
3. initializing the media_entity embedded in the subdev
4. calling a new "initialization finalization" function that will
   initialize the state embedded in v4l2_subdev (as opposed to
   allocating it dynamically as done today)

That's the simplest option as it really ressembles what we do today, and
would more or less only require embedded the state in v4l2_subdev and
renaming v4l2_subdev_alloc_state() to v4l2_subdev_init_done() (or
similar, that's likely not a great name).

A more intrusive change would restructure the subdev initialization
further, by handling the media_entity initialization transparently
(which would require passing more arguments to v4l2_subdev_init(),
probably creating a new version of that function). That's more complex
and I think it's out of scope for this series, but it may still be
useful to keep the long term goal in mind to try and align with the
direction.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 03/36] media: subdev: add 'which' to subdev state

[Laurent Pinchart]

Hello,

On Thu, Sep 16, 2021 at 03:07:52PM +0200, Jacopo Mondi wrote:
> On Mon, Sep 13, 2021 at 05:26:45PM +0300, Tomi Valkeinen wrote:
> > On 13/09/2021 16:38, Jacopo Mondi wrote:
> > > On Mon, Sep 13, 2021 at 03:17:01PM +0300, Tomi Valkeinen wrote:
> > > > On 13/09/2021 14:41, Jacopo Mondi wrote:
> > > > > On Mon, Aug 30, 2021 at 02:00:43PM +0300, Tomi Valkeinen wrote:
> > > > > > The subdev state is passed to functions in the media drivers, and
> > > > > > usually either V4L2_SUBDEV_FORMAT_ACTIVE or V4L2_SUBDEV_FORMAT_TRY is
> > > > > > also given to the function in one way or another.
> > > > > >
> > > > > > One op where this is not the case is v4l2_subdev_pad_ops.init_cfg. One
> > > > > > could argue that the initialization of the state should be the same for
> > > > > > both ACTIVE and TRY cases, but unfortunately that is not the case:
> > > > > >
> > > > > > - Some drivers do also other things than just touch the state when
> > > > > > dealing with ACTIVE, e.g. if there is extra state outside the standard
> > > > > > subdev state.
> > > > > > - Some drivers might need to create, say, struct v4l2_subdev_format
> > > > > > which has 'which' field, and that needs to be filled with either ACTIVE
> > > > > > or TRY.
> > > > > >
> > > > > > Currently init_cfg is only called for TRY case from the v4l2 framework,
> > > > > > passing the TRY state. Some drivers call their own init_cfg, passing
> > > > > > NULL as the state, which is used to indicate ACTIVE case.
> > > > > >
> > > > > > In the future we want to pass subdev's active state from the v4l2
> > > > > > framework side, so we need a solution to this.
> > > > > >
> > > > > > We could change the init_cfg() to include the TRY/ACTIVE value, which
> > > > > > would require changing more or less all the drivers. Instead, I have
> > > > > > added 'which' field to the subdev state itself, filled at state
> > > > > > allocation time, which only requires changes to the drivers that
> > > > > > allocate a state themselves.
> > > > > >
> > > > > > Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > > > > > ---
> > > > > >    drivers/media/platform/rcar-vin/rcar-v4l2.c |  2 +-
> > > > > >    drivers/media/platform/vsp1/vsp1_entity.c   |  2 +-
> > > > > >    drivers/media/v4l2-core/v4l2-subdev.c       | 10 +++++++---
> > > > > >    drivers/staging/media/tegra-video/vi.c      |  2 +-
> > > > > >    include/media/v4l2-subdev.h                 |  7 ++++++-
> > > > > >    5 files changed, 16 insertions(+), 7 deletions(-)
> > > > > >
> > > > > > diff --git a/drivers/media/platform/rcar-vin/rcar-v4l2.c b/drivers/media/platform/rcar-vin/rcar-v4l2.c
> > > > > > index 5f4fa8c48f68..1de30d5b437f 100644
> > > > > > --- a/drivers/media/platform/rcar-vin/rcar-v4l2.c
> > > > > > +++ b/drivers/media/platform/rcar-vin/rcar-v4l2.c
> > > > > > @@ -252,7 +252,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
> > > > > >    	u32 width, height;
> > > > > >    	int ret;
> > > > > >
> > > > > > -	sd_state = v4l2_alloc_subdev_state(sd);
> > > > > > +	sd_state = v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_ACTIVE);
> > > > >
> > > > > Shouldn't the 'which' parameters be used to decide if either ACTIVE or
> > > > > TRY have to be used ? this function is also used to set TRY formats,
> > > > > in example...
> > > > >
> > > > > Oh, maybe I got how it works, the state's which is not
> > > > > relevant but the v4l2_subdev_format's which is, as it will be used in
> > > > > the next patch to decide if the subdev's state of the file-handle's
> > > > > state should be passed to the ioctl.
> > > >
> > > > Yes. It's messy, but it's how it worked before also.
> > > >
> > > > The drivers can't really allocate TRY state as it must come from the core,
> > > > based on the filehandle. Now as I say that, makes me wonder why even expose
> > > > the option to drivers. Maybe v4l2_alloc_subdev_state() should take just the
> > > > sd parameter, and always allocate ACTIVE state, and the v4l2 core can use
> > > > another way to create the TRY state.

Drivers should not allocate state manually, it should always be done by
the core (when opening a file handle for the TRY states, and when
initializing the subdev in the probe function for the ACTIVE state).

Looking at the three drivers that call v4l2_alloc_subdev_state(), we
have the vsp1 driver that uses it to allocate its own ACTIVE state at
init time. This one is easy, it should be replaced with
v4l2_subdev_alloc_state() (or whatever that function will end up being
named, see comments to 02/36).

The rcar-vin and tegra-video drivers are doing it wrong. rcar-vin has
legacy code to support unfortunate design decisions (everything in the
"V4L2" section in rcar-v4l2.c), and newer code that gets it right (the
"V4L2 Media Controller" section in the same file). The legacy code is
used on older platforms, and the newer code on newer platforms. I know
that Niklas would love to get rid of the legacy code, and I'd also be
happy to see it go. If that's not possible, we'll have to keep exposing
v4l2_alloc_subdev_state() for this driver. Niklas, what do you think,
could we drop the legacy code after all those years ?

The situation in tegra-video is similar. They got it wrong. The driver
is in staging though, so that's fixable.

I'd propose renaming v4l2_subdev_alloc_state() to
__v4l2_subdev_alloc_state() (or something that shows it's internal),
documenting that it must not be used by new drivers, and adding an entry
in the TODO file of the tegra-video driver to fix this.

> > > init_cfg() as well as other operations used to received an
> > > array of per fh's pad_configs, and the sd pointer itself. The fh's pad
> > > configs were allocated by the core, as well as the core now allocates
> > > the per-fh's state.
> > >
> > > Before the introduction of 'state', if the 'which' flags was set to
> > > TRY then information were stored/retrieved/initialized in the per-fh
> > > pad_config, otherwise the active configuration (usually stored in the
> > > driver main structure) was used.

Correct, and the active configuration is in that case stored in a
driver-specific way (except in the vsp1 driver that uses the pad config
structure to store it), with ad-hoc accessors and lots of manual checks
in the code paths.

> > > So we had a clear separation of per-fh information and the active
> > > state. The core didn't care afaict, and passed in both, then driver had
> > > to deal with them doing the right thing by inspecting the 'which' flag.
> > >
> > > The typical pattern was:
> > >
> > >          static int subdev_ops(sd, pad_cfg, which)
> > >          {
> > >                  if (which == TRY)
> > >                          /* Operate on config stored in pad_cfg */
> > >
> > >                  else
> > >                          /*
> > >                           * Operate on config stored in subdev (and
> > >                           * applied to HW)
> > >                           */
> > >          }
> > >
> > > Or am I overlooking some cases or do you agree with my understanding
> > > so far ?
> >
> > More or less, yes. I think there are (used to be) three kinds of ops:
> >
> > - Ops that get pad_cfg and 'which' in an op specific struct. E.g. set_fmt.
> > The pad_cfg is TRY pad_config, even if 'which' == ACTIVE.

And pad_cfg is ignored in those drivers when which == ACTIVE (or it
should be at least, if it's not, it's a driver bug).

> > - Ops that don't get pad_cfg, like s_stream. 'which' is implicitly ACTIVE.

.s_stream() on a TRY configuration would be an interesting concept :-)

> Also note that operations like s_stream do not take a state as
> parameter. The driver has to fetch it from the subdev anyway
> (this in reply to the idea of having the active state as parameter vs
> retrieving it from the subdev if ACTIVE)

We could pass the state as a parameter, but given that the operation
always operates on the ACTIVE state by definition, I think this is
redundant.

> While porting the R-Car drivers on top of this series I found myself
> in the need to (in the s_stream call chain)
> 
> static int rcsi2_start_receiver(struct rcar_csi2 *priv)
> {
> 	const struct v4l2_subdev_state *state = priv->subdev.state;
> 	const struct v4l2_subdev_stream_configs *configs = &state->stream_configs;
> 
>         ...
> 
> 	/*
> 	 * Configure field handling inspecting the formats of the
> 	 * single sink pad streams.
> 	 */
> 	for (i = 0; i < configs->num_configs; ++i) {
> 		const struct v4l2_subdev_stream_config *config = configs->configs;
> 		if (config->pad != RCAR_CSI2_SINK)
> 			continue;
> 
> 		if (config->fmt.field != V4L2_FIELD_ALTERNATE)
> 			continue;
> 
> 		fld |= FLD_DET_SEL(1);
> 		fld |= FLD_FLD_EN(config->stream);
> 
> 		/* PAL vs NTSC. */
> 		if (config->fmt.height == 240)
> 			fld |= FLD_FLD_NUM(0);
> 		else
> 			fld |= FLD_FLD_NUM(1);
> 	}
> 
>         ...
> 
> }
> 
> Am I doing it wrong, or is this a case for the subdev to have to
> directly access sd->state ?

(Will reply to Tomi's reply on this)

> > - init_cfg which gets pad_cfg, but no which (as 'which' is always implicitly
> > TRY)

As proposed in 02/36, I think .init_cfg() should operate the same way
regardless of whether the state is TRY or ACTIVE, so there's no need for
a 'which' parameter (it would only open the door to abuses).

> > So pad_cfg was TRY state. Drivers could use pad_configs internally to track
> > ACTIVE state, but the core had no knowledge about this.

Correct.

> > > Now we have a 'state' that holds the array of pad_configs and along
> > > the continuation of the series will end up holding per-pad
> > > configurations.
> > >
> > > We now also have one 'state' per file-handle, and one
> > > per-subdev. As I see this, it would be natual for drivers to receive
> > > one state without knowing where it comes from. In the next patch you
> >
> > Note that only subdev's that explicitly support the new state code, and
> > allocate the state, have the subdev active state. Which means only the
> > drivers in my work branch.
> >
> > The "old" drivers work like they used to: they get the state (essentially
> > repackaged pad_cfg) for TRY cases, NULL otherwise.
> >
> > And yes, it would be natural to just get a state, but the subdev drivers
> > need to know if the context is TRY/ACTIVE. As you can see from the bullet
> > list above, the driver knows this in all the other places except init_cfg.

I agree that porting all the drivers as part of this series isn't
feasible, so passing NULL for the state in case the driver hasn't
explicitly opted-in seems fine to me.

> > > instrument the core to do exactly this: inspect the which flag and
> > > pass in the 'right' state. Ofc drivers need to have access to 'which'
> > > to know if they have to apply settings to the HW or not.
> > >
> > > Looking ahead in your series I see these structures:
> > >
> > >          struct v4l2_subdev_pad_config {
> > >                  struct v4l2_mbus_framefmt try_fmt;
> > >                  struct v4l2_rect try_crop;
> > >                  struct v4l2_rect try_compose;
> > >          };
> > >
> > >          struct v4l2_subdev_stream_config {
> > >                  u32 pad;
> > >                  u32 stream;
> > >
> > >                  struct v4l2_mbus_framefmt fmt;
> > >                  struct v4l2_rect crop;
> > >                  struct v4l2_rect compose;
> > >          };
> > >
> > >          struct v4l2_subdev_stream_configs {
> > >                  u32 num_configs;
> > >                  struct v4l2_subdev_stream_config *configs;
> > >          };
> > >
> > > All of them part of state:
> > >
> > > struct v4l2_subdev_state {
> > > 	struct mutex lock;
> > > 	u32 which;
> > > 	struct v4l2_subdev_pad_config *pads;
> > > 	struct v4l2_subdev_krouting routing;
> > > 	struct v4l2_subdev_stream_configs stream_configs;
> > > };
> > >
> > > So 'state' will hold 'TRY' information (only used for 'state'
> > > instances allocated in the fh) and 'ACTIVE' ones (used for states
> > > allocated in the sd).
> >
> > Right.
> >
> > > Looking at 'v4l2_subdev_pad_config' and 'v4l2_subdev_stream_config' they
> > > seem to describe more or less the same things: fmt, crop and compose
> > > (per pad-stream in case of stream_config). I wonder if those shouldn't
> > > be unified so that:
> > >
> > > 1) Drivers receive one state: the core passes in the 'correct' one
> > > (per-fh or per-sd) as you do in next patch
> >
> > Yes. But note that "old" drivers don't have active state.
> >
> > > 2) The 'which' information is not stored in the state but it's only
> > > 'contextual' (as in a parameter to the subdev operation) so that
> > > drivers inspect it to know if they have to apply settings to hw or not
> >
> > Yes, except we have init_cfg...

This sounds like .init_cfg() is the only blocker, which I think is
promissing :-)

> > > 3) v4l2_subdev_pad_config can be re-used and expanded, to maintain per-pad
> > > configurations regardless if they're ACTIVE or TRY, as this only depends
> > > on where the state is stored.
> >
> > pad_config is a static array of per-pad configs. stream_configs is a dynamic
> > per-stream config.

Do I understand correctly that it's both per-stream and per-pad ? I
wonder if it could make sense to store it that way, with an array of
per-pad configurations (v4l2_subdev_pad_config), which will in turn
contain per-stream configuration. I suppose it depends on the usage
patterns, which I'll understand better when reading the rest of the
series.

> > stream_configs is a super-set of pad-configs, so we could
> > drop pad_configs, but it would require changing all the drivers in
> > non-trivial ways.

I like Jacopo's proposal to unify the two, but I do agree that it's not
trivial. Here again I think it falls in the category of setting a long
term goal and trying to go in that direction, without necessarily
reaching it just yet. I wonder if we could have helper functions (maybe
they're already included later in the series ?) to abstract the
pad_config/stream_configs difference for drivers. This would also make
further reworks easier (such as storing the per-stream configuration in
v4l2_subdev_pad_config as proposed above) if we decide to address some
of the changes later.

> > v4l2_subdev_pad_config is not used or even allocated by the "new" drivers.
> > And routing & stream_configs are not used by the "old" drivers.
> >
> > > As I immagine it a subdev pad operation could look like:
> > >
> > >          static int subdev_op(sd, pad, state, which, ...)
> > >          {
> > >                  /* Doesn't matter if state is per-fh or the sd one. */
> > >                  state->pads[pad].fmt = ....;
> > >
> > >                  if (which == TRY)
> > >                          return;
> > >
> > >                  /* Apply to the HW. */
> > >          }
> > >
> > > Does it make any sense to you ? I might have missed some reason why
> > > this is not possible.
> >
> > It makes sense, but there are the buts =). I've tried to explain these in
> > the commit messages, but it's kind of confusing.
> >
> > One but I haven't mentioned in the emails is that when subdev drivers call
> > ops in other subdev drivers they pass NULL in the state.

We should really really try to avoid that. s_stream() is the obvious
main exception. Other subdev operations, such as .get_fmt() or
.set_fmt(), shouldn't be called cross-subdevs. A subdev should only care
about its own configuration, not about its neighbours. Are there
blockers here ?

> > This is fine for
> > the "old" drivers, as they expect a state only for TRY case. However, the
> > "new" drivers unfortunately expect to get a state on both TRY and ACTIVE
> > cases, and the only sensible way I figured out to handle this was the
> > v4l2_subdev_validate_state() function (patch 6).
> >
> > So, all this could be much neater, but would require modifying all subdev
> > drivers in non-trivial ways. I think this is something that can be done
> > slowly in the future.
> >
> > > > > >    	if (IS_ERR(sd_state))
> > > > > >    		return PTR_ERR(sd_state);
> > > > > >
> > > > > > diff --git a/drivers/media/platform/vsp1/vsp1_entity.c b/drivers/media/platform/vsp1/vsp1_entity.c
> > > > > > index e40bca254b8b..63ea5e472c33 100644
> > > > > > --- a/drivers/media/platform/vsp1/vsp1_entity.c
> > > > > > +++ b/drivers/media/platform/vsp1/vsp1_entity.c
> > > > > > @@ -675,7 +675,7 @@ int vsp1_entity_init(struct vsp1_device *vsp1, struct vsp1_entity *entity,
> > > > > >    	 * Allocate the pad configuration to store formats and selection
> > > > > >    	 * rectangles.
> > > > > >    	 */
> > > > > > -	entity->config = v4l2_alloc_subdev_state(&entity->subdev);
> > > > > > +	entity->config = v4l2_alloc_subdev_state(&entity->subdev, V4L2_SUBDEV_FORMAT_ACTIVE);
> > > > > >    	if (IS_ERR(entity->config)) {
> > > > > >    		media_entity_cleanup(&entity->subdev.entity);
> > > > > >    		return PTR_ERR(entity->config);
> > > > > > diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> > > > > > index e1a794f69815..04ad319fb150 100644
> > > > > > --- a/drivers/media/v4l2-core/v4l2-subdev.c
> > > > > > +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> > > > > > @@ -28,7 +28,7 @@ static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
> > > > > >    {
> > > > > >    	struct v4l2_subdev_state *state;
> > > > > >
> > > > > > -	state = v4l2_alloc_subdev_state(sd);
> > > > > > +	state = v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_TRY);
> > > > >
> > > > > At the same time I'm not sure I get the purpose of this. Don't
> > > > > init_cfg() callback implementations deal with try formats themeselves
> > > > > ? I mean, it's not a fixed rule, they can as well initialize their
> > > > > default 'active' formats, but what matters is that they initialize
> > > > > their per-fh try states ?
> > > >
> > > > That is what they do currently. init_cfg() only deals with TRY state, as
> > > > that's the only "state" (i.e. pad_config) there used to be from v4l2 core's
> > > > perspective.
> > > >
> > > > > Shouldn't init_cfg receive the fh's state so that it can initialize
> > > > > it, and just in case they need to, access their subdev's state and
> > > > > initialize them ? I'm missing what the purpose of the flag is tbh.
> > > >
> > > > Now we have (a possibility to have) state for both TRY and ACTIVE on the
> > > > v4l2 core side. The active state has to be initialized also, and a logical
> > > > way to do that is to use the init_cfg().
> > >
> > > The 'ACTIVE' state is stored in the subdev, to which init_cfg() has
> > > access, and it receives the 'TRY' state as a parameter.
> >
> > No, init_cfg gets either ACTIVE or TRY state, whichever is being allocated.
> > For "old" drivers, ACTIVE state is never allocated so they don't get
> > init_cfg calls for ACTIVE at all.
> >
> > Aaand while writing that, I realized that some drivers manually do allocate
> > ACTIVE state temporarily, which would cause init_cfg with ACTIVE state to be
> > called. I wonder if they explode... Need to check.

Is this only rcar-vin and tegra-video (vsp1 also allocates an ACTIVE
state, but it's not temporary), or are there other drivers ?

> > > It is possible to access both states and initialize them properly if
> > > I'm not mistaken.
> > >
> > > > So now, for drivers that support the new active state, init_cfg() can get
> > > > either TRY or ACTIVE state. And if you want to call, say, the driver's
> > > > set_routing() to setup the routing in the state, you have to set the 'which'
> > > > in the routing struct to a value. So somehow init_cfg needs to know if it's
> > > > initializing an ACTIVE or TRY state.
> > >
> > > I'm not sure I got this part. set_routing() as other ops will receive
> > > a state and 'which'. If my proposal above makes sensem where the state
> >
> > Yes, but if it's init_cfg calling set_routing, init_cfg has to figure out
> > the 'which' from somewhere.
> >
> > E.g. init_cfg from ub913 driver:
> >
> > static int ub913_init_cfg(struct v4l2_subdev *sd,
> > 			  struct v4l2_subdev_state *state)
> > {
> > 	u32 which = state->which;
> >
> > 	struct v4l2_subdev_route routes[] = {
> > 		{
> > 			.sink_pad = 0,
> > 			.sink_stream = 0,
> > 			.source_pad = 1,
> > 			.source_stream = 0,
> > 			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> > 		},
> > 	};
> >
> > 	struct v4l2_subdev_krouting routing = {
> > 		.which = which,
> > 		.num_routes = ARRAY_SIZE(routes),
> > 		.routes = routes,
> > 	};
> >
> > 	return ub913_set_routing(sd, state, &routing);
> > }
> >
> > It uses set_routing to setup a default routing (and set_routing in turn also
> > initializes the formats), but set_routing needs 'which'.

Could we avoid such patterns, by either initializing the routing
manually here, or splitting the logic of ub913_set_routing() in two
functions, with common code called by ub913_set_routing() and
ub913_init_cfg() ? I think calling set functions from .init_cfg() opens
the door to abuses or just bugs, and I increasingly think it should be
discouraged. I know I've written quite a few drivers that call
.set_fmt() from .init_cfg() though, it does make things easier to some
extent, but I wonder if we could do better.

This leads to the question of where to initialize the hardware state,
which is the part of the set functions that is only executed in the
ACTIVE case. I don't think this should be done at probe time in general,
but at .s_stream() time. Only drivers that allow changing formats or
routing while streaming would need to perform any hardware configutation
in the set functions, and I'm not even sure we have a single driver in
the kernel that allows this (or, to be precise, if we have a single
driver that knowingly allows this, I'm sure we have a bunch of drivers
that don't prevent this situation and will crash or behave in other bad
ways if userspace tried to change the configuration while streaming).
Maybe we should split .set_fmt() and .set_routing() in two for these
very uncommon cases, with the hardware configuration moved to separate
functions ? It has the potential to simplify the set operations and make
them safer.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 03/36] media: subdev: add 'which' to subdev state

[Laurent Pinchart]

Hi Tomi,

On Thu, Sep 16, 2021 at 04:24:19PM +0300, Tomi Valkeinen wrote:
> On 16/09/2021 16:07, Jacopo Mondi wrote:
> 
> > Also note that operations like s_stream do not take a state as
> > parameter. The driver has to fetch it from the subdev anyway
> > (this in reply to the idea of having the active state as parameter vs
> > retrieving it from the subdev if ACTIVE)
> > 
> > While porting the R-Car drivers on top of this series I found myself
> > in the need to (in the s_stream call chain)
> > 
> > static int rcsi2_start_receiver(struct rcar_csi2 *priv)
> > {
> > 	const struct v4l2_subdev_state *state = priv->subdev.state;
> > 	const struct v4l2_subdev_stream_configs *configs = &state->stream_configs;
> > 
> >          ...
> > 
> > 	/*
> > 	 * Configure field handling inspecting the formats of the
> > 	 * single sink pad streams.
> > 	 */
> > 	for (i = 0; i < configs->num_configs; ++i) {
> > 		const struct v4l2_subdev_stream_config *config = configs->configs;
> > 		if (config->pad != RCAR_CSI2_SINK)
> > 			continue;
> > 
> > 		if (config->fmt.field != V4L2_FIELD_ALTERNATE)
> > 			continue;
> > 
> > 		fld |= FLD_DET_SEL(1);
> > 		fld |= FLD_FLD_EN(config->stream);
> > 
> > 		/* PAL vs NTSC. */
> > 		if (config->fmt.height == 240)
> > 			fld |= FLD_FLD_NUM(0);
> > 		else
> > 			fld |= FLD_FLD_NUM(1);
> > 	}
> > 
> >          ...
> > 
> > }
> > 
> > Am I doing it wrong, or is this a case for the subdev to have to
> > directly access sd->state ?
> 
> In s_stream path you should:
> 
> 	state = v4l2_subdev_lock_active_state(sd);
> 
> 	<do the work with the state>
> 
> 	v4l2_subdev_unlock_state(state);
> 
> If you already have the state, e.g. in set_fmt:
> 
> 	state = v4l2_subdev_validate_and_lock_state(sd, state);
> 
> 	<do the work with the state>
> 
> 	v4l2_subdev_unlock_state(state);
> 
> Accessing the stream_configs directly is fine but not that nice. I did 
> think about some helpers, perhaps for_each_stream_config(), but I didn't 
> add that as I didn't have the need.
> 
> There's v4l2_state_get_stream_format() which can be used in many cases, 
> but we probably need something else if you need to iterate over all the 
> configs.

I really like forcing drivers to call functions that will lock the
state, at least until we can move the locks to the core (if ever). We
should move the fields of v4l2_subdev that drivers are not supposed to
access directly under a big PRIVATE comment.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 04/36] media: subdev: pass also the active state to subdevs from ioctls

[Laurent Pinchart]

Hello,

(Readding the dropped context to reply to all comments in one go, please
avoid dropping it when there's an ongoing conversation)

On Thu, Sep 16, 2021 at 11:43:17AM +0300, Tomi Valkeinen wrote:
> On Thu, Sep 16, 2021 at 10:02:31AM +0200, Jacopo Mondi wrote:
> > On Thu, Sep 16, 2021 at 09:44:23AM +0300, Tomi Valkeinen wrote:
> > > On 15/09/2021 13:17, Jacopo Mondi wrote:
> > > > On Mon, Aug 30, 2021 at 02:00:44PM +0300, Tomi Valkeinen wrote:
> > > > > At the moment when a subdev op is called, the TRY subdev state
> > > > > (subdev_fh->state) is passed as a parameter even for ACTIVE case, or
> > > > > alternatively a NULL can be passed for ACTIVE case. This used to make
> > > > > sense, as the ACTIVE state was handled internally by the subdev drivers.
> > > > >
> > > > > We now have a state for ACTIVE case in a standard place, and can pass
> > > > > that alto to the drivers. This patch changes the subdev ioctls to either
> > > > > pass the TRY or ACTIVE state to the subdev.
> > > > >
> > > > > Unfortunately many drivers call ops from other subdevs, and implicitly
> > > > > pass NULL as the state, so this is just a partial solution. A coccinelle
> > > > > spatch could perhaps be created which fixes the drivers' subdev calls.
> > > > >
> > > > > For all current upstream drivers this doesn't matter, as they do not
> > > > > expect to get a valid state for ACTIVE case. But future drivers which
> > > > > support multiplexed streaming and routing will depend on getting a state
> > > > > for both active and try cases, and the simplest way to avoid this
> > > > > problem is to introduce a helper function, used by the new drivers,
> > > > > which makes sure the driver has either the TRY or ACTIVE state. This
> > > > > helper will be introduced in a follow-up patch.
> > > > >
> > > > > Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > > > > ---
> > > > >   drivers/media/v4l2-core/v4l2-subdev.c | 73 +++++++++++++++++++++++----
> > > > >   1 file changed, 63 insertions(+), 10 deletions(-)
> > > > >
> > > > > diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> > > > > index 04ad319fb150..b3637cddca58 100644
> > > > > --- a/drivers/media/v4l2-core/v4l2-subdev.c
> > > > > +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> > > > > @@ -353,6 +353,53 @@ const struct v4l2_subdev_ops v4l2_subdev_call_wrappers = {
> > > > >   EXPORT_SYMBOL(v4l2_subdev_call_wrappers);
> > > > >
> > > > >   #if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
> > > > > +
> > > > > +static struct v4l2_subdev_state *
> > > > > +subdev_ioctl_get_state(struct v4l2_subdev *sd, struct v4l2_subdev_fh *subdev_fh,
> > > > > +		       unsigned int cmd, void *arg)
> > > > > +{
> > > > > +	u32 which;
> > > > > +
> > > > > +	switch (cmd) {
> > > > > +	default:
> > > > > +		return NULL;
> > > > > +
> > > > > +	case VIDIOC_SUBDEV_G_FMT:
> > > > > +	case VIDIOC_SUBDEV_S_FMT: {
> > > > > +		which = ((struct v4l2_subdev_format *)arg)->which;
> > > > > +		break;
> > > > > +	}
> > > > > +	case VIDIOC_SUBDEV_G_CROP:
> > > > > +	case VIDIOC_SUBDEV_S_CROP: {
> > > > > +		which = ((struct v4l2_subdev_crop *)arg)->which;
> > > > > +		break;
> > > > > +	}
> > > > > +	case VIDIOC_SUBDEV_ENUM_MBUS_CODE: {
> > > > > +		which = ((struct v4l2_subdev_mbus_code_enum *)arg)->which;
> > > > > +		break;
> > > > > +	}
> > > > > +	case VIDIOC_SUBDEV_ENUM_FRAME_SIZE: {
> > > > > +		which = ((struct v4l2_subdev_frame_size_enum *)arg)->which;
> > > > > +		break;
> > > > > +	}
> > > > > +
> > > > > +	case VIDIOC_SUBDEV_ENUM_FRAME_INTERVAL: {
> > > > > +		which = ((struct v4l2_subdev_frame_interval_enum *)arg)->which;
> > > > > +		break;
> > > > > +	}
> > > > > +
> > > > > +	case VIDIOC_SUBDEV_G_SELECTION:
> > > > > +	case VIDIOC_SUBDEV_S_SELECTION: {
> > > > > +		which = ((struct v4l2_subdev_selection *)arg)->which;
> > > > > +		break;
> > > > > +	}
> > > > > +	}
> > > > > +
> > > > > +	return which == V4L2_SUBDEV_FORMAT_TRY ?
> > > > > +			     subdev_fh->state :
> > > > > +			     v4l2_subdev_get_active_state(sd);
> > > >
> > > > Why this additional indirection layer ?
> > > >
> > > > v4l2_subdev_get_active_state(struct v4l2_subdev *sd)
> > > > {
> > > >      return sd->state;
> > > > }
> > >
> > > I wanted to hide all direct accesses to the state to make it easier to
> > > figure out how and where the state is accessed.

And it's an inline function, so it should be totally fine.

> > > > I understand you want to have the core to fish the 'right' state for
> > > > the drivers, but this then requires to protect against bridge drivers
> > > > calling an op through v4l2_subdev_call() with a NULL state by using
> > > > one more indirection like
> > > >
> > > > 	state = v4l2_subdev_validate_state(sd, state);
> > > >
> > > >          static inline struct v4l2_subdev_state *
> > > >          v4l2_subdev_validate_state(struct v4l2_subdev *sd,
> > > >                                     struct v4l2_subdev_state *state)
> > > >          {
> > > >                  return state ? state : sd->state;
> > > >          }
> > > >
> > > > Which I very much don't like as it implicitly changes what state the
> > > > driver receives to work-around a design flaw (the fact that even if
> > > > the core tries to, there's no gurantee state is valid).
> >>
> >> I don't like it either. My idea was that in the future the subdevs would
> >> always get the correct state. In other words, all the subdev drivers calling
> >> ops in other subdevs would be changed to pass the state correctly. Thus the
> >> v4l2_subdev_validate_state() is a helper for the transition period, which
> >> can easily be dropped when the drivers work correctly.
> > 
> > Most of the drivers that call v4l2_subdev_call() with a NULL state are
> > bridge drivers operating in the ACTIVE use case. Even if we get to a
> > point where we remove all calls passing in a NULL state, what are the
> > bridges expected to provide as a state to the subdev they call
> > operations on ? The subdev's state as well ? something like
> > 
> >          v4l2_subdev_call(sd, pad, set_fmt, sd->state, ...)
> 
> Yes. Although we should hide it, so that when calling ops that support 
> state, the subdev drivers do:
> 
> v4l2_subdev_call_state(sd, pad, set_fmt, ...)
> 
> and v4l2_subdev_call_state macro (maybe needs a better name...) uses 
> sd->state as the second parameter to the op.

Even better, we should have a v4l2_subdev_call_pad_set_fmt(sd, &fmt)
(name to be bikeshedded).

> > With your current dynamicaly allocated state, sd->state could very well
> > be NULL.
> 
> Yes, that sounds logical to me. The subdev drivers don't have active 
> state, and th driver code doesn't use it, so they get NULL.
> 
> > I still think this could be way simpler if we assume that the state
> > received as parameter is the file-handle's one (like it was for
> > pad_configs) and in the active case we let driver use their own
> > sd->state.
> 
> I'm kind of okay-ish with that too.
> 
> It doesn't feel logical to me, and afaik the drivers should not touch 
> the file-handle's state when dealing with active case so passing it is 
> kind of wrong, but I agree that it is how things have been.
> 
> I don't think it's any simpler, though. This change wouldn't affect the 
> old drivers, and the new drivers would just use another helper instead 
> of v4l2_subdev_validate_state. And if we change the v4l2_subdev_call() 
> call as discussed above, the new drivers can drop the 
> v4l2_subdev_validate_state().
> 
> So I would argue that the new approach is (will be) simpler, but it's 
> different than what we have now.

I agree with Tomi here, I think passing the correct state will lead to
the best end-result, even if it complicates the transition.

> >>> If feel like it would be much simpler if:
> >>>
> >>> 1) The core passes in a state which always come from the fh (the
> >>>      try_state) when it do_ioctl()
> >>>
> >>> 2) Drivers use their 'active' states embedded in the subdev or the
> >>>      'try' state passed in as parameter and decide
> >>>      which one to use based on the context. It's a pattern we have
> >>>      everywere already when using the per-fh try formats
> >>>
> >>> 	switch (which) {
> >>> 	case V4L2_SUBDEV_FORMAT_TRY:
> >>> 		return v4l2_subdev_get_try_format(&sd, sd_state, pad);
> >>> 	case V4L2_SUBDEV_FORMAT_ACTIVE:
> >>> 		return &sd->fmt;
> >>> 	default:
> >>> 		return NULL;
> >>> 	}
> >>
> >> This is possible, of course. We could do this if we decide we don't want the
> >> subdev drivers to pass the state properly in the future.
> >>
> >> However, if, in my series, I currently call this in a subdev driver:
> >>
> >> state = v4l2_subdev_validate_state(sd, state);
> >>
> >> With the change you suggest I'd just do (possibly with a helper):
> >>
> >> state = which == V4L2_SUBDEV_FORMAT_TRY ? state : sd->state;
> >>
> >> Is it any better?
> >>
> >>> I liked the idea to have the core pass in a state without the driver
> >>> having to care where it comes from, but it requires too many
> >>> indirections and implicities like the above shown
> >>> v4l2_subdev_validate_state().
> >>>
> >>> One middle-ground could be to have the core pass in the 'correct' state as it
> >>> does in your series, and default it to sd->state if a bridge driver
> >>> calls an op through v4l2_subdev_call() with a NULL state, as the
> >>> format is implicitly ACTIVE in that case.
> >>
> >> If you mean changing all the bridge drivers so that they would give the
> >> state properly, yes, that was my plan (I think I mentioned it in a commit
> >> desc, perhaps). It's not a trivial change, though, as v4l2_subdev_call()
> >> cannot handle this at the moment.
> > 
> > Unfortunately this cannot be done automatically in v4l2_subdev_call(),
> > at least not easily.
> > 
> >> I believe it should be doable with coccinelle. Maybe add a new macro,
> >> v4l2_subdev_call_state() or such, which gives the active state in the second
> >> parameter (looks like all the ops have the state as the second param). Then
> >> use coccinelle to find all the v4l2_subdev_call uses which call ops that get
> >> a state, verify that the current caller uses NULL as the state, and change
> >> v4l2_subdev_call to v4l2_subdev_call_state.
> > 
> > Even if we beautify it, I think bridge drivers passing as parameter to
> > a subdev operation a subdev attribute, like in the above shown
> > 
> >          v4l2_subdev_call(sd, pad, set_fmt, sd->state, ...)
> > 
> > is unecessary and a possible source of confusion, with the subdev
> > driver having to infer where the state comes from and the possibility
> 
> Why do the drivers need to infer where the state comes from? Except for 
> the init_cfg case, but that can be fixed other ways
> 
> > of it being NULL anyway if the bridge operates with a non-state aware
> > subdev which has not allocated a state (which is harmelss now, as they
> > won't be interested in 'state').
> 
> Yes, it can be NULL, but it can be NULL already now, and as you say, 
> it's harmless.
> 
> > It could be made easier if we clearly say drivers "if it's TRY, expect
> > a state, if is ACTIVE use your own one (if you want to)". This seems
> > impossible to get wrong to me for subdev drivers.

It makes it more complex for drivers, compared to using the state they
receive without having to care if it's TRY or ACTIVE. It also opens the
door to abuses, by allowing access to a TRY state even when operating on
the ACTIVE configuration (and the other way around). I really like the
idea of simplifying subdev drivers as much as possible by removing the
need to make decisions there. Look at all the subdevs we have, and the
different ways they implement the same thing for no specific reason (and
of course most of them broken in different more or less subtle ways).

> We can write such a clear statement for this new approach also.
> 
> >>> This ofc requires the state to be embedded (ie it's always there) and
> >>> that state-aware drivers to have properly initialized it, but that's a
> >>> given.
> >>
> >> Why does the state need to be embedded? If the subdev driver is not
> >> state-aware, it does not expect to get a state except for the TRY case.
> >> Passing NULL for those drivers should be fine.
> > 
> > It doesn't -need- to be, I just think it avoids allocation and
> > releasing at run-time and offers a place where to store subdev-wide
> > configurations to all drivers as an opt-in feature.
> 
> They do have that option already, they just need to manually allocate 
> the state. If we embed the state, the subdev drivers need to manually 
> initialize the state. It doesn't really change much, except now we have 
> a clear indication (sd->state != NULL) that the driver is state aware. 

I'd prefer embedding it in the long term, but I like the fact that we
have a clear indication that the driver is state-aware. Maybe we can
record the fact that is should be embedded, to transition once drivers
get converted (it should then be a simple change if all accesses to the
state are done in the core, and even more so if they all go through
helper functions).

> And also, 99% of the drivers don't need the state, which might have some 
> memory use impact.

The impact is small, it's the pad and stream arrays that are bigger.
Let's also not forget that drivers should transition to the new model,
so they will end up needing the state. Still, as stated above, a dynamic
allocation may be the best option to start with.

> And the reason for the subdev drivers having to manually allocate/init 
> the state is that there's no place in core to do that. Maybe the various 
> v4l2_*_register_subdev might do it, but it wasn't clear to me if it 
> would work in practice or not.
> 
> So at the moment you have to call the v4l2_subdev_alloc_state() after 
> media_entity_pads_init() but before registering the subdev (or possibly 
> before registering an async notifier).
> 
> > Of course we pay a little price in the size of the subdev, but it's
> > all in-kernel stuff and going forward the state could very wel just
> > become the standard 'subdev_config'
> > 
> >          struct v4l2_subdev {
> >                  ....
> > 
> >                  struct v4l2_subdev_config {
> >                          struct v4l2_subdev_routing routes;
> >                          struct v4l2_subdev_streams streams;
> >                  } config;
> >          };
> > 
> > But yeah, allocated or embedded is tangential and I defer this call to
> > maintainers which know better than me for sure.
> 
> With the wrapper functions, subdev drivers never touch the sd->state 
> directly, and thus changing it from allocated to embedded in the future 
> should be trivial.
> 
> >>> Nonetheless, this considerations do not defeat the purpose of having a
> >>> 'state', as currently we have
> >>>
> >>> struct v4l2_subdev_state {
> >>>           struct v4l2_subdev_krouting; /* Use for TRY and ACTIVE */
> >>>           struct v4l2_stream_configs; /* Use for ACTIVE */
> >>
> >> stream_configs is used for TRY also.
> >>
> >>>           struct v4l2_pad_configs; /* Used for TRY */
> >>
> >> Probably no point in this, but this _could_ also be used for ACTIVE. We
> >> could have state aware drivers that don't use routing or streams, and use
> >> just a plain old pad_configs array. This would allow moving the ACTIVE
> >> pad_configs from the driver to the core.
> > 
> > That would be nice, but it would be better is stream_configs could be
> > used for pad-only drivers (it's just about assuming stream = 0 for all
> > of them, right ?). But yes, my point is about trying to centralize the
> > subdev configuration in one place. But that's probably for later
> > indeed.
> 
> The stream configs require routing to be set first, as routing defines 
> the number of stream configs. There are probably ways to hide the 
> routing part for simple drivers that don't really need routing but would 
> still want to use stream configs.

Unification would be nice, let's try to think about it.

> > > But, as you suggest, probably a better direction is to try to get rid of
> > > pad_configs instead.
> > >
> > > > };
> > > >
> > > > and v4l2_stream_configs is a super-set of v4l2_pad_configs
> > > >
> > > > If we could get to
> > > >
> > > > struct v4l2_subdev_state {
> > > >          struct v4l2_subdev_krouting; /* Use for TRY and ACTIVE */
> > > >          struct v4l2_stream_configs; /* Use for TRY and ACTIVE */
> > > > };
> > > >
> > > > This could turn out to be pretty neat, as it allows 'new' drivers to
> > > > maintain their current formats and routings in a subdev 'state'
> > > > instead of scattering those information in the driver-wide structure
> > > > as they currently do for formats, crops and whatnot. This can ofc go
> > > > on top.
> > >
> > > Yes, that's the long term plan, but it's a huge change. And when I say plan,
> > > I don't mean I'm planning to change all the current drivers, I'm just saying
> > > my series is designed so that it allows these to be done in the future.

Sounds good to me. As mentioned before, setting the direction is
required, but we don't need to get there in one go.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 00/36] v4l: subdev internal routing and streams

[Laurent Pinchart]

Hi Tomi,

On Mon, Sep 20, 2021 at 01:19:54PM +0300, Tomi Valkeinen wrote:
> On 30/08/2021 14:00, Tomi Valkeinen wrote:
> > Hi,
> > 
> > This is v8 of the multiplexed streams series. v7 can be found from:
> > 
> > https://lore.kernel.org/linux-media/20210524104408.599645-1-tomi.valkeinen@ideasonboard.com/
> > 
> > The main change in this version is the implementation and use of
> > centralized active state for subdevs.
> > 
> > I have pushed my work branch to:
> > 
> > git://git.kernel.org/pub/scm/linux/kernel/git/tomba/linux.git multistream/work-v8
> > 
> > which contains the patches in this series, along with subdev drivers
> > using multiplexed streams.
> > 
> > Both this series and the branch above are based on top of today's
> > git://linuxtv.org/media_tree.git master.
> > 
> > The documentation still needs improving, but I hope the docs in this
> > series, and the drivers in the work branch, are enough to give the
> > reviewers enough information to do a review.
> > 
> > As can be guessed from the work branch, I have been testing this series
> > with TI's FPDLink setup. I have also done a "backwards compatibility"
> > test by dropping all multiplexed streams patches from the CAL driver
> > (the CSI-2 RX on the TI SoC), and using the FPDLink drivers with
> > single-stream configuration.
> 
> We've had good discussions with Jacopo about this series.

I hope my recent contribution was also useful to some extent :-) Up to
patch 04/36, I like the direction this is taking and I'm quite confident
that we'll reach an agreement. We need to get feedback from Sakari too
though.

> I chose the approaches in this series based on what I think the API 
> should be, even if the API has behaved differently before. And I think 
> I'm also leaning forward a bit, in the sense that the full benefit of 
> the API can only be had after more changes to the core and subdev 
> drivers (changes which may or may not happen).
> 
> If I understood Jacopo correctly, his comments were essentially that my 
> approach is different than the current one, and as the current drivers 
> anyway do things the old way, this is very confusing. Basically I create 
> two different kinds of subdev drivers: the old and new ones, which 
> manage state differently.
> 
> I want to summarize two particular topics:
> 
> 1) Active state & subdev ops
> 
> In upstream we have v4l2_subdev_state which contains only the pad_config 
> array. This state is "try" state, it's allocated per file-handle, and 
> passed to the subdev drivers when executing subdev ioctls in try-mode 
> (which == V4L2_SUBDEV_FORMAT_TRY). This try-state is sometimes also 
> passed to the subdev drivers when executing in active-mode 
> (V4L2_SUBDEV_FORMAT_ACTIVE), but the drivers are supposed to ignore it.
> 
> There is also an active-state, but it's driver-specific and 
> driver-internal. The drivers check the 'which' value, and either use the 
> passed try-state, or the internal state.

To be very clear here, let's note that the driver-internal state is
stored in a driver-specific format, which does not reuse the state
structure used for the TRY state.

> What I did in this series aims to have both try- and active-states in 
> v4l2 core, and passing the correct state to subdevs so that they don't 
> (necessarily) need any internal state. There are some issues with it, 
> which have been discussed, but I believe those issues can be fixed.
> 
> The subdev drivers need to be written to use this new active-state, so 
> it doesn't affect the current drivers.
> 
> The question is, do we want to go that way?

__     __  _______   ________
\ \   / / |  _____| |  ______|
 \ \ / /  | |       | |
  \ v /   | |_____  | |______
   | |    |  _____| |______  |
   | |    | |              | |
   | |    | |_____   ______| |
   |_|    |_______| |________|

(please let me know if you require additional clarification)

> We could as well keep the 
> current behavior of subdev drivers only getting the try-state as a 
> parameter, and the drivers digging out the active state manually. This 
> active state could either be internal to the driver, or it could be in 
> the base struct v4l2_subdev (see also topic 2).
> 
> 2) Shared subdev active-state
> 
> The try-state is specific to a file-handle, and afaics have no real 
> race-issues as it's not really shared. Although I guess in theory an 
> application could call subdev ioctls from multiple threads using the 
> same fd.

That's right. We could possibly serialize ioctl calls in v4l2-subdev.c.

> In upstream the subdev drivers' internal state is managed fully by the 
> subdev drivers. The drivers are expected to handle necessary locking in 
> their subdev ops and interrupt handlers. If, say, v4l2 core needs to get 
> a format from the subdev, it calls a subdev op to get it.

"supposed to" is the correct term here. Most of them don't (including
drivers I have written myself), which I believe shows quite clearly that
the API is wrong and that this shouldn't be left to drivers to handle.

> In my series I aimed to a shared active-state. The state is located in a 
> known place, struct v4l2_subdev, and can be accessed without the subdev 
> driver's help. This requires locking, which I have implemented.
> 
> At the moment the only real benefit with this is reading the routing 
> table while doing pipeline validation: Instead of having to dynamically 
> allocate memory and call the subdev op to create a copy of the routing 
> table (for each subdev, possibly multiple times), the validator can just 
> lock the state, and use it. And, in fact, there is no get_routing subdev 
> op at all.
> 
> But this means that the subdev drivers that support this new 
> active-state have to handle locking for the active state, and the 
> "mindset" is different than previously.

That's the right mindset I believe, and forcing drivers to use helper
functions that ensure proper locking is the right way to go in my
opinion.

> So the question is, do we want to go that way? We could as well mandate 
> that the active-state can only be accessed via subdev's ops (and add the 
> get-routing, of course), and the subdev manages the locking internally.

Been there, failed, let's not repeat the same mistake.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 05/36] media: subdev: add subdev state locking

[Laurent Pinchart]

Hi Tomi,

Thank you for the patch.

On Mon, Aug 30, 2021 at 02:00:45PM +0300, Tomi Valkeinen wrote:
> The V4L2 subdevs have managed without centralized locking for the state
> (previously pad_config), as the TRY state is supposedly safe (although I
> believe two TRY ioctls for the same fd would race), and the ACTIVE
> state, and its locking, is managed by the drivers internally.
> 
> We now have ACTIVE state in a centralized position, and need locking.
> Strictly speaking the locking is only needed for new drivers that use
> the new state, as the current drivers continue behaving as they used to.
> 
> Add a mutex to the struct v4l2_subdev_state, along with a few helper
> functions for locking/unlocking.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 43 +++++++++++++++++----
>  include/media/v4l2-subdev.h           | 55 +++++++++++++++++++++++++--
>  2 files changed, 88 insertions(+), 10 deletions(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index b3637cddca58..b1e65488210d 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -26,9 +26,11 @@
>  #if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
>  static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
>  {
> +	static struct lock_class_key __key;
>  	struct v4l2_subdev_state *state;
>  
> -	state = v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_TRY);
> +	state = __v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_TRY,
> +					  "v4l2_subdev_fh->state", &__key);

What's the reason for not using the v4l2_alloc_subdev_state() macro here
?

>  	if (IS_ERR(state))
>  		return PTR_ERR(state);
>  
> @@ -924,8 +926,10 @@ int v4l2_subdev_link_validate(struct media_link *link)
>  EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
>  
>  struct v4l2_subdev_state *
> -v4l2_alloc_subdev_state(struct v4l2_subdev *sd,
> -			enum v4l2_subdev_format_whence which)
> +__v4l2_alloc_subdev_state(struct v4l2_subdev *sd,
> +			  enum v4l2_subdev_format_whence which,
> +			  const char *lock_name,
> +			  struct lock_class_key *lock_key)
>  {
>  	struct v4l2_subdev_state *state;
>  	int ret;
> @@ -934,6 +938,8 @@ v4l2_alloc_subdev_state(struct v4l2_subdev *sd,
>  	if (!state)
>  		return ERR_PTR(-ENOMEM);
>  
> +	__mutex_init(&state->lock, lock_name, lock_key);
> +
>  	state->which = which;
>  
>  	if (sd->entity.num_pads) {
> @@ -960,13 +966,15 @@ v4l2_alloc_subdev_state(struct v4l2_subdev *sd,
>  
>  	return ERR_PTR(ret);
>  }
> -EXPORT_SYMBOL_GPL(v4l2_alloc_subdev_state);
> +EXPORT_SYMBOL_GPL(__v4l2_alloc_subdev_state);
>  
>  void v4l2_free_subdev_state(struct v4l2_subdev_state *state)
>  {
>  	if (!state)
>  		return;
>  
> +	mutex_destroy(&state->lock);
> +
>  	kvfree(state->pads);
>  	kfree(state);
>  }
> @@ -1001,11 +1009,12 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
>  
> -int v4l2_subdev_alloc_state(struct v4l2_subdev *sd)
> +int __v4l2_subdev_alloc_state(struct v4l2_subdev *sd, const char *name,
> +			      struct lock_class_key *key)
>  {
>  	struct v4l2_subdev_state *state;
>  
> -	state = v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_ACTIVE);
> +	state = __v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_ACTIVE, name, key);

I already know that Sakari will ask for a line wrap at 80 columns, and
that would be my preference as well :-) I won't repeat the comment in
the rest of the series. Going over 80 columns is fine when it improves
readability, but in many places keeping lines short enough would be
nicer.

>  	if (IS_ERR(state))
>  		return PTR_ERR(state);
>  
> @@ -1013,7 +1022,7 @@ int v4l2_subdev_alloc_state(struct v4l2_subdev *sd)
>  
>  	return 0;
>  }
> -EXPORT_SYMBOL_GPL(v4l2_subdev_alloc_state);
> +EXPORT_SYMBOL_GPL(__v4l2_subdev_alloc_state);
>  
>  void v4l2_subdev_free_state(struct v4l2_subdev *sd)
>  {
> @@ -1021,3 +1030,23 @@ void v4l2_subdev_free_state(struct v4l2_subdev *sd)
>  	sd->state = NULL;
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_free_state);
> +
> +struct v4l2_subdev_state *v4l2_subdev_lock_active_state(struct v4l2_subdev *sd)
> +{
> +	mutex_lock(&sd->state->lock);
> +
> +	return sd->state;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_lock_active_state);
> +
> +void v4l2_subdev_lock_state(struct v4l2_subdev_state *state)
> +{
> +	mutex_lock(&state->lock);
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_lock_state);
> +
> +void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state)
> +{
> +	mutex_unlock(&state->lock);
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_unlock_state);
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 5ec78ffda4f5..52a725281b23 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -655,6 +655,7 @@ struct v4l2_subdev_pad_config {
>  /**
>   * struct v4l2_subdev_state - Used for storing subdev state information.
>   *
> + * @lock: mutex for the state
>   * @which: state type (from enum v4l2_subdev_format_whence)
>   * @pads: &struct v4l2_subdev_pad_config array
>   *
> @@ -663,6 +664,7 @@ struct v4l2_subdev_pad_config {
>   * %V4L2_SUBDEV_FORMAT_ACTIVE it is safe to pass %NULL.
>   */
>  struct v4l2_subdev_state {
> +	struct mutex lock;
>  	u32 which;
>  	struct v4l2_subdev_pad_config *pads;
>  };
> @@ -1147,9 +1149,18 @@ int v4l2_subdev_link_validate(struct media_link *link);
>   *
>   * Must call v4l2_free_subdev_state() when state is no longer needed.
>   */
> +#define v4l2_alloc_subdev_state(sd, which)                                     \
> +	({                                                                     \
> +		static struct lock_class_key __key;                            \
> +		const char *name = KBUILD_BASENAME                             \
> +			":" __stringify(__LINE__) ":sd->state->lock";          \
> +		__v4l2_alloc_subdev_state(sd, which, name, &__key);            \
> +	})
> +
>  struct v4l2_subdev_state *
> -v4l2_alloc_subdev_state(struct v4l2_subdev *sd,
> -			enum v4l2_subdev_format_whence which);
> +__v4l2_alloc_subdev_state(struct v4l2_subdev *sd,
> +			  enum v4l2_subdev_format_whence which,
> +			  const char *lock_name, struct lock_class_key *key);
>  
>  /**
>   * v4l2_free_subdev_state - free a v4l2_subdev_state
> @@ -1234,7 +1245,16 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
>   *
>   * Must call v4l2_subdev_free_state() when the state is no longer needed.
>   */
> -int v4l2_subdev_alloc_state(struct v4l2_subdev *sd);
> +#define v4l2_subdev_alloc_state(sd)                                            \
> +	({                                                                     \
> +		static struct lock_class_key __key;                            \
> +		const char *name = KBUILD_BASENAME                             \
> +			":" __stringify(__LINE__) ":sd->state->lock";          \
> +		__v4l2_subdev_alloc_state(sd, name, &__key);                   \
> +	})
> +
> +int __v4l2_subdev_alloc_state(struct v4l2_subdev *sd, const char *name,
> +			      struct lock_class_key *key);
>  
>  /**
>   * v4l2_subdev_free_state() - Free the active subdev state for subdevice
> @@ -1258,4 +1278,33 @@ v4l2_subdev_get_active_state(struct v4l2_subdev *sd)
>  	return sd->state;
>  }
>  
> +/**
> + * v4l2_subdev_lock_active_state() - Lock and return the active subdev state for subdevice
> + * @sd: The subdevice
> + *
> + * Return the locked active state for the subdevice, or NULL if the subdev
> + * does not support active state.
> + *
> + * Must be unlocked with v4l2_subdev_unlock_state() after use.
> + */
> +struct v4l2_subdev_state *v4l2_subdev_lock_active_state(struct v4l2_subdev *sd);
> +
> +/**
> + * v4l2_subdev_lock_state() - Lock the subdev state
> + * @state: The subdevice state
> + *
> + * Lock the given subdev state.
> + *
> + * Must be unlocked with v4l2_subdev_unlock_state() after use.
> + */
> +void v4l2_subdev_lock_state(struct v4l2_subdev_state *state);

This seems to be used only to lock the state passed to the subdev
operation by the caller. Could the caller lock the state instead ? This
could possibly be done by wrapping the v4l2_subdev_call() calls in
dedicated helper functions.

> +
> +/**
> + * v4l2_subdev_unlock_state() - Unlock the subdev state
> + * @state: The subdevice state
> + *
> + * Unlock the given subdev state.
> + */
> +void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state);
> +
>  #endif

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 06/36] media: subdev: Add v4l2_subdev_validate(_and_lock)_state()

[Laurent Pinchart]

Hi Tomi,

Thank you for the patch.

On Mon, Aug 30, 2021 at 02:00:46PM +0300, Tomi Valkeinen wrote:
> All suitable subdev ops are now passed either the TRY or the ACTIVE
> state by the v4l2 core. However, other subdrivers can still call the ops
> passing NULL as the state, implying the active case.
> 
> Thus all subdev drivers supporting active state need to handle the NULL
> state case.

Do they ? Can't we mandate that the callers pass the correct state ? Do
you think that would make the transition too difficult ?

The way I understand the issue, this would only be needed to facilitate
the transition. Is there a reason why the drivers you've ported (CAL &
co.) use v4l2_subdev_validate_and_lock_state() after completing the
transition, or is the correct state always passed by the caller ?

> Additionally, the subdev drivers usually need to lock the
> state.
> 
> Add two helper functions to easen the transition to centrally managed
> ACTIVE state. v4l2_subdev_validate_state() ensures that the state is not
> NULL, and v4l2_subdev_validate_and_lock_state() does the same and
> additionally locks the state.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  include/media/v4l2-subdev.h | 41 +++++++++++++++++++++++++++++++++++++
>  1 file changed, 41 insertions(+)
> 
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 52a725281b23..2290b5025fc0 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -1307,4 +1307,45 @@ void v4l2_subdev_lock_state(struct v4l2_subdev_state *state);
>   */
>  void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state);
>  
> +/**
> + * v4l2_subdev_validate_state() - Gets the TRY or ACTIVE subdev state
> + * @sd: subdevice
> + * @state: subdevice state as passed to the subdev op
> + *
> + * Subdev ops used to be sometimes called with NULL as the state for ACTIVE
> + * case. Even if the v4l2 core now passes proper state for both TRY and
> + * ACTIVE cases, a subdev driver may call an op in another subdev driver,
> + * passing NULL.
> + *
> + * This function can be used as a helper to get the state also for the ACTIVE
> + * case. The subdev driver that supports ACTIVE state can use this function
> + * as the first thing in its ops, ensuring that the state variable contains
> + * either the TRY or ACTIVE state.
> + */
> +static inline struct v4l2_subdev_state *
> +v4l2_subdev_validate_state(struct v4l2_subdev *sd,
> +			   struct v4l2_subdev_state *state)
> +{
> +	return state ? state : sd->state;
> +}

This doesn't seem to be used by the drivers you've ported, or by the
R-Car and GMSL patches from Jacopo. Do we need this function ?

> +
> +/**
> + * v4l2_subdev_validate_and_lock_state() - Gets locked TRY or ACTIVE subdev state
> + * @sd: subdevice
> + * @state: subdevice state as passed to the subdev op
> + *
> + * This is a helper function which does the same as v4l2_subdev_validate_state
> + * () except that it also locks the state.
> + */
> +static inline struct v4l2_subdev_state *
> +v4l2_subdev_validate_and_lock_state(struct v4l2_subdev *sd,
> +				    struct v4l2_subdev_state *state)
> +{
> +	state = state ? state : sd->state;
> +
> +	v4l2_subdev_lock_state(state);
> +
> +	return state;
> +}
> +
>  #endif

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 01/36] media: subdev: rename subdev-state alloc & free

[Tomi Valkeinen]

On 27/09/2021 02:06, Laurent Pinchart wrote:
> Hi Tomi,
> 
> Thank you for the patch.
> 
> On Mon, Aug 30, 2021 at 02:00:41PM +0300, Tomi Valkeinen wrote:
>> The recently added v4l2_subdev_alloc_state() and
>> v4l2_subdev_free_state() were somewhat badly named. They allocate/free
>> the state that can be used for a subdev, but they don't change the
>> subdev itself in any way.
>>
>> In the future we want to have the subdev state available easily for all
>> subdevs, and we want to store the state to subdev struct. Thus we will
>> be needing a new function which allocates the state and stores it into
>> the subdev struct.
>>
>> This patch renames v4l2_subdev_alloc/free_state functions to
>> v4l2_alloc/free_subdev_state, so that we can later use
>> v4l2_subdev_alloc/free_state for the versions which work on the subdev
>> struct.
> 
> With have video_device_alloc() and media_request_alloc(), should we use
> v4l2_subdev_state_alloc() and v4l2_subdev_state_free() to be consistent
> ?
> 
> With or without this (but preferably with ;-)),
> 
> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> 
> Note that regardless of the name, if we have both
> v4l2_subdev_alloc_state() (to allocate the state and store it in the
> subdev structure) and v4l2_subdev_alloc_state() (to allocate the state),
> it could be confusing for driver authors. Let's discuss this further in
> the patch series when the problem arises (if it does).

It is confusing. I guess to prove the point you used 
"v4l2_subdev_alloc_state" twice above ;).

To me, the change you propose makes it more confusing than my version, 
but I think that's up to the reader.

My thinking was that v4l2_subdev_xyz function does something to the 
subdev, thus v4l2_alloc_subdev_state does something that won't affect 
the subdev. With v4l2_subdev_alloc_state() and v4l2_subdev_state_alloc() 
that logic doesn't hold.

Perhaps we should just use something totally else for the version that 
allocates and stores the state? v4l2_subdev_setup_state or something in 
that direction? Or v4l2_subdev_setup, i.e. don't even refer to "state".

You propose "v4l2_subdev_init_done" in a later reply, so I think however 
we restructure (or don't) the subdev init, we should rename 
v4l2_subdev_alloc_state to something else.

We can discuss that rename in the other thread. As to your proposal for 
this patch, I agree with it presuming we also rename 
v4l2_subdev_alloc_state.

  Tomi

Re: [PATCH v8 02/36] media: subdev: add active state to struct v4l2_subdev

[Tomi Valkeinen]

On 27/09/2021 02:58, Laurent Pinchart wrote:
> Hello,
> 
> On Thu, Sep 16, 2021 at 12:36:33PM +0300, Tomi Valkeinen wrote:
>> On 16/09/2021 11:08, Jacopo Mondi wrote:
>>> On Thu, Sep 16, 2021 at 09:52:42AM +0300, Tomi Valkeinen wrote:
>>>> On 16/09/2021 09:17, Tomi Valkeinen wrote:
>>>>> On 15/09/2021 12:44, Jacopo Mondi wrote:
>>>>>> On Mon, Aug 30, 2021 at 02:00:42PM +0300, Tomi Valkeinen wrote:
>>>>>>> Add a new 'state' field to struct v4l2_subdev to which we can store the
>>>>>>> active state of a subdev. This will place the subdev configuration into
>>>>>>> a known place, allowing us to use the state directly from the v4l2
>>>>>>> framework, thus simplifying the drivers.
>>>>>>>
>>>>>>> We also add v4l2_subdev_alloc_state() and v4l2_subdev_free_state(),
>>>>>>> which need to be used by the drivers that support subdev state in struct
>>>>>>> v4l2_subdev.
>>>>>>>
>>>>>>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>>>>>>> ---
>>>>>>>     drivers/media/v4l2-core/v4l2-subdev.c | 21 ++++++++++++++++
>>>>>>>     include/media/v4l2-subdev.h           | 36 +++++++++++++++++++++++++++
>>>>>>>     2 files changed, 57 insertions(+)
>>>>>>>
>>>>>>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c
>>>>>>> b/drivers/media/v4l2-core/v4l2-subdev.c
>>>>>>> index 26a34a8e3d37..e1a794f69815 100644
>>>>>>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>>>>>>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>>>>>>> @@ -943,3 +943,24 @@ void v4l2_subdev_notify_event(struct
>>>>>>> v4l2_subdev *sd,
>>>>>>>         v4l2_subdev_notify(sd, V4L2_DEVICE_NOTIFY_EVENT, (void *)ev);
>>>>>>>     }
>>>>>>>     EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
>>>>>>> +
>>>>>>> +int v4l2_subdev_alloc_state(struct v4l2_subdev *sd)
>>>>>>> +{
>>>>>>> +    struct v4l2_subdev_state *state;
>>>>>>> +
>>>>>>> +    state = v4l2_alloc_subdev_state(sd);
>>>>
>>>> Replying to this again, as the second email didn't actually cover all the
>>>> topics...
>>>>
>>>>>> So, I think this is one source of confusion about init_cfg.
>>>>>>
>>>>>> v4l2_alloc_subdev_state() calls init_cfg() and 'state-aware' driver
>>>>>> are now supposed to allocate their state by calling
>>>>>> v4l2_subdev_alloc_state(), in the same way as the core does for the
>>>>>> file-handle ones.
>>>>>>
>>>>>> This will lead to init_cfg to be called for the 'active' (ie owned by
>>>>>> the subdev) state, and then you need to add context to the state (by
>>>>>> adding a 'which' field) to know what state you're dealing with.
>>>>>>
>>>>>> According to the init_cfg() documentation
>>>>>>
>>>>>>     * @init_cfg: initialize the pad config to default values
>>>>>>
>>>>>> the op has to be called in order to initialize the per-file-handle
>>>>>> context, not the active one.
>>>>>
>>>>> I have missed updating the documentation there =).
>>>>
>>>> The documentation above doesn't imply per-file-handle context or TRY case,
>>>> afaics. It just says "initialize state to default". Unless "pad config"
>>>> always means TRY, which I think it doesn't as the drivers have internally
>>>> been using pad configs.
>>>
>>> If they do, they would have the 'active' pad_configs  allocated or
>>> embedded somewhere in their driver structures, they would not receive
>>> it as parameter. Or have I missed where the core is capable of fishing
>>> the 'right' pad_configs ? I think the same should happen for state.
>>
>> Yes, that is correct. I was just saying that the documentation doesn't
>> say that init_cfg is only used for the TRY case, even if that has been
>> the case in practice.
>>
>>>> But it's true that so far init_cfg has only been called for TRY case, and
>>>> perhaps that's enough of a reason to keep it so.
>>>>
>>>>>> I would rather just embed 'struct v4l2_subdev_state' in 'struct
>>>>>> v4l2_subdev', have the core going through the
>>>>>
>>>>> Why would embedding the state change anything?
>>>>>
>>>>>> 'v4l2_subdev_alloc_state()' to initialize the per-fh state, but have
>>>>>> drivers initialize their own state at probe time. If they need for
>>>>>> some reason to access their 'active' state at init_cfg() time, they
>>>>>> caan fish it from their subdev.
>>>>>>
>>>>>> If I'm not mistaken this will remove the need to have a which filed in
>>>>>> the state, as I think the 'context' should be inferred from the
>>>>>> 'which' argument embedded in the ops-specific structures, and not held
>>>>>> in the state itself.
>>>>
>>>> It's true that the state's which field is mainly (probably only) needed for
>>>> handling the init_cfg. It could be solved in other ways too:
>>>>
>>>> - New subdev op to initialize active state
>>>> - New subdev op which gets 'which' as a parameter, to initialize both states
>>>> (state-aware drivers wouldn't need to implement the old init_cfg)
>>>> - Coccinelle to change init_cfg to get the which as a parameter
>>>>
>>>> Without doing any deep thinking, the middle one sounds best to me.
>>>
>>> Isn't it simpler if you just don't call init_cfg for the 'active'
>>> state ? Driver will initialize them at probe time and that's it, then
>>> you can remove 'which' from the state (and from routing tables too if I'm
>>> not mistaken).
>>
>> Routing needs the 'which' similarly to other config structs, like
>> v4l2_subdev_format.
>>
>> I did a quick implementation for the second case, which allows me to
>> remove 'which' from the state.
>>
>> As for "simpler"... Both ways have pros and cons. I'm my mind this new
>> way is simpler.
>>
>> Afaics, in the end (i.e. after doing the v4l2_subdev_call change), the
>> only bigger difference is how the state-aware drivers implement their
>> ops. With my approach, they get the correct state and that's it. With
>> your approach, they need to use a helper function (or do it manually) to
>> get the state based on 'which'.
> 
> I'd like to propose a third approach (or at least I believe it's a third
> one). I agree with Jacopo that the state structure should not have a
> which field, that's a layering violation. The state is a state,
> regardless of whether it holds TRY or ACTIVE data. What are the current
> blockers that would prevent that ?

Oh, I agree with that too. I didn't add the 'which' field to state 
because I thought it's good =). It is supposed to be temporary solution. 
init_cfg() is the issue here.

It think I had these options:

- Change init_cfg() to take 'which' as a parameter
- Change init_cfg() implementations to not use 'which'
- Add new version for new drivers, init_cfg_state() or such, which 
either gets the which as a parameter or doesn't use which.
- Add 'which' to state.

I chose the fourth one as it's a very small change, and can be removed 
easily in the future when the underlying problem is solved.

> However, I think .init_cfg() should be used to initialize *all* states,
> both TRY and ACTIVE. That will simplify driver implementation (but
> centralizing the initialization in a single place) and ensure that the
> default value for the ACTIVE state always matches the default value for
> the TRY state, which I think is important.

I agree.

> When it comes to calling v4l2_subdev_alloc_state() in drivers, I also
> agree with Jacopo, I'm not extremely fond of it. This should be
> automatic, drivers shouldn't have to care about allocating the ACTIVE
> state. However, I also agree with Tomi that there's no real place to do
> this today. I think we need to restructure subdev initialization, which
> currently has very few rules. This could take the form, for instance, of

I agree with your agreeings above too.

> 1. calling v4l2_subdev_init() as an early subdev init function
> 2. setting fields of v4l2_subdev manually
> 3. initializing the media_entity embedded in the subdev
> 4. calling a new "initialization finalization" function that will
>     initialize the state embedded in v4l2_subdev (as opposed to
>     allocating it dynamically as done today)

Why is the allocated/embedded relevant? The drivers won't know which one 
is used.

> That's the simplest option as it really ressembles what we do today, and
> would more or less only require embedded the state in v4l2_subdev and
> renaming v4l2_subdev_alloc_state() to v4l2_subdev_init_done() (or
> similar, that's likely not a great name).

Yes. But I don't see embedding relevant. So we could just rename 
v4l2_subdev_alloc_state() and we should be there.

> A more intrusive change would restructure the subdev initialization
> further, by handling the media_entity initialization transparently
> (which would require passing more arguments to v4l2_subdev_init(),
> probably creating a new version of that function). That's more complex
> and I think it's out of scope for this series, but it may still be
> useful to keep the long term goal in mind to try and align with the
> direction.

Yes, there are many things that can be improved further. But in the 
context of this series, I'd like to keep the improvements such that they 
require no changes to the existing drivers.

  Tomi

Re: [PATCH v8 03/36] media: subdev: add 'which' to subdev state

[Tomi Valkeinen]

On 27/09/2021 03:46, Laurent Pinchart wrote:
> Hello,
> 
> On Thu, Sep 16, 2021 at 03:07:52PM +0200, Jacopo Mondi wrote:
>> On Mon, Sep 13, 2021 at 05:26:45PM +0300, Tomi Valkeinen wrote:
>>> On 13/09/2021 16:38, Jacopo Mondi wrote:
>>>> On Mon, Sep 13, 2021 at 03:17:01PM +0300, Tomi Valkeinen wrote:
>>>>> On 13/09/2021 14:41, Jacopo Mondi wrote:
>>>>>> On Mon, Aug 30, 2021 at 02:00:43PM +0300, Tomi Valkeinen wrote:
>>>>>>> The subdev state is passed to functions in the media drivers, and
>>>>>>> usually either V4L2_SUBDEV_FORMAT_ACTIVE or V4L2_SUBDEV_FORMAT_TRY is
>>>>>>> also given to the function in one way or another.
>>>>>>>
>>>>>>> One op where this is not the case is v4l2_subdev_pad_ops.init_cfg. One
>>>>>>> could argue that the initialization of the state should be the same for
>>>>>>> both ACTIVE and TRY cases, but unfortunately that is not the case:
>>>>>>>
>>>>>>> - Some drivers do also other things than just touch the state when
>>>>>>> dealing with ACTIVE, e.g. if there is extra state outside the standard
>>>>>>> subdev state.
>>>>>>> - Some drivers might need to create, say, struct v4l2_subdev_format
>>>>>>> which has 'which' field, and that needs to be filled with either ACTIVE
>>>>>>> or TRY.
>>>>>>>
>>>>>>> Currently init_cfg is only called for TRY case from the v4l2 framework,
>>>>>>> passing the TRY state. Some drivers call their own init_cfg, passing
>>>>>>> NULL as the state, which is used to indicate ACTIVE case.
>>>>>>>
>>>>>>> In the future we want to pass subdev's active state from the v4l2
>>>>>>> framework side, so we need a solution to this.
>>>>>>>
>>>>>>> We could change the init_cfg() to include the TRY/ACTIVE value, which
>>>>>>> would require changing more or less all the drivers. Instead, I have
>>>>>>> added 'which' field to the subdev state itself, filled at state
>>>>>>> allocation time, which only requires changes to the drivers that
>>>>>>> allocate a state themselves.
>>>>>>>
>>>>>>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>>>>>>> ---
>>>>>>>     drivers/media/platform/rcar-vin/rcar-v4l2.c |  2 +-
>>>>>>>     drivers/media/platform/vsp1/vsp1_entity.c   |  2 +-
>>>>>>>     drivers/media/v4l2-core/v4l2-subdev.c       | 10 +++++++---
>>>>>>>     drivers/staging/media/tegra-video/vi.c      |  2 +-
>>>>>>>     include/media/v4l2-subdev.h                 |  7 ++++++-
>>>>>>>     5 files changed, 16 insertions(+), 7 deletions(-)
>>>>>>>
>>>>>>> diff --git a/drivers/media/platform/rcar-vin/rcar-v4l2.c b/drivers/media/platform/rcar-vin/rcar-v4l2.c
>>>>>>> index 5f4fa8c48f68..1de30d5b437f 100644
>>>>>>> --- a/drivers/media/platform/rcar-vin/rcar-v4l2.c
>>>>>>> +++ b/drivers/media/platform/rcar-vin/rcar-v4l2.c
>>>>>>> @@ -252,7 +252,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
>>>>>>>     	u32 width, height;
>>>>>>>     	int ret;
>>>>>>>
>>>>>>> -	sd_state = v4l2_alloc_subdev_state(sd);
>>>>>>> +	sd_state = v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_ACTIVE);
>>>>>>
>>>>>> Shouldn't the 'which' parameters be used to decide if either ACTIVE or
>>>>>> TRY have to be used ? this function is also used to set TRY formats,
>>>>>> in example...
>>>>>>
>>>>>> Oh, maybe I got how it works, the state's which is not
>>>>>> relevant but the v4l2_subdev_format's which is, as it will be used in
>>>>>> the next patch to decide if the subdev's state of the file-handle's
>>>>>> state should be passed to the ioctl.
>>>>>
>>>>> Yes. It's messy, but it's how it worked before also.
>>>>>
>>>>> The drivers can't really allocate TRY state as it must come from the core,
>>>>> based on the filehandle. Now as I say that, makes me wonder why even expose
>>>>> the option to drivers. Maybe v4l2_alloc_subdev_state() should take just the
>>>>> sd parameter, and always allocate ACTIVE state, and the v4l2 core can use
>>>>> another way to create the TRY state.
> 
> Drivers should not allocate state manually, it should always be done by
> the core (when opening a file handle for the TRY states, and when
> initializing the subdev in the probe function for the ACTIVE state).
> 
> Looking at the three drivers that call v4l2_alloc_subdev_state(), we
> have the vsp1 driver that uses it to allocate its own ACTIVE state at
> init time. This one is easy, it should be replaced with
> v4l2_subdev_alloc_state() (or whatever that function will end up being
> named, see comments to 02/36).

I don't follow here. First you say that drivers should not allocate 
state. But then, I think, you say that vsp1 allocating state is fine 
after some change?

My understanding is that vsp1 is fine. Of course, if the function it 
calls gets renamed we need to rename it in vsp1 too.

> The rcar-vin and tegra-video drivers are doing it wrong. rcar-vin has
> legacy code to support unfortunate design decisions (everything in the
> "V4L2" section in rcar-v4l2.c), and newer code that gets it right (the
> "V4L2 Media Controller" section in the same file). The legacy code is
> used on older platforms, and the newer code on newer platforms. I know
> that Niklas would love to get rid of the legacy code, and I'd also be
> happy to see it go. If that's not possible, we'll have to keep exposing
> v4l2_alloc_subdev_state() for this driver. Niklas, what do you think,
> could we drop the legacy code after all those years ?
> 
> The situation in tegra-video is similar. They got it wrong. The driver
> is in staging though, so that's fixable.
> 
> I'd propose renaming v4l2_subdev_alloc_state() to
> __v4l2_subdev_alloc_state() (or something that shows it's internal),
> documenting that it must not be used by new drivers, and adding an entry
> in the TODO file of the tegra-video driver to fix this.

Yes, that sounds fine. Subdev drivers allocating the state is not 
something that should be done in any new drivers.

>>>> init_cfg() as well as other operations used to received an
>>>> array of per fh's pad_configs, and the sd pointer itself. The fh's pad
>>>> configs were allocated by the core, as well as the core now allocates
>>>> the per-fh's state.
>>>>
>>>> Before the introduction of 'state', if the 'which' flags was set to
>>>> TRY then information were stored/retrieved/initialized in the per-fh
>>>> pad_config, otherwise the active configuration (usually stored in the
>>>> driver main structure) was used.
> 
> Correct, and the active configuration is in that case stored in a
> driver-specific way (except in the vsp1 driver that uses the pad config
> structure to store it), with ad-hoc accessors and lots of manual checks
> in the code paths.
> 
>>>> So we had a clear separation of per-fh information and the active
>>>> state. The core didn't care afaict, and passed in both, then driver had
>>>> to deal with them doing the right thing by inspecting the 'which' flag.
>>>>
>>>> The typical pattern was:
>>>>
>>>>           static int subdev_ops(sd, pad_cfg, which)
>>>>           {
>>>>                   if (which == TRY)
>>>>                           /* Operate on config stored in pad_cfg */
>>>>
>>>>                   else
>>>>                           /*
>>>>                            * Operate on config stored in subdev (and
>>>>                            * applied to HW)
>>>>                            */
>>>>           }
>>>>
>>>> Or am I overlooking some cases or do you agree with my understanding
>>>> so far ?
>>>
>>> More or less, yes. I think there are (used to be) three kinds of ops:
>>>
>>> - Ops that get pad_cfg and 'which' in an op specific struct. E.g. set_fmt.
>>> The pad_cfg is TRY pad_config, even if 'which' == ACTIVE.
> 
> And pad_cfg is ignored in those drivers when which == ACTIVE (or it
> should be at least, if it's not, it's a driver bug).
> 
>>> - Ops that don't get pad_cfg, like s_stream. 'which' is implicitly ACTIVE.
> 
> .s_stream() on a TRY configuration would be an interesting concept :-)

I think it would be a good one, though =). It could be used to validate 
the whole pipeline. But it would probably be rather difficult to 
implement with the current v4l2 framework.

>> Also note that operations like s_stream do not take a state as
>> parameter. The driver has to fetch it from the subdev anyway
>> (this in reply to the idea of having the active state as parameter vs
>> retrieving it from the subdev if ACTIVE)
> 
> We could pass the state as a parameter, but given that the operation
> always operates on the ACTIVE state by definition, I think this is
> redundant.
> 
>> While porting the R-Car drivers on top of this series I found myself
>> in the need to (in the s_stream call chain)
>>
>> static int rcsi2_start_receiver(struct rcar_csi2 *priv)
>> {
>> 	const struct v4l2_subdev_state *state = priv->subdev.state;
>> 	const struct v4l2_subdev_stream_configs *configs = &state->stream_configs;
>>
>>          ...
>>
>> 	/*
>> 	 * Configure field handling inspecting the formats of the
>> 	 * single sink pad streams.
>> 	 */
>> 	for (i = 0; i < configs->num_configs; ++i) {
>> 		const struct v4l2_subdev_stream_config *config = configs->configs;
>> 		if (config->pad != RCAR_CSI2_SINK)
>> 			continue;
>>
>> 		if (config->fmt.field != V4L2_FIELD_ALTERNATE)
>> 			continue;
>>
>> 		fld |= FLD_DET_SEL(1);
>> 		fld |= FLD_FLD_EN(config->stream);
>>
>> 		/* PAL vs NTSC. */
>> 		if (config->fmt.height == 240)
>> 			fld |= FLD_FLD_NUM(0);
>> 		else
>> 			fld |= FLD_FLD_NUM(1);
>> 	}
>>
>>          ...
>>
>> }
>>
>> Am I doing it wrong, or is this a case for the subdev to have to
>> directly access sd->state ?
> 
> (Will reply to Tomi's reply on this)
> 
>>> - init_cfg which gets pad_cfg, but no which (as 'which' is always implicitly
>>> TRY)
> 
> As proposed in 02/36, I think .init_cfg() should operate the same way
> regardless of whether the state is TRY or ACTIVE, so there's no need for
> a 'which' parameter (it would only open the door to abuses).
> 
>>> So pad_cfg was TRY state. Drivers could use pad_configs internally to track
>>> ACTIVE state, but the core had no knowledge about this.
> 
> Correct.
> 
>>>> Now we have a 'state' that holds the array of pad_configs and along
>>>> the continuation of the series will end up holding per-pad
>>>> configurations.
>>>>
>>>> We now also have one 'state' per file-handle, and one
>>>> per-subdev. As I see this, it would be natual for drivers to receive
>>>> one state without knowing where it comes from. In the next patch you
>>>
>>> Note that only subdev's that explicitly support the new state code, and
>>> allocate the state, have the subdev active state. Which means only the
>>> drivers in my work branch.
>>>
>>> The "old" drivers work like they used to: they get the state (essentially
>>> repackaged pad_cfg) for TRY cases, NULL otherwise.
>>>
>>> And yes, it would be natural to just get a state, but the subdev drivers
>>> need to know if the context is TRY/ACTIVE. As you can see from the bullet
>>> list above, the driver knows this in all the other places except init_cfg.
> 
> I agree that porting all the drivers as part of this series isn't
> feasible, so passing NULL for the state in case the driver hasn't
> explicitly opted-in seems fine to me.
> 
>>>> instrument the core to do exactly this: inspect the which flag and
>>>> pass in the 'right' state. Ofc drivers need to have access to 'which'
>>>> to know if they have to apply settings to the HW or not.
>>>>
>>>> Looking ahead in your series I see these structures:
>>>>
>>>>           struct v4l2_subdev_pad_config {
>>>>                   struct v4l2_mbus_framefmt try_fmt;
>>>>                   struct v4l2_rect try_crop;
>>>>                   struct v4l2_rect try_compose;
>>>>           };
>>>>
>>>>           struct v4l2_subdev_stream_config {
>>>>                   u32 pad;
>>>>                   u32 stream;
>>>>
>>>>                   struct v4l2_mbus_framefmt fmt;
>>>>                   struct v4l2_rect crop;
>>>>                   struct v4l2_rect compose;
>>>>           };
>>>>
>>>>           struct v4l2_subdev_stream_configs {
>>>>                   u32 num_configs;
>>>>                   struct v4l2_subdev_stream_config *configs;
>>>>           };
>>>>
>>>> All of them part of state:
>>>>
>>>> struct v4l2_subdev_state {
>>>> 	struct mutex lock;
>>>> 	u32 which;
>>>> 	struct v4l2_subdev_pad_config *pads;
>>>> 	struct v4l2_subdev_krouting routing;
>>>> 	struct v4l2_subdev_stream_configs stream_configs;
>>>> };
>>>>
>>>> So 'state' will hold 'TRY' information (only used for 'state'
>>>> instances allocated in the fh) and 'ACTIVE' ones (used for states
>>>> allocated in the sd).
>>>
>>> Right.
>>>
>>>> Looking at 'v4l2_subdev_pad_config' and 'v4l2_subdev_stream_config' they
>>>> seem to describe more or less the same things: fmt, crop and compose
>>>> (per pad-stream in case of stream_config). I wonder if those shouldn't
>>>> be unified so that:
>>>>
>>>> 1) Drivers receive one state: the core passes in the 'correct' one
>>>> (per-fh or per-sd) as you do in next patch
>>>
>>> Yes. But note that "old" drivers don't have active state.
>>>
>>>> 2) The 'which' information is not stored in the state but it's only
>>>> 'contextual' (as in a parameter to the subdev operation) so that
>>>> drivers inspect it to know if they have to apply settings to hw or not
>>>
>>> Yes, except we have init_cfg...
> 
> This sounds like .init_cfg() is the only blocker, which I think is
> promissing :-)
> 
>>>> 3) v4l2_subdev_pad_config can be re-used and expanded, to maintain per-pad
>>>> configurations regardless if they're ACTIVE or TRY, as this only depends
>>>> on where the state is stored.
>>>
>>> pad_config is a static array of per-pad configs. stream_configs is a dynamic
>>> per-stream config.
> 
> Do I understand correctly that it's both per-stream and per-pad ? I
> wonder if it could make sense to store it that way, with an array of
> per-pad configurations (v4l2_subdev_pad_config), which will in turn
> contain per-stream configuration. I suppose it depends on the usage
> patterns, which I'll understand better when reading the rest of the
> series.

I'm not sure what you mean with "both per-stream and per-pad". It's per 
(stream, pad) tuple.

At the moment the internal storage is essentially just an array of
{ pad, stream, config } items. It could also be changed as you suggest, 
but afaics that would require more dynamic allocations. The APIs should 
be designed so that the drivers don't need to care about the internal 
storage.

>>> stream_configs is a super-set of pad-configs, so we could
>>> drop pad_configs, but it would require changing all the drivers in
>>> non-trivial ways.
> 
> I like Jacopo's proposal to unify the two, but I do agree that it's not
> trivial. Here again I think it falls in the category of setting a long
> term goal and trying to go in that direction, without necessarily
> reaching it just yet. I wonder if we could have helper functions (maybe
> they're already included later in the series ?) to abstract the
> pad_config/stream_configs difference for drivers. This would also make
> further reworks easier (such as storing the per-stream configuration in
> v4l2_subdev_pad_config as proposed above) if we decide to address some
> of the changes later.

If you mean do we have functions to hide the stream (i.e. stream = 0), 
no, I haven't added such. I have not worked towards making 
non-multiplexed-streams subdev drivers use the new APIs.

That said, I have tried to design this series so that it is possible. 
And "new" and "old" drivers should be compatible with each other.

>>> v4l2_subdev_pad_config is not used or even allocated by the "new" drivers.
>>> And routing & stream_configs are not used by the "old" drivers.
>>>
>>>> As I immagine it a subdev pad operation could look like:
>>>>
>>>>           static int subdev_op(sd, pad, state, which, ...)
>>>>           {
>>>>                   /* Doesn't matter if state is per-fh or the sd one. */
>>>>                   state->pads[pad].fmt = ....;
>>>>
>>>>                   if (which == TRY)
>>>>                           return;
>>>>
>>>>                   /* Apply to the HW. */
>>>>           }
>>>>
>>>> Does it make any sense to you ? I might have missed some reason why
>>>> this is not possible.
>>>
>>> It makes sense, but there are the buts =). I've tried to explain these in
>>> the commit messages, but it's kind of confusing.
>>>
>>> One but I haven't mentioned in the emails is that when subdev drivers call
>>> ops in other subdev drivers they pass NULL in the state.
> 
> We should really really try to avoid that. s_stream() is the obvious
> main exception. Other subdev operations, such as .get_fmt() or
> .set_fmt(), shouldn't be called cross-subdevs. A subdev should only care
> about its own configuration, not about its neighbours. Are there
> blockers here ?

There are a lot of subdevs using v4l2_subdev_call(), including uses of 
set_fmt.

>>> This is fine for
>>> the "old" drivers, as they expect a state only for TRY case. However, the
>>> "new" drivers unfortunately expect to get a state on both TRY and ACTIVE
>>> cases, and the only sensible way I figured out to handle this was the
>>> v4l2_subdev_validate_state() function (patch 6).
>>>
>>> So, all this could be much neater, but would require modifying all subdev
>>> drivers in non-trivial ways. I think this is something that can be done
>>> slowly in the future.
>>>
>>>>>>>     	if (IS_ERR(sd_state))
>>>>>>>     		return PTR_ERR(sd_state);
>>>>>>>
>>>>>>> diff --git a/drivers/media/platform/vsp1/vsp1_entity.c b/drivers/media/platform/vsp1/vsp1_entity.c
>>>>>>> index e40bca254b8b..63ea5e472c33 100644
>>>>>>> --- a/drivers/media/platform/vsp1/vsp1_entity.c
>>>>>>> +++ b/drivers/media/platform/vsp1/vsp1_entity.c
>>>>>>> @@ -675,7 +675,7 @@ int vsp1_entity_init(struct vsp1_device *vsp1, struct vsp1_entity *entity,
>>>>>>>     	 * Allocate the pad configuration to store formats and selection
>>>>>>>     	 * rectangles.
>>>>>>>     	 */
>>>>>>> -	entity->config = v4l2_alloc_subdev_state(&entity->subdev);
>>>>>>> +	entity->config = v4l2_alloc_subdev_state(&entity->subdev, V4L2_SUBDEV_FORMAT_ACTIVE);
>>>>>>>     	if (IS_ERR(entity->config)) {
>>>>>>>     		media_entity_cleanup(&entity->subdev.entity);
>>>>>>>     		return PTR_ERR(entity->config);
>>>>>>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>>>>>>> index e1a794f69815..04ad319fb150 100644
>>>>>>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>>>>>>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>>>>>>> @@ -28,7 +28,7 @@ static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
>>>>>>>     {
>>>>>>>     	struct v4l2_subdev_state *state;
>>>>>>>
>>>>>>> -	state = v4l2_alloc_subdev_state(sd);
>>>>>>> +	state = v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_TRY);
>>>>>>
>>>>>> At the same time I'm not sure I get the purpose of this. Don't
>>>>>> init_cfg() callback implementations deal with try formats themeselves
>>>>>> ? I mean, it's not a fixed rule, they can as well initialize their
>>>>>> default 'active' formats, but what matters is that they initialize
>>>>>> their per-fh try states ?
>>>>>
>>>>> That is what they do currently. init_cfg() only deals with TRY state, as
>>>>> that's the only "state" (i.e. pad_config) there used to be from v4l2 core's
>>>>> perspective.
>>>>>
>>>>>> Shouldn't init_cfg receive the fh's state so that it can initialize
>>>>>> it, and just in case they need to, access their subdev's state and
>>>>>> initialize them ? I'm missing what the purpose of the flag is tbh.
>>>>>
>>>>> Now we have (a possibility to have) state for both TRY and ACTIVE on the
>>>>> v4l2 core side. The active state has to be initialized also, and a logical
>>>>> way to do that is to use the init_cfg().
>>>>
>>>> The 'ACTIVE' state is stored in the subdev, to which init_cfg() has
>>>> access, and it receives the 'TRY' state as a parameter.
>>>
>>> No, init_cfg gets either ACTIVE or TRY state, whichever is being allocated.
>>> For "old" drivers, ACTIVE state is never allocated so they don't get
>>> init_cfg calls for ACTIVE at all.
>>>
>>> Aaand while writing that, I realized that some drivers manually do allocate
>>> ACTIVE state temporarily, which would cause init_cfg with ACTIVE state to be
>>> called. I wonder if they explode... Need to check.
> 
> Is this only rcar-vin and tegra-video (vsp1 also allocates an ACTIVE
> state, but it's not temporary), or are there other drivers ?

It's those three drivers. And the above is not an issue, I didn't think 
it to the end.

Those three drivers used to call v4l2_subdev_alloc_pad_config(), which 
always calls init_cfg. So the situation is no different with this series.

>>>> It is possible to access both states and initialize them properly if
>>>> I'm not mistaken.
>>>>
>>>>> So now, for drivers that support the new active state, init_cfg() can get
>>>>> either TRY or ACTIVE state. And if you want to call, say, the driver's
>>>>> set_routing() to setup the routing in the state, you have to set the 'which'
>>>>> in the routing struct to a value. So somehow init_cfg needs to know if it's
>>>>> initializing an ACTIVE or TRY state.
>>>>
>>>> I'm not sure I got this part. set_routing() as other ops will receive
>>>> a state and 'which'. If my proposal above makes sensem where the state
>>>
>>> Yes, but if it's init_cfg calling set_routing, init_cfg has to figure out
>>> the 'which' from somewhere.
>>>
>>> E.g. init_cfg from ub913 driver:
>>>
>>> static int ub913_init_cfg(struct v4l2_subdev *sd,
>>> 			  struct v4l2_subdev_state *state)
>>> {
>>> 	u32 which = state->which;
>>>
>>> 	struct v4l2_subdev_route routes[] = {
>>> 		{
>>> 			.sink_pad = 0,
>>> 			.sink_stream = 0,
>>> 			.source_pad = 1,
>>> 			.source_stream = 0,
>>> 			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
>>> 		},
>>> 	};
>>>
>>> 	struct v4l2_subdev_krouting routing = {
>>> 		.which = which,
>>> 		.num_routes = ARRAY_SIZE(routes),
>>> 		.routes = routes,
>>> 	};
>>>
>>> 	return ub913_set_routing(sd, state, &routing);
>>> }
>>>
>>> It uses set_routing to setup a default routing (and set_routing in turn also
>>> initializes the formats), but set_routing needs 'which'.
> 
> Could we avoid such patterns, by either initializing the routing
> manually here, or splitting the logic of ub913_set_routing() in two
> functions, with common code called by ub913_set_routing() and
> ub913_init_cfg() ? I think calling set functions from .init_cfg() opens
> the door to abuses or just bugs, and I increasingly think it should be
> discouraged. I know I've written quite a few drivers that call
> .set_fmt() from .init_cfg() though, it does make things easier to some
> extent, but I wonder if we could do better.

It is possible, at least in theory.

The problem is that the 'which' is embedded into many structs used in 
this context, e.g. struct v4l2_subdev_format and struct 
v4l2_subdev_krouting. If we don't have 'which', none of the functions 
using those structs is usable. But what's worse, those structs are used 
in the subdev state. So even if we write new helper funcs that don't 
need 'which', we still need it in the end when storing the data to the 
subdev state.

The annoying thing is that I don't think we really need the 'which' in 
any of those functions or the state. Afaik the only real purpose of 
'which' is as a parameter when the userspace calls a subdev ioctl.

But getting rid of 'which' is again something I'd rather not start in 
the context of this series, as I believe it will be a huge task in itself.

> This leads to the question of where to initialize the hardware state,
> which is the part of the set functions that is only executed in the
> ACTIVE case. I don't think this should be done at probe time in general,
> but at .s_stream() time. Only drivers that allow changing formats or

Yes, I agree.

> routing while streaming would need to perform any hardware configutation
> in the set functions, and I'm not even sure we have a single driver in
> the kernel that allows this (or, to be precise, if we have a single
> driver that knowingly allows this, I'm sure we have a bunch of drivers
> that don't prevent this situation and will crash or behave in other bad
> ways if userspace tried to change the configuration while streaming).
> Maybe we should split .set_fmt() and .set_routing() in two for these
> very uncommon cases, with the hardware configuration moved to separate
> functions ? It has the potential to simplify the set operations and make
> them safer.

Right, that's a good point: if the driver supports changing fmt while 
streaming is enabled, set_fmt needs to have the 'which'. At the moment 
they have it, via the struct v4l2_subdev_format. If we were to remove 
the 'which' field (which would require separating the uAPI and kAPI 
facing struct v4l2_subdev_format), we would either need to pass 'which' 
some other way or have a new op, as you suggest.

I would put this to the "later" bucket. It certainly won't be a simple 
change.

  Tomi

Re: [PATCH v8 03/36] media: subdev: add 'which' to subdev state

[Tomi Valkeinen]

On 27/09/2021 03:48, Laurent Pinchart wrote:
> Hi Tomi,
> 
> On Thu, Sep 16, 2021 at 04:24:19PM +0300, Tomi Valkeinen wrote:
>> On 16/09/2021 16:07, Jacopo Mondi wrote:
>>
>>> Also note that operations like s_stream do not take a state as
>>> parameter. The driver has to fetch it from the subdev anyway
>>> (this in reply to the idea of having the active state as parameter vs
>>> retrieving it from the subdev if ACTIVE)
>>>
>>> While porting the R-Car drivers on top of this series I found myself
>>> in the need to (in the s_stream call chain)
>>>
>>> static int rcsi2_start_receiver(struct rcar_csi2 *priv)
>>> {
>>> 	const struct v4l2_subdev_state *state = priv->subdev.state;
>>> 	const struct v4l2_subdev_stream_configs *configs = &state->stream_configs;
>>>
>>>           ...
>>>
>>> 	/*
>>> 	 * Configure field handling inspecting the formats of the
>>> 	 * single sink pad streams.
>>> 	 */
>>> 	for (i = 0; i < configs->num_configs; ++i) {
>>> 		const struct v4l2_subdev_stream_config *config = configs->configs;
>>> 		if (config->pad != RCAR_CSI2_SINK)
>>> 			continue;
>>>
>>> 		if (config->fmt.field != V4L2_FIELD_ALTERNATE)
>>> 			continue;
>>>
>>> 		fld |= FLD_DET_SEL(1);
>>> 		fld |= FLD_FLD_EN(config->stream);
>>>
>>> 		/* PAL vs NTSC. */
>>> 		if (config->fmt.height == 240)
>>> 			fld |= FLD_FLD_NUM(0);
>>> 		else
>>> 			fld |= FLD_FLD_NUM(1);
>>> 	}
>>>
>>>           ...
>>>
>>> }
>>>
>>> Am I doing it wrong, or is this a case for the subdev to have to
>>> directly access sd->state ?
>>
>> In s_stream path you should:
>>
>> 	state = v4l2_subdev_lock_active_state(sd);
>>
>> 	<do the work with the state>
>>
>> 	v4l2_subdev_unlock_state(state);
>>
>> If you already have the state, e.g. in set_fmt:
>>
>> 	state = v4l2_subdev_validate_and_lock_state(sd, state);
>>
>> 	<do the work with the state>
>>
>> 	v4l2_subdev_unlock_state(state);
>>
>> Accessing the stream_configs directly is fine but not that nice. I did
>> think about some helpers, perhaps for_each_stream_config(), but I didn't
>> add that as I didn't have the need.
>>
>> There's v4l2_state_get_stream_format() which can be used in many cases,
>> but we probably need something else if you need to iterate over all the
>> configs.
> 
> I really like forcing drivers to call functions that will lock the
> state, at least until we can move the locks to the core (if ever). We
> should move the fields of v4l2_subdev that drivers are not supposed to
> access directly under a big PRIVATE comment.

Can you clarify what you mean here? Did you mean you like functions that 
will _check_ the lock? Or did you just mean that you like 
v4l2_subdev_lock_state() better than mutex_lock(state->lock)?

Well, in any case, I think my series does both =). I can add the private 
comment to subdev. In fact, at one time I did not have sd->state, but 
sd->_state, just to make sure no one accesses it.

  Tomi

Re: [PATCH v8 02/36] media: subdev: add active state to struct v4l2_subdev

[Laurent Pinchart]

Hi Tomi,

On Mon, Sep 27, 2021 at 10:05:12AM +0300, Tomi Valkeinen wrote:
> On 27/09/2021 02:58, Laurent Pinchart wrote:
> > On Thu, Sep 16, 2021 at 12:36:33PM +0300, Tomi Valkeinen wrote:
> >> On 16/09/2021 11:08, Jacopo Mondi wrote:
> >>> On Thu, Sep 16, 2021 at 09:52:42AM +0300, Tomi Valkeinen wrote:
> >>>> On 16/09/2021 09:17, Tomi Valkeinen wrote:
> >>>>> On 15/09/2021 12:44, Jacopo Mondi wrote:
> >>>>>> On Mon, Aug 30, 2021 at 02:00:42PM +0300, Tomi Valkeinen wrote:
> >>>>>>> Add a new 'state' field to struct v4l2_subdev to which we can store the
> >>>>>>> active state of a subdev. This will place the subdev configuration into
> >>>>>>> a known place, allowing us to use the state directly from the v4l2
> >>>>>>> framework, thus simplifying the drivers.
> >>>>>>>
> >>>>>>> We also add v4l2_subdev_alloc_state() and v4l2_subdev_free_state(),
> >>>>>>> which need to be used by the drivers that support subdev state in struct
> >>>>>>> v4l2_subdev.
> >>>>>>>
> >>>>>>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> >>>>>>> ---
> >>>>>>>     drivers/media/v4l2-core/v4l2-subdev.c | 21 ++++++++++++++++
> >>>>>>>     include/media/v4l2-subdev.h           | 36 +++++++++++++++++++++++++++
> >>>>>>>     2 files changed, 57 insertions(+)
> >>>>>>>
> >>>>>>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c
> >>>>>>> b/drivers/media/v4l2-core/v4l2-subdev.c
> >>>>>>> index 26a34a8e3d37..e1a794f69815 100644
> >>>>>>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> >>>>>>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> >>>>>>> @@ -943,3 +943,24 @@ void v4l2_subdev_notify_event(struct
> >>>>>>> v4l2_subdev *sd,
> >>>>>>>         v4l2_subdev_notify(sd, V4L2_DEVICE_NOTIFY_EVENT, (void *)ev);
> >>>>>>>     }
> >>>>>>>     EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
> >>>>>>> +
> >>>>>>> +int v4l2_subdev_alloc_state(struct v4l2_subdev *sd)
> >>>>>>> +{
> >>>>>>> +    struct v4l2_subdev_state *state;
> >>>>>>> +
> >>>>>>> +    state = v4l2_alloc_subdev_state(sd);
> >>>>
> >>>> Replying to this again, as the second email didn't actually cover all the
> >>>> topics...
> >>>>
> >>>>>> So, I think this is one source of confusion about init_cfg.
> >>>>>>
> >>>>>> v4l2_alloc_subdev_state() calls init_cfg() and 'state-aware' driver
> >>>>>> are now supposed to allocate their state by calling
> >>>>>> v4l2_subdev_alloc_state(), in the same way as the core does for the
> >>>>>> file-handle ones.
> >>>>>>
> >>>>>> This will lead to init_cfg to be called for the 'active' (ie owned by
> >>>>>> the subdev) state, and then you need to add context to the state (by
> >>>>>> adding a 'which' field) to know what state you're dealing with.
> >>>>>>
> >>>>>> According to the init_cfg() documentation
> >>>>>>
> >>>>>>     * @init_cfg: initialize the pad config to default values
> >>>>>>
> >>>>>> the op has to be called in order to initialize the per-file-handle
> >>>>>> context, not the active one.
> >>>>>
> >>>>> I have missed updating the documentation there =).
> >>>>
> >>>> The documentation above doesn't imply per-file-handle context or TRY case,
> >>>> afaics. It just says "initialize state to default". Unless "pad config"
> >>>> always means TRY, which I think it doesn't as the drivers have internally
> >>>> been using pad configs.
> >>>
> >>> If they do, they would have the 'active' pad_configs  allocated or
> >>> embedded somewhere in their driver structures, they would not receive
> >>> it as parameter. Or have I missed where the core is capable of fishing
> >>> the 'right' pad_configs ? I think the same should happen for state.
> >>
> >> Yes, that is correct. I was just saying that the documentation doesn't
> >> say that init_cfg is only used for the TRY case, even if that has been
> >> the case in practice.
> >>
> >>>> But it's true that so far init_cfg has only been called for TRY case, and
> >>>> perhaps that's enough of a reason to keep it so.
> >>>>
> >>>>>> I would rather just embed 'struct v4l2_subdev_state' in 'struct
> >>>>>> v4l2_subdev', have the core going through the
> >>>>>
> >>>>> Why would embedding the state change anything?
> >>>>>
> >>>>>> 'v4l2_subdev_alloc_state()' to initialize the per-fh state, but have
> >>>>>> drivers initialize their own state at probe time. If they need for
> >>>>>> some reason to access their 'active' state at init_cfg() time, they
> >>>>>> caan fish it from their subdev.
> >>>>>>
> >>>>>> If I'm not mistaken this will remove the need to have a which filed in
> >>>>>> the state, as I think the 'context' should be inferred from the
> >>>>>> 'which' argument embedded in the ops-specific structures, and not held
> >>>>>> in the state itself.
> >>>>
> >>>> It's true that the state's which field is mainly (probably only) needed for
> >>>> handling the init_cfg. It could be solved in other ways too:
> >>>>
> >>>> - New subdev op to initialize active state
> >>>> - New subdev op which gets 'which' as a parameter, to initialize both states
> >>>> (state-aware drivers wouldn't need to implement the old init_cfg)
> >>>> - Coccinelle to change init_cfg to get the which as a parameter
> >>>>
> >>>> Without doing any deep thinking, the middle one sounds best to me.
> >>>
> >>> Isn't it simpler if you just don't call init_cfg for the 'active'
> >>> state ? Driver will initialize them at probe time and that's it, then
> >>> you can remove 'which' from the state (and from routing tables too if I'm
> >>> not mistaken).
> >>
> >> Routing needs the 'which' similarly to other config structs, like
> >> v4l2_subdev_format.
> >>
> >> I did a quick implementation for the second case, which allows me to
> >> remove 'which' from the state.
> >>
> >> As for "simpler"... Both ways have pros and cons. I'm my mind this new
> >> way is simpler.
> >>
> >> Afaics, in the end (i.e. after doing the v4l2_subdev_call change), the
> >> only bigger difference is how the state-aware drivers implement their
> >> ops. With my approach, they get the correct state and that's it. With
> >> your approach, they need to use a helper function (or do it manually) to
> >> get the state based on 'which'.
> > 
> > I'd like to propose a third approach (or at least I believe it's a third
> > one). I agree with Jacopo that the state structure should not have a
> > which field, that's a layering violation. The state is a state,
> > regardless of whether it holds TRY or ACTIVE data. What are the current
> > blockers that would prevent that ?
> 
> Oh, I agree with that too. I didn't add the 'which' field to state 
> because I thought it's good =). It is supposed to be temporary solution. 
> init_cfg() is the issue here.
> 
> It think I had these options:
> 
> - Change init_cfg() to take 'which' as a parameter
> - Change init_cfg() implementations to not use 'which'
> - Add new version for new drivers, init_cfg_state() or such, which 
> either gets the which as a parameter or doesn't use which.
> - Add 'which' to state.
> 
> I chose the fourth one as it's a very small change, and can be removed 
> easily in the future when the underlying problem is solved.
> 
> > However, I think .init_cfg() should be used to initialize *all* states,
> > both TRY and ACTIVE. That will simplify driver implementation (but
> > centralizing the initialization in a single place) and ensure that the
> > default value for the ACTIVE state always matches the default value for
> > the TRY state, which I think is important.
> 
> I agree.

That's the second option from you list above, right ?

As ACTIVE state support is opt-in, it seems to me that we won't need to
mass-fix drivers as part of this series if we go for this option. Am I
missing something ?

> > When it comes to calling v4l2_subdev_alloc_state() in drivers, I also
> > agree with Jacopo, I'm not extremely fond of it. This should be
> > automatic, drivers shouldn't have to care about allocating the ACTIVE
> > state. However, I also agree with Tomi that there's no real place to do
> > this today. I think we need to restructure subdev initialization, which
> > currently has very few rules. This could take the form, for instance, of
> 
> I agree with your agreeings above too.
> 
> > 1. calling v4l2_subdev_init() as an early subdev init function
> > 2. setting fields of v4l2_subdev manually
> > 3. initializing the media_entity embedded in the subdev
> > 4. calling a new "initialization finalization" function that will
> >     initialize the state embedded in v4l2_subdev (as opposed to
> >     allocating it dynamically as done today)
> 
> Why is the allocated/embedded relevant? The drivers won't know which one 
> is used.

Embedding will lower the number of allocations. On the other hand, as
replied to another patch in this series, allocating the state
dynamically may be better short term, so I'm OK with that.

> > That's the simplest option as it really ressembles what we do today, and
> > would more or less only require embedded the state in v4l2_subdev and
> > renaming v4l2_subdev_alloc_state() to v4l2_subdev_init_done() (or
> > similar, that's likely not a great name).
> 
> Yes. But I don't see embedding relevant. So we could just rename 
> v4l2_subdev_alloc_state() and we should be there.
> 
> > A more intrusive change would restructure the subdev initialization
> > further, by handling the media_entity initialization transparently
> > (which would require passing more arguments to v4l2_subdev_init(),
> > probably creating a new version of that function). That's more complex
> > and I think it's out of scope for this series, but it may still be
> > useful to keep the long term goal in mind to try and align with the
> > direction.
> 
> Yes, there are many things that can be improved further. But in the 
> context of this series, I'd like to keep the improvements such that they 
> require no changes to the existing drivers.

I'd propose minmizing the need of such changes, but not ruling them out
completely in case the gain is worth the pain. Of course it largely
depends who gets the gain and who's inflicted the pain ;-)

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 05/36] media: subdev: add subdev state locking

[Tomi Valkeinen]

On 27/09/2021 04:35, Laurent Pinchart wrote:
> Hi Tomi,
> 
> Thank you for the patch.
> 
> On Mon, Aug 30, 2021 at 02:00:45PM +0300, Tomi Valkeinen wrote:
>> The V4L2 subdevs have managed without centralized locking for the state
>> (previously pad_config), as the TRY state is supposedly safe (although I
>> believe two TRY ioctls for the same fd would race), and the ACTIVE
>> state, and its locking, is managed by the drivers internally.
>>
>> We now have ACTIVE state in a centralized position, and need locking.
>> Strictly speaking the locking is only needed for new drivers that use
>> the new state, as the current drivers continue behaving as they used to.
>>
>> Add a mutex to the struct v4l2_subdev_state, along with a few helper
>> functions for locking/unlocking.
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>>   drivers/media/v4l2-core/v4l2-subdev.c | 43 +++++++++++++++++----
>>   include/media/v4l2-subdev.h           | 55 +++++++++++++++++++++++++--
>>   2 files changed, 88 insertions(+), 10 deletions(-)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>> index b3637cddca58..b1e65488210d 100644
>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>> @@ -26,9 +26,11 @@
>>   #if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
>>   static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
>>   {
>> +	static struct lock_class_key __key;
>>   	struct v4l2_subdev_state *state;
>>   
>> -	state = v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_TRY);
>> +	state = __v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_TRY,
>> +					  "v4l2_subdev_fh->state", &__key);
> 
> What's the reason for not using the v4l2_alloc_subdev_state() macro here
> ?

It has a different name for the lock. I'm not sure if that's really 
needed or not, as v4l2_alloc_subdev_state anyway adds a filename and 
line-number.

I guess one reason is that at some point v4l2_alloc_subdev_state() was 
supposed to be only for allocating active configuration. Which is 
actually what the lock name there refers to "sd->state->lock".

>>   	if (IS_ERR(state))
>>   		return PTR_ERR(state);
>>   
>> @@ -924,8 +926,10 @@ int v4l2_subdev_link_validate(struct media_link *link)
>>   EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
>>   
>>   struct v4l2_subdev_state *
>> -v4l2_alloc_subdev_state(struct v4l2_subdev *sd,
>> -			enum v4l2_subdev_format_whence which)
>> +__v4l2_alloc_subdev_state(struct v4l2_subdev *sd,
>> +			  enum v4l2_subdev_format_whence which,
>> +			  const char *lock_name,
>> +			  struct lock_class_key *lock_key)
>>   {
>>   	struct v4l2_subdev_state *state;
>>   	int ret;
>> @@ -934,6 +938,8 @@ v4l2_alloc_subdev_state(struct v4l2_subdev *sd,
>>   	if (!state)
>>   		return ERR_PTR(-ENOMEM);
>>   
>> +	__mutex_init(&state->lock, lock_name, lock_key);
>> +
>>   	state->which = which;
>>   
>>   	if (sd->entity.num_pads) {
>> @@ -960,13 +966,15 @@ v4l2_alloc_subdev_state(struct v4l2_subdev *sd,
>>   
>>   	return ERR_PTR(ret);
>>   }
>> -EXPORT_SYMBOL_GPL(v4l2_alloc_subdev_state);
>> +EXPORT_SYMBOL_GPL(__v4l2_alloc_subdev_state);
>>   
>>   void v4l2_free_subdev_state(struct v4l2_subdev_state *state)
>>   {
>>   	if (!state)
>>   		return;
>>   
>> +	mutex_destroy(&state->lock);
>> +
>>   	kvfree(state->pads);
>>   	kfree(state);
>>   }
>> @@ -1001,11 +1009,12 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
>>   }
>>   EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
>>   
>> -int v4l2_subdev_alloc_state(struct v4l2_subdev *sd)
>> +int __v4l2_subdev_alloc_state(struct v4l2_subdev *sd, const char *name,
>> +			      struct lock_class_key *key)
>>   {
>>   	struct v4l2_subdev_state *state;
>>   
>> -	state = v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_ACTIVE);
>> +	state = __v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_ACTIVE, name, key);
> 
> I already know that Sakari will ask for a line wrap at 80 columns, and
> that would be my preference as well :-) I won't repeat the comment in
> the rest of the series. Going over 80 columns is fine when it improves
> readability, but in many places keeping lines short enough would be
> nicer.

Hmm... I think I went over all the 80+ lines at some point, but this 
change probably came afterwards. I'll go through them again.

A bit of a side topic, but an annoying think about complying to 80 
columns is that my editor indents with tabs and aligns with spaces. And 
you (and I guess Sakar) want both be done with tabs, and with only a 
minimal amount of spaces. I personally like my editor's behavior better, 
though, as it works fine when changing tab widths too.

I can use clang-format to do the indentation instead, but it has the 
super annoying feature that it likes to do this:

v4l2_subdev_function_name(
	sd, bar, long_argument_here);

>>   	if (IS_ERR(state))
>>   		return PTR_ERR(state);
>>   
>> @@ -1013,7 +1022,7 @@ int v4l2_subdev_alloc_state(struct v4l2_subdev *sd)
>>   
>>   	return 0;
>>   }
>> -EXPORT_SYMBOL_GPL(v4l2_subdev_alloc_state);
>> +EXPORT_SYMBOL_GPL(__v4l2_subdev_alloc_state);
>>   
>>   void v4l2_subdev_free_state(struct v4l2_subdev *sd)
>>   {
>> @@ -1021,3 +1030,23 @@ void v4l2_subdev_free_state(struct v4l2_subdev *sd)
>>   	sd->state = NULL;
>>   }
>>   EXPORT_SYMBOL_GPL(v4l2_subdev_free_state);
>> +
>> +struct v4l2_subdev_state *v4l2_subdev_lock_active_state(struct v4l2_subdev *sd)
>> +{
>> +	mutex_lock(&sd->state->lock);
>> +
>> +	return sd->state;
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_subdev_lock_active_state);
>> +
>> +void v4l2_subdev_lock_state(struct v4l2_subdev_state *state)
>> +{
>> +	mutex_lock(&state->lock);
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_subdev_lock_state);
>> +
>> +void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state)
>> +{
>> +	mutex_unlock(&state->lock);
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_subdev_unlock_state);
>> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
>> index 5ec78ffda4f5..52a725281b23 100644
>> --- a/include/media/v4l2-subdev.h
>> +++ b/include/media/v4l2-subdev.h
>> @@ -655,6 +655,7 @@ struct v4l2_subdev_pad_config {
>>   /**
>>    * struct v4l2_subdev_state - Used for storing subdev state information.
>>    *
>> + * @lock: mutex for the state
>>    * @which: state type (from enum v4l2_subdev_format_whence)
>>    * @pads: &struct v4l2_subdev_pad_config array
>>    *
>> @@ -663,6 +664,7 @@ struct v4l2_subdev_pad_config {
>>    * %V4L2_SUBDEV_FORMAT_ACTIVE it is safe to pass %NULL.
>>    */
>>   struct v4l2_subdev_state {
>> +	struct mutex lock;
>>   	u32 which;
>>   	struct v4l2_subdev_pad_config *pads;
>>   };
>> @@ -1147,9 +1149,18 @@ int v4l2_subdev_link_validate(struct media_link *link);
>>    *
>>    * Must call v4l2_free_subdev_state() when state is no longer needed.
>>    */
>> +#define v4l2_alloc_subdev_state(sd, which)                                     \
>> +	({                                                                     \
>> +		static struct lock_class_key __key;                            \
>> +		const char *name = KBUILD_BASENAME                             \
>> +			":" __stringify(__LINE__) ":sd->state->lock";          \
>> +		__v4l2_alloc_subdev_state(sd, which, name, &__key);            \
>> +	})
>> +
>>   struct v4l2_subdev_state *
>> -v4l2_alloc_subdev_state(struct v4l2_subdev *sd,
>> -			enum v4l2_subdev_format_whence which);
>> +__v4l2_alloc_subdev_state(struct v4l2_subdev *sd,
>> +			  enum v4l2_subdev_format_whence which,
>> +			  const char *lock_name, struct lock_class_key *key);
>>   
>>   /**
>>    * v4l2_free_subdev_state - free a v4l2_subdev_state
>> @@ -1234,7 +1245,16 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
>>    *
>>    * Must call v4l2_subdev_free_state() when the state is no longer needed.
>>    */
>> -int v4l2_subdev_alloc_state(struct v4l2_subdev *sd);
>> +#define v4l2_subdev_alloc_state(sd)                                            \
>> +	({                                                                     \
>> +		static struct lock_class_key __key;                            \
>> +		const char *name = KBUILD_BASENAME                             \
>> +			":" __stringify(__LINE__) ":sd->state->lock";          \
>> +		__v4l2_subdev_alloc_state(sd, name, &__key);                   \
>> +	})
>> +
>> +int __v4l2_subdev_alloc_state(struct v4l2_subdev *sd, const char *name,
>> +			      struct lock_class_key *key);
>>   
>>   /**
>>    * v4l2_subdev_free_state() - Free the active subdev state for subdevice
>> @@ -1258,4 +1278,33 @@ v4l2_subdev_get_active_state(struct v4l2_subdev *sd)
>>   	return sd->state;
>>   }
>>   
>> +/**
>> + * v4l2_subdev_lock_active_state() - Lock and return the active subdev state for subdevice
>> + * @sd: The subdevice
>> + *
>> + * Return the locked active state for the subdevice, or NULL if the subdev
>> + * does not support active state.
>> + *
>> + * Must be unlocked with v4l2_subdev_unlock_state() after use.
>> + */
>> +struct v4l2_subdev_state *v4l2_subdev_lock_active_state(struct v4l2_subdev *sd);
>> +
>> +/**
>> + * v4l2_subdev_lock_state() - Lock the subdev state
>> + * @state: The subdevice state
>> + *
>> + * Lock the given subdev state.
>> + *
>> + * Must be unlocked with v4l2_subdev_unlock_state() after use.
>> + */
>> +void v4l2_subdev_lock_state(struct v4l2_subdev_state *state);
> 
> This seems to be used only to lock the state passed to the subdev
> operation by the caller. Could the caller lock the state instead ? This
> could possibly be done by wrapping the v4l2_subdev_call() calls in
> dedicated helper functions.

Maybe, but it's easy to get into problems that way. One of the problems 
is that subdev drivers already have locks for other things (say, 
xyz-lock), and the locking order between state-lock and xyz-lock has to 
be kept the same. And as we have subdev ops that don't get a state 
(mainly s_stream), those have to do the locking themselves, and also 
keep the order the same.

I was hitting lockdep issues constantly when the v4l2-core was taking 
the lock before calling the ops. I did sort some of those out, but 
sorting some of those out cleanly wasn't trivial, and it felt like 
swimming against the current. So I instead decided to go this way.

It might also introduce deadlocks in drivers. I don't have a real 
example, but I'm not sure if it's too far fetched to imagine a case with 
two subdev drivers, part of a single driver module, where, say, subdev A 
gets a set_fmt call, it then calls set_fmt in subdev B, and B calls 
get_fmt in subdev A. Yes, it's a contrived example, but are you sure 
things like that are not done? =)

I'm sure this can be sorted out, but I have a gut feeling it won't be easy.

  Tomi

Re: [PATCH v8 03/36] media: subdev: add 'which' to subdev state

[Laurent Pinchart]

Hi Tomi,

On Mon, Sep 27, 2021 at 11:35:53AM +0300, Tomi Valkeinen wrote:
> On 27/09/2021 03:46, Laurent Pinchart wrote:
> > On Thu, Sep 16, 2021 at 03:07:52PM +0200, Jacopo Mondi wrote:
> >> On Mon, Sep 13, 2021 at 05:26:45PM +0300, Tomi Valkeinen wrote:
> >>> On 13/09/2021 16:38, Jacopo Mondi wrote:
> >>>> On Mon, Sep 13, 2021 at 03:17:01PM +0300, Tomi Valkeinen wrote:
> >>>>> On 13/09/2021 14:41, Jacopo Mondi wrote:
> >>>>>> On Mon, Aug 30, 2021 at 02:00:43PM +0300, Tomi Valkeinen wrote:
> >>>>>>> The subdev state is passed to functions in the media drivers, and
> >>>>>>> usually either V4L2_SUBDEV_FORMAT_ACTIVE or V4L2_SUBDEV_FORMAT_TRY is
> >>>>>>> also given to the function in one way or another.
> >>>>>>>
> >>>>>>> One op where this is not the case is v4l2_subdev_pad_ops.init_cfg. One
> >>>>>>> could argue that the initialization of the state should be the same for
> >>>>>>> both ACTIVE and TRY cases, but unfortunately that is not the case:
> >>>>>>>
> >>>>>>> - Some drivers do also other things than just touch the state when
> >>>>>>> dealing with ACTIVE, e.g. if there is extra state outside the standard
> >>>>>>> subdev state.
> >>>>>>> - Some drivers might need to create, say, struct v4l2_subdev_format
> >>>>>>> which has 'which' field, and that needs to be filled with either ACTIVE
> >>>>>>> or TRY.
> >>>>>>>
> >>>>>>> Currently init_cfg is only called for TRY case from the v4l2 framework,
> >>>>>>> passing the TRY state. Some drivers call their own init_cfg, passing
> >>>>>>> NULL as the state, which is used to indicate ACTIVE case.
> >>>>>>>
> >>>>>>> In the future we want to pass subdev's active state from the v4l2
> >>>>>>> framework side, so we need a solution to this.
> >>>>>>>
> >>>>>>> We could change the init_cfg() to include the TRY/ACTIVE value, which
> >>>>>>> would require changing more or less all the drivers. Instead, I have
> >>>>>>> added 'which' field to the subdev state itself, filled at state
> >>>>>>> allocation time, which only requires changes to the drivers that
> >>>>>>> allocate a state themselves.
> >>>>>>>
> >>>>>>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> >>>>>>> ---
> >>>>>>>     drivers/media/platform/rcar-vin/rcar-v4l2.c |  2 +-
> >>>>>>>     drivers/media/platform/vsp1/vsp1_entity.c   |  2 +-
> >>>>>>>     drivers/media/v4l2-core/v4l2-subdev.c       | 10 +++++++---
> >>>>>>>     drivers/staging/media/tegra-video/vi.c      |  2 +-
> >>>>>>>     include/media/v4l2-subdev.h                 |  7 ++++++-
> >>>>>>>     5 files changed, 16 insertions(+), 7 deletions(-)
> >>>>>>>
> >>>>>>> diff --git a/drivers/media/platform/rcar-vin/rcar-v4l2.c b/drivers/media/platform/rcar-vin/rcar-v4l2.c
> >>>>>>> index 5f4fa8c48f68..1de30d5b437f 100644
> >>>>>>> --- a/drivers/media/platform/rcar-vin/rcar-v4l2.c
> >>>>>>> +++ b/drivers/media/platform/rcar-vin/rcar-v4l2.c
> >>>>>>> @@ -252,7 +252,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
> >>>>>>>     	u32 width, height;
> >>>>>>>     	int ret;
> >>>>>>>
> >>>>>>> -	sd_state = v4l2_alloc_subdev_state(sd);
> >>>>>>> +	sd_state = v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_ACTIVE);
> >>>>>>
> >>>>>> Shouldn't the 'which' parameters be used to decide if either ACTIVE or
> >>>>>> TRY have to be used ? this function is also used to set TRY formats,
> >>>>>> in example...
> >>>>>>
> >>>>>> Oh, maybe I got how it works, the state's which is not
> >>>>>> relevant but the v4l2_subdev_format's which is, as it will be used in
> >>>>>> the next patch to decide if the subdev's state of the file-handle's
> >>>>>> state should be passed to the ioctl.
> >>>>>
> >>>>> Yes. It's messy, but it's how it worked before also.
> >>>>>
> >>>>> The drivers can't really allocate TRY state as it must come from the core,
> >>>>> based on the filehandle. Now as I say that, makes me wonder why even expose
> >>>>> the option to drivers. Maybe v4l2_alloc_subdev_state() should take just the
> >>>>> sd parameter, and always allocate ACTIVE state, and the v4l2 core can use
> >>>>> another way to create the TRY state.
> > 
> > Drivers should not allocate state manually, it should always be done by
> > the core (when opening a file handle for the TRY states, and when
> > initializing the subdev in the probe function for the ACTIVE state).
> > 
> > Looking at the three drivers that call v4l2_alloc_subdev_state(), we
> > have the vsp1 driver that uses it to allocate its own ACTIVE state at
> > init time. This one is easy, it should be replaced with
> > v4l2_subdev_alloc_state() (or whatever that function will end up being
> > named, see comments to 02/36).
> 
> I don't follow here. First you say that drivers should not allocate 
> state. But then, I think, you say that vsp1 allocating state is fine 
> after some change?

I mean that the vsp1 driver uses the function for its intended purpose,
but this should be moved to core-allocated state (with
v4l2_subdev_init_done() or however we decide to call it). The purpose
matches, so it should be easy. The other two drivers allocate ACTIVE
states for different purposes, which I don't think is right.

> My understanding is that vsp1 is fine. Of course, if the function it 
> calls gets renamed we need to rename it in vsp1 too.
> 
> > The rcar-vin and tegra-video drivers are doing it wrong. rcar-vin has
> > legacy code to support unfortunate design decisions (everything in the
> > "V4L2" section in rcar-v4l2.c), and newer code that gets it right (the
> > "V4L2 Media Controller" section in the same file). The legacy code is
> > used on older platforms, and the newer code on newer platforms. I know
> > that Niklas would love to get rid of the legacy code, and I'd also be
> > happy to see it go. If that's not possible, we'll have to keep exposing
> > v4l2_alloc_subdev_state() for this driver. Niklas, what do you think,
> > could we drop the legacy code after all those years ?
> > 
> > The situation in tegra-video is similar. They got it wrong. The driver
> > is in staging though, so that's fixable.
> > 
> > I'd propose renaming v4l2_subdev_alloc_state() to
> > __v4l2_subdev_alloc_state() (or something that shows it's internal),
> > documenting that it must not be used by new drivers, and adding an entry
> > in the TODO file of the tegra-video driver to fix this.
> 
> Yes, that sounds fine. Subdev drivers allocating the state is not 
> something that should be done in any new drivers.
> 
> >>>> init_cfg() as well as other operations used to received an
> >>>> array of per fh's pad_configs, and the sd pointer itself. The fh's pad
> >>>> configs were allocated by the core, as well as the core now allocates
> >>>> the per-fh's state.
> >>>>
> >>>> Before the introduction of 'state', if the 'which' flags was set to
> >>>> TRY then information were stored/retrieved/initialized in the per-fh
> >>>> pad_config, otherwise the active configuration (usually stored in the
> >>>> driver main structure) was used.
> > 
> > Correct, and the active configuration is in that case stored in a
> > driver-specific way (except in the vsp1 driver that uses the pad config
> > structure to store it), with ad-hoc accessors and lots of manual checks
> > in the code paths.
> > 
> >>>> So we had a clear separation of per-fh information and the active
> >>>> state. The core didn't care afaict, and passed in both, then driver had
> >>>> to deal with them doing the right thing by inspecting the 'which' flag.
> >>>>
> >>>> The typical pattern was:
> >>>>
> >>>>           static int subdev_ops(sd, pad_cfg, which)
> >>>>           {
> >>>>                   if (which == TRY)
> >>>>                           /* Operate on config stored in pad_cfg */
> >>>>
> >>>>                   else
> >>>>                           /*
> >>>>                            * Operate on config stored in subdev (and
> >>>>                            * applied to HW)
> >>>>                            */
> >>>>           }
> >>>>
> >>>> Or am I overlooking some cases or do you agree with my understanding
> >>>> so far ?
> >>>
> >>> More or less, yes. I think there are (used to be) three kinds of ops:
> >>>
> >>> - Ops that get pad_cfg and 'which' in an op specific struct. E.g. set_fmt.
> >>> The pad_cfg is TRY pad_config, even if 'which' == ACTIVE.
> > 
> > And pad_cfg is ignored in those drivers when which == ACTIVE (or it
> > should be at least, if it's not, it's a driver bug).
> > 
> >>> - Ops that don't get pad_cfg, like s_stream. 'which' is implicitly ACTIVE.
> > 
> > .s_stream() on a TRY configuration would be an interesting concept :-)
> 
> I think it would be a good one, though =). It could be used to validate 
> the whole pipeline. But it would probably be rather difficult to 
> implement with the current v4l2 framework.
> 
> >> Also note that operations like s_stream do not take a state as
> >> parameter. The driver has to fetch it from the subdev anyway
> >> (this in reply to the idea of having the active state as parameter vs
> >> retrieving it from the subdev if ACTIVE)
> > 
> > We could pass the state as a parameter, but given that the operation
> > always operates on the ACTIVE state by definition, I think this is
> > redundant.
> > 
> >> While porting the R-Car drivers on top of this series I found myself
> >> in the need to (in the s_stream call chain)
> >>
> >> static int rcsi2_start_receiver(struct rcar_csi2 *priv)
> >> {
> >> 	const struct v4l2_subdev_state *state = priv->subdev.state;
> >> 	const struct v4l2_subdev_stream_configs *configs = &state->stream_configs;
> >>
> >>          ...
> >>
> >> 	/*
> >> 	 * Configure field handling inspecting the formats of the
> >> 	 * single sink pad streams.
> >> 	 */
> >> 	for (i = 0; i < configs->num_configs; ++i) {
> >> 		const struct v4l2_subdev_stream_config *config = configs->configs;
> >> 		if (config->pad != RCAR_CSI2_SINK)
> >> 			continue;
> >>
> >> 		if (config->fmt.field != V4L2_FIELD_ALTERNATE)
> >> 			continue;
> >>
> >> 		fld |= FLD_DET_SEL(1);
> >> 		fld |= FLD_FLD_EN(config->stream);
> >>
> >> 		/* PAL vs NTSC. */
> >> 		if (config->fmt.height == 240)
> >> 			fld |= FLD_FLD_NUM(0);
> >> 		else
> >> 			fld |= FLD_FLD_NUM(1);
> >> 	}
> >>
> >>          ...
> >>
> >> }
> >>
> >> Am I doing it wrong, or is this a case for the subdev to have to
> >> directly access sd->state ?
> > 
> > (Will reply to Tomi's reply on this)
> > 
> >>> - init_cfg which gets pad_cfg, but no which (as 'which' is always implicitly
> >>> TRY)
> > 
> > As proposed in 02/36, I think .init_cfg() should operate the same way
> > regardless of whether the state is TRY or ACTIVE, so there's no need for
> > a 'which' parameter (it would only open the door to abuses).
> > 
> >>> So pad_cfg was TRY state. Drivers could use pad_configs internally to track
> >>> ACTIVE state, but the core had no knowledge about this.
> > 
> > Correct.
> > 
> >>>> Now we have a 'state' that holds the array of pad_configs and along
> >>>> the continuation of the series will end up holding per-pad
> >>>> configurations.
> >>>>
> >>>> We now also have one 'state' per file-handle, and one
> >>>> per-subdev. As I see this, it would be natual for drivers to receive
> >>>> one state without knowing where it comes from. In the next patch you
> >>>
> >>> Note that only subdev's that explicitly support the new state code, and
> >>> allocate the state, have the subdev active state. Which means only the
> >>> drivers in my work branch.
> >>>
> >>> The "old" drivers work like they used to: they get the state (essentially
> >>> repackaged pad_cfg) for TRY cases, NULL otherwise.
> >>>
> >>> And yes, it would be natural to just get a state, but the subdev drivers
> >>> need to know if the context is TRY/ACTIVE. As you can see from the bullet
> >>> list above, the driver knows this in all the other places except init_cfg.
> > 
> > I agree that porting all the drivers as part of this series isn't
> > feasible, so passing NULL for the state in case the driver hasn't
> > explicitly opted-in seems fine to me.
> > 
> >>>> instrument the core to do exactly this: inspect the which flag and
> >>>> pass in the 'right' state. Ofc drivers need to have access to 'which'
> >>>> to know if they have to apply settings to the HW or not.
> >>>>
> >>>> Looking ahead in your series I see these structures:
> >>>>
> >>>>           struct v4l2_subdev_pad_config {
> >>>>                   struct v4l2_mbus_framefmt try_fmt;
> >>>>                   struct v4l2_rect try_crop;
> >>>>                   struct v4l2_rect try_compose;
> >>>>           };
> >>>>
> >>>>           struct v4l2_subdev_stream_config {
> >>>>                   u32 pad;
> >>>>                   u32 stream;
> >>>>
> >>>>                   struct v4l2_mbus_framefmt fmt;
> >>>>                   struct v4l2_rect crop;
> >>>>                   struct v4l2_rect compose;
> >>>>           };
> >>>>
> >>>>           struct v4l2_subdev_stream_configs {
> >>>>                   u32 num_configs;
> >>>>                   struct v4l2_subdev_stream_config *configs;
> >>>>           };
> >>>>
> >>>> All of them part of state:
> >>>>
> >>>> struct v4l2_subdev_state {
> >>>> 	struct mutex lock;
> >>>> 	u32 which;
> >>>> 	struct v4l2_subdev_pad_config *pads;
> >>>> 	struct v4l2_subdev_krouting routing;
> >>>> 	struct v4l2_subdev_stream_configs stream_configs;
> >>>> };
> >>>>
> >>>> So 'state' will hold 'TRY' information (only used for 'state'
> >>>> instances allocated in the fh) and 'ACTIVE' ones (used for states
> >>>> allocated in the sd).
> >>>
> >>> Right.
> >>>
> >>>> Looking at 'v4l2_subdev_pad_config' and 'v4l2_subdev_stream_config' they
> >>>> seem to describe more or less the same things: fmt, crop and compose
> >>>> (per pad-stream in case of stream_config). I wonder if those shouldn't
> >>>> be unified so that:
> >>>>
> >>>> 1) Drivers receive one state: the core passes in the 'correct' one
> >>>> (per-fh or per-sd) as you do in next patch
> >>>
> >>> Yes. But note that "old" drivers don't have active state.
> >>>
> >>>> 2) The 'which' information is not stored in the state but it's only
> >>>> 'contextual' (as in a parameter to the subdev operation) so that
> >>>> drivers inspect it to know if they have to apply settings to hw or not
> >>>
> >>> Yes, except we have init_cfg...
> > 
> > This sounds like .init_cfg() is the only blocker, which I think is
> > promissing :-)
> > 
> >>>> 3) v4l2_subdev_pad_config can be re-used and expanded, to maintain per-pad
> >>>> configurations regardless if they're ACTIVE or TRY, as this only depends
> >>>> on where the state is stored.
> >>>
> >>> pad_config is a static array of per-pad configs. stream_configs is a dynamic
> >>> per-stream config.
> > 
> > Do I understand correctly that it's both per-stream and per-pad ? I
> > wonder if it could make sense to store it that way, with an array of
> > per-pad configurations (v4l2_subdev_pad_config), which will in turn
> > contain per-stream configuration. I suppose it depends on the usage
> > patterns, which I'll understand better when reading the rest of the
> > series.
> 
> I'm not sure what you mean with "both per-stream and per-pad". It's per 
> (stream, pad) tuple.

That's what I meant, not just a configuration for each stream that
traverses the subdev, but one for each pad that carries the stream, for
each stream.

> At the moment the internal storage is essentially just an array of
> { pad, stream, config } items. It could also be changed as you suggest, 
> but afaics that would require more dynamic allocations. The APIs should 
> be designed so that the drivers don't need to care about the internal 
> storage.

Agreed, the less drivers have to care about the storage, the better.

> >>> stream_configs is a super-set of pad-configs, so we could
> >>> drop pad_configs, but it would require changing all the drivers in
> >>> non-trivial ways.
> > 
> > I like Jacopo's proposal to unify the two, but I do agree that it's not
> > trivial. Here again I think it falls in the category of setting a long
> > term goal and trying to go in that direction, without necessarily
> > reaching it just yet. I wonder if we could have helper functions (maybe
> > they're already included later in the series ?) to abstract the
> > pad_config/stream_configs difference for drivers. This would also make
> > further reworks easier (such as storing the per-stream configuration in
> > v4l2_subdev_pad_config as proposed above) if we decide to address some
> > of the changes later.
> 
> If you mean do we have functions to hide the stream (i.e. stream = 0), 
> no, I haven't added such. I have not worked towards making 
> non-multiplexed-streams subdev drivers use the new APIs.

I meant helpers to access the configuration without having to care in
subdev drivers about how it's actually stored.

> That said, I have tried to design this series so that it is possible. 
> And "new" and "old" drivers should be compatible with each other.
> 
> >>> v4l2_subdev_pad_config is not used or even allocated by the "new" drivers.
> >>> And routing & stream_configs are not used by the "old" drivers.
> >>>
> >>>> As I immagine it a subdev pad operation could look like:
> >>>>
> >>>>           static int subdev_op(sd, pad, state, which, ...)
> >>>>           {
> >>>>                   /* Doesn't matter if state is per-fh or the sd one. */
> >>>>                   state->pads[pad].fmt = ....;
> >>>>
> >>>>                   if (which == TRY)
> >>>>                           return;
> >>>>
> >>>>                   /* Apply to the HW. */
> >>>>           }
> >>>>
> >>>> Does it make any sense to you ? I might have missed some reason why
> >>>> this is not possible.
> >>>
> >>> It makes sense, but there are the buts =). I've tried to explain these in
> >>> the commit messages, but it's kind of confusing.
> >>>
> >>> One but I haven't mentioned in the emails is that when subdev drivers call
> >>> ops in other subdev drivers they pass NULL in the state.
> > 
> > We should really really try to avoid that. s_stream() is the obvious
> > main exception. Other subdev operations, such as .get_fmt() or
> > .set_fmt(), shouldn't be called cross-subdevs. A subdev should only care
> > about its own configuration, not about its neighbours. Are there
> > blockers here ?
> 
> There are a lot of subdevs using v4l2_subdev_call(), including uses of 
> set_fmt.

It doesn't mean it's right though ;-) It's a bit different for video
node drivers, they have legitimate needs to call .get_fmt() to validate
the pipeline. They can also call .set_fmt() when they're not MC-based,
in which case those drivers effectively fulfil the role of the V4L2
core.

> >>> This is fine for
> >>> the "old" drivers, as they expect a state only for TRY case. However, the
> >>> "new" drivers unfortunately expect to get a state on both TRY and ACTIVE
> >>> cases, and the only sensible way I figured out to handle this was the
> >>> v4l2_subdev_validate_state() function (patch 6).
> >>>
> >>> So, all this could be much neater, but would require modifying all subdev
> >>> drivers in non-trivial ways. I think this is something that can be done
> >>> slowly in the future.
> >>>
> >>>>>>>     	if (IS_ERR(sd_state))
> >>>>>>>     		return PTR_ERR(sd_state);
> >>>>>>>
> >>>>>>> diff --git a/drivers/media/platform/vsp1/vsp1_entity.c b/drivers/media/platform/vsp1/vsp1_entity.c
> >>>>>>> index e40bca254b8b..63ea5e472c33 100644
> >>>>>>> --- a/drivers/media/platform/vsp1/vsp1_entity.c
> >>>>>>> +++ b/drivers/media/platform/vsp1/vsp1_entity.c
> >>>>>>> @@ -675,7 +675,7 @@ int vsp1_entity_init(struct vsp1_device *vsp1, struct vsp1_entity *entity,
> >>>>>>>     	 * Allocate the pad configuration to store formats and selection
> >>>>>>>     	 * rectangles.
> >>>>>>>     	 */
> >>>>>>> -	entity->config = v4l2_alloc_subdev_state(&entity->subdev);
> >>>>>>> +	entity->config = v4l2_alloc_subdev_state(&entity->subdev, V4L2_SUBDEV_FORMAT_ACTIVE);
> >>>>>>>     	if (IS_ERR(entity->config)) {
> >>>>>>>     		media_entity_cleanup(&entity->subdev.entity);
> >>>>>>>     		return PTR_ERR(entity->config);
> >>>>>>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> >>>>>>> index e1a794f69815..04ad319fb150 100644
> >>>>>>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> >>>>>>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> >>>>>>> @@ -28,7 +28,7 @@ static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
> >>>>>>>     {
> >>>>>>>     	struct v4l2_subdev_state *state;
> >>>>>>>
> >>>>>>> -	state = v4l2_alloc_subdev_state(sd);
> >>>>>>> +	state = v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_TRY);
> >>>>>>
> >>>>>> At the same time I'm not sure I get the purpose of this. Don't
> >>>>>> init_cfg() callback implementations deal with try formats themeselves
> >>>>>> ? I mean, it's not a fixed rule, they can as well initialize their
> >>>>>> default 'active' formats, but what matters is that they initialize
> >>>>>> their per-fh try states ?
> >>>>>
> >>>>> That is what they do currently. init_cfg() only deals with TRY state, as
> >>>>> that's the only "state" (i.e. pad_config) there used to be from v4l2 core's
> >>>>> perspective.
> >>>>>
> >>>>>> Shouldn't init_cfg receive the fh's state so that it can initialize
> >>>>>> it, and just in case they need to, access their subdev's state and
> >>>>>> initialize them ? I'm missing what the purpose of the flag is tbh.
> >>>>>
> >>>>> Now we have (a possibility to have) state for both TRY and ACTIVE on the
> >>>>> v4l2 core side. The active state has to be initialized also, and a logical
> >>>>> way to do that is to use the init_cfg().
> >>>>
> >>>> The 'ACTIVE' state is stored in the subdev, to which init_cfg() has
> >>>> access, and it receives the 'TRY' state as a parameter.
> >>>
> >>> No, init_cfg gets either ACTIVE or TRY state, whichever is being allocated.
> >>> For "old" drivers, ACTIVE state is never allocated so they don't get
> >>> init_cfg calls for ACTIVE at all.
> >>>
> >>> Aaand while writing that, I realized that some drivers manually do allocate
> >>> ACTIVE state temporarily, which would cause init_cfg with ACTIVE state to be
> >>> called. I wonder if they explode... Need to check.
> > 
> > Is this only rcar-vin and tegra-video (vsp1 also allocates an ACTIVE
> > state, but it's not temporary), or are there other drivers ?
> 
> It's those three drivers. And the above is not an issue, I didn't think 
> it to the end.
> 
> Those three drivers used to call v4l2_subdev_alloc_pad_config(), which 
> always calls init_cfg. So the situation is no different with this series.
> 
> >>>> It is possible to access both states and initialize them properly if
> >>>> I'm not mistaken.
> >>>>
> >>>>> So now, for drivers that support the new active state, init_cfg() can get
> >>>>> either TRY or ACTIVE state. And if you want to call, say, the driver's
> >>>>> set_routing() to setup the routing in the state, you have to set the 'which'
> >>>>> in the routing struct to a value. So somehow init_cfg needs to know if it's
> >>>>> initializing an ACTIVE or TRY state.
> >>>>
> >>>> I'm not sure I got this part. set_routing() as other ops will receive
> >>>> a state and 'which'. If my proposal above makes sensem where the state
> >>>
> >>> Yes, but if it's init_cfg calling set_routing, init_cfg has to figure out
> >>> the 'which' from somewhere.
> >>>
> >>> E.g. init_cfg from ub913 driver:
> >>>
> >>> static int ub913_init_cfg(struct v4l2_subdev *sd,
> >>> 			  struct v4l2_subdev_state *state)
> >>> {
> >>> 	u32 which = state->which;
> >>>
> >>> 	struct v4l2_subdev_route routes[] = {
> >>> 		{
> >>> 			.sink_pad = 0,
> >>> 			.sink_stream = 0,
> >>> 			.source_pad = 1,
> >>> 			.source_stream = 0,
> >>> 			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> >>> 		},
> >>> 	};
> >>>
> >>> 	struct v4l2_subdev_krouting routing = {
> >>> 		.which = which,
> >>> 		.num_routes = ARRAY_SIZE(routes),
> >>> 		.routes = routes,
> >>> 	};
> >>>
> >>> 	return ub913_set_routing(sd, state, &routing);
> >>> }
> >>>
> >>> It uses set_routing to setup a default routing (and set_routing in turn also
> >>> initializes the formats), but set_routing needs 'which'.
> > 
> > Could we avoid such patterns, by either initializing the routing
> > manually here, or splitting the logic of ub913_set_routing() in two
> > functions, with common code called by ub913_set_routing() and
> > ub913_init_cfg() ? I think calling set functions from .init_cfg() opens
> > the door to abuses or just bugs, and I increasingly think it should be
> > discouraged. I know I've written quite a few drivers that call
> > .set_fmt() from .init_cfg() though, it does make things easier to some
> > extent, but I wonder if we could do better.
> 
> It is possible, at least in theory.
> 
> The problem is that the 'which' is embedded into many structs used in 
> this context, e.g. struct v4l2_subdev_format and struct 
> v4l2_subdev_krouting. If we don't have 'which', none of the functions 
> using those structs is usable. But what's worse, those structs are used 
> in the subdev state. So even if we write new helper funcs that don't 
> need 'which', we still need it in the end when storing the data to the 
> subdev state.
>
> The annoying thing is that I don't think we really need the 'which' in 
> any of those functions or the state. Afaik the only real purpose of 
> 'which' is as a parameter when the userspace calls a subdev ioctl.
> 
> But getting rid of 'which' is again something I'd rather not start in 
> the context of this series, as I believe it will be a huge task in itself.

It's only v4l2_subdev_krouting that has a which field as far as I can
tell, the rest of the state uses v4l2_mbus_framefmt to store the format,
not v4l2_subdev_format. Given that v4l2_subdev_krouting is added by this
series, I'd like to try and address it.

> > This leads to the question of where to initialize the hardware state,
> > which is the part of the set functions that is only executed in the
> > ACTIVE case. I don't think this should be done at probe time in general,
> > but at .s_stream() time. Only drivers that allow changing formats or
> 
> Yes, I agree.
> 
> > routing while streaming would need to perform any hardware configutation
> > in the set functions, and I'm not even sure we have a single driver in
> > the kernel that allows this (or, to be precise, if we have a single
> > driver that knowingly allows this, I'm sure we have a bunch of drivers
> > that don't prevent this situation and will crash or behave in other bad
> > ways if userspace tried to change the configuration while streaming).
> > Maybe we should split .set_fmt() and .set_routing() in two for these
> > very uncommon cases, with the hardware configuration moved to separate
> > functions ? It has the potential to simplify the set operations and make
> > them safer.
> 
> Right, that's a good point: if the driver supports changing fmt while 
> streaming is enabled, set_fmt needs to have the 'which'. At the moment 
> they have it, via the struct v4l2_subdev_format. If we were to remove 
> the 'which' field (which would require separating the uAPI and kAPI 
> facing struct v4l2_subdev_format), we would either need to pass 'which' 
> some other way or have a new op, as you suggest.

.set_fmt() + .apply_fmt() is tempting. Or .atomic_check() and
.atomic_commit() ;-)

> I would put this to the "later" bucket. It certainly won't be a simple 
> change.

Agreed.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 05/36] media: subdev: add subdev state locking

[Laurent Pinchart]

Hi Tomi,

On Mon, Sep 27, 2021 at 12:49:35PM +0300, Tomi Valkeinen wrote:
> On 27/09/2021 04:35, Laurent Pinchart wrote:
> > On Mon, Aug 30, 2021 at 02:00:45PM +0300, Tomi Valkeinen wrote:
> >> The V4L2 subdevs have managed without centralized locking for the state
> >> (previously pad_config), as the TRY state is supposedly safe (although I
> >> believe two TRY ioctls for the same fd would race), and the ACTIVE
> >> state, and its locking, is managed by the drivers internally.
> >>
> >> We now have ACTIVE state in a centralized position, and need locking.
> >> Strictly speaking the locking is only needed for new drivers that use
> >> the new state, as the current drivers continue behaving as they used to.
> >>
> >> Add a mutex to the struct v4l2_subdev_state, along with a few helper
> >> functions for locking/unlocking.
> >>
> >> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> >> ---
> >>   drivers/media/v4l2-core/v4l2-subdev.c | 43 +++++++++++++++++----
> >>   include/media/v4l2-subdev.h           | 55 +++++++++++++++++++++++++--
> >>   2 files changed, 88 insertions(+), 10 deletions(-)
> >>
> >> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> >> index b3637cddca58..b1e65488210d 100644
> >> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> >> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> >> @@ -26,9 +26,11 @@
> >>   #if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
> >>   static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
> >>   {
> >> +	static struct lock_class_key __key;
> >>   	struct v4l2_subdev_state *state;
> >>   
> >> -	state = v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_TRY);
> >> +	state = __v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_TRY,
> >> +					  "v4l2_subdev_fh->state", &__key);
> > 
> > What's the reason for not using the v4l2_alloc_subdev_state() macro here
> > ?
> 
> It has a different name for the lock. I'm not sure if that's really 
> needed or not, as v4l2_alloc_subdev_state anyway adds a filename and 
> line-number.
> 
> I guess one reason is that at some point v4l2_alloc_subdev_state() was 
> supposed to be only for allocating active configuration. Which is 
> actually what the lock name there refers to "sd->state->lock".
> 
> >>   	if (IS_ERR(state))
> >>   		return PTR_ERR(state);
> >>   
> >> @@ -924,8 +926,10 @@ int v4l2_subdev_link_validate(struct media_link *link)
> >>   EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
> >>   
> >>   struct v4l2_subdev_state *
> >> -v4l2_alloc_subdev_state(struct v4l2_subdev *sd,
> >> -			enum v4l2_subdev_format_whence which)
> >> +__v4l2_alloc_subdev_state(struct v4l2_subdev *sd,
> >> +			  enum v4l2_subdev_format_whence which,
> >> +			  const char *lock_name,
> >> +			  struct lock_class_key *lock_key)
> >>   {
> >>   	struct v4l2_subdev_state *state;
> >>   	int ret;
> >> @@ -934,6 +938,8 @@ v4l2_alloc_subdev_state(struct v4l2_subdev *sd,
> >>   	if (!state)
> >>   		return ERR_PTR(-ENOMEM);
> >>   
> >> +	__mutex_init(&state->lock, lock_name, lock_key);
> >> +
> >>   	state->which = which;
> >>   
> >>   	if (sd->entity.num_pads) {
> >> @@ -960,13 +966,15 @@ v4l2_alloc_subdev_state(struct v4l2_subdev *sd,
> >>   
> >>   	return ERR_PTR(ret);
> >>   }
> >> -EXPORT_SYMBOL_GPL(v4l2_alloc_subdev_state);
> >> +EXPORT_SYMBOL_GPL(__v4l2_alloc_subdev_state);
> >>   
> >>   void v4l2_free_subdev_state(struct v4l2_subdev_state *state)
> >>   {
> >>   	if (!state)
> >>   		return;
> >>   
> >> +	mutex_destroy(&state->lock);
> >> +
> >>   	kvfree(state->pads);
> >>   	kfree(state);
> >>   }
> >> @@ -1001,11 +1009,12 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
> >>   }
> >>   EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
> >>   
> >> -int v4l2_subdev_alloc_state(struct v4l2_subdev *sd)
> >> +int __v4l2_subdev_alloc_state(struct v4l2_subdev *sd, const char *name,
> >> +			      struct lock_class_key *key)
> >>   {
> >>   	struct v4l2_subdev_state *state;
> >>   
> >> -	state = v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_ACTIVE);
> >> +	state = __v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_ACTIVE, name, key);
> > 
> > I already know that Sakari will ask for a line wrap at 80 columns, and
> > that would be my preference as well :-) I won't repeat the comment in
> > the rest of the series. Going over 80 columns is fine when it improves
> > readability, but in many places keeping lines short enough would be
> > nicer.
> 
> Hmm... I think I went over all the 80+ lines at some point, but this 
> change probably came afterwards. I'll go through them again.
> 
> A bit of a side topic, but an annoying think about complying to 80 
> columns is that my editor indents with tabs and aligns with spaces. And 
> you (and I guess Sakar) want both be done with tabs, and with only a 
> minimal amount of spaces. I personally like my editor's behavior better, 
> though, as it works fine when changing tab widths too.

The kernel coding style is to use tabs instead of spaces as much as
possible, and even if we don't wrap lines slightly over 80 columns,
there will always be lines that need to be wrapped, so I'm afraid it's
something that you need to handle in any case.

I don't know what editor you use (and I'm sure I don't want to know
:-)), but can't you instruct it to comply with the kernel coding style
by using tabs ?

> I can use clang-format to do the indentation instead, but it has the 
> super annoying feature that it likes to do this:
> 
> v4l2_subdev_function_name(
> 	sd, bar, long_argument_here);
> 
> >>   	if (IS_ERR(state))
> >>   		return PTR_ERR(state);
> >>   
> >> @@ -1013,7 +1022,7 @@ int v4l2_subdev_alloc_state(struct v4l2_subdev *sd)
> >>   
> >>   	return 0;
> >>   }
> >> -EXPORT_SYMBOL_GPL(v4l2_subdev_alloc_state);
> >> +EXPORT_SYMBOL_GPL(__v4l2_subdev_alloc_state);
> >>   
> >>   void v4l2_subdev_free_state(struct v4l2_subdev *sd)
> >>   {
> >> @@ -1021,3 +1030,23 @@ void v4l2_subdev_free_state(struct v4l2_subdev *sd)
> >>   	sd->state = NULL;
> >>   }
> >>   EXPORT_SYMBOL_GPL(v4l2_subdev_free_state);
> >> +
> >> +struct v4l2_subdev_state *v4l2_subdev_lock_active_state(struct v4l2_subdev *sd)
> >> +{
> >> +	mutex_lock(&sd->state->lock);
> >> +
> >> +	return sd->state;
> >> +}
> >> +EXPORT_SYMBOL_GPL(v4l2_subdev_lock_active_state);
> >> +
> >> +void v4l2_subdev_lock_state(struct v4l2_subdev_state *state)
> >> +{
> >> +	mutex_lock(&state->lock);
> >> +}
> >> +EXPORT_SYMBOL_GPL(v4l2_subdev_lock_state);
> >> +
> >> +void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state)
> >> +{
> >> +	mutex_unlock(&state->lock);
> >> +}
> >> +EXPORT_SYMBOL_GPL(v4l2_subdev_unlock_state);
> >> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> >> index 5ec78ffda4f5..52a725281b23 100644
> >> --- a/include/media/v4l2-subdev.h
> >> +++ b/include/media/v4l2-subdev.h
> >> @@ -655,6 +655,7 @@ struct v4l2_subdev_pad_config {
> >>   /**
> >>    * struct v4l2_subdev_state - Used for storing subdev state information.
> >>    *
> >> + * @lock: mutex for the state
> >>    * @which: state type (from enum v4l2_subdev_format_whence)
> >>    * @pads: &struct v4l2_subdev_pad_config array
> >>    *
> >> @@ -663,6 +664,7 @@ struct v4l2_subdev_pad_config {
> >>    * %V4L2_SUBDEV_FORMAT_ACTIVE it is safe to pass %NULL.
> >>    */
> >>   struct v4l2_subdev_state {
> >> +	struct mutex lock;
> >>   	u32 which;
> >>   	struct v4l2_subdev_pad_config *pads;
> >>   };
> >> @@ -1147,9 +1149,18 @@ int v4l2_subdev_link_validate(struct media_link *link);
> >>    *
> >>    * Must call v4l2_free_subdev_state() when state is no longer needed.
> >>    */
> >> +#define v4l2_alloc_subdev_state(sd, which)                                     \
> >> +	({                                                                     \
> >> +		static struct lock_class_key __key;                            \
> >> +		const char *name = KBUILD_BASENAME                             \
> >> +			":" __stringify(__LINE__) ":sd->state->lock";          \
> >> +		__v4l2_alloc_subdev_state(sd, which, name, &__key);            \
> >> +	})
> >> +
> >>   struct v4l2_subdev_state *
> >> -v4l2_alloc_subdev_state(struct v4l2_subdev *sd,
> >> -			enum v4l2_subdev_format_whence which);
> >> +__v4l2_alloc_subdev_state(struct v4l2_subdev *sd,
> >> +			  enum v4l2_subdev_format_whence which,
> >> +			  const char *lock_name, struct lock_class_key *key);
> >>   
> >>   /**
> >>    * v4l2_free_subdev_state - free a v4l2_subdev_state
> >> @@ -1234,7 +1245,16 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
> >>    *
> >>    * Must call v4l2_subdev_free_state() when the state is no longer needed.
> >>    */
> >> -int v4l2_subdev_alloc_state(struct v4l2_subdev *sd);
> >> +#define v4l2_subdev_alloc_state(sd)                                            \
> >> +	({                                                                     \
> >> +		static struct lock_class_key __key;                            \
> >> +		const char *name = KBUILD_BASENAME                             \
> >> +			":" __stringify(__LINE__) ":sd->state->lock";          \
> >> +		__v4l2_subdev_alloc_state(sd, name, &__key);                   \
> >> +	})
> >> +
> >> +int __v4l2_subdev_alloc_state(struct v4l2_subdev *sd, const char *name,
> >> +			      struct lock_class_key *key);
> >>   
> >>   /**
> >>    * v4l2_subdev_free_state() - Free the active subdev state for subdevice
> >> @@ -1258,4 +1278,33 @@ v4l2_subdev_get_active_state(struct v4l2_subdev *sd)
> >>   	return sd->state;
> >>   }
> >>   
> >> +/**
> >> + * v4l2_subdev_lock_active_state() - Lock and return the active subdev state for subdevice
> >> + * @sd: The subdevice
> >> + *
> >> + * Return the locked active state for the subdevice, or NULL if the subdev
> >> + * does not support active state.
> >> + *
> >> + * Must be unlocked with v4l2_subdev_unlock_state() after use.
> >> + */
> >> +struct v4l2_subdev_state *v4l2_subdev_lock_active_state(struct v4l2_subdev *sd);
> >> +
> >> +/**
> >> + * v4l2_subdev_lock_state() - Lock the subdev state
> >> + * @state: The subdevice state
> >> + *
> >> + * Lock the given subdev state.
> >> + *
> >> + * Must be unlocked with v4l2_subdev_unlock_state() after use.
> >> + */
> >> +void v4l2_subdev_lock_state(struct v4l2_subdev_state *state);
> > 
> > This seems to be used only to lock the state passed to the subdev
> > operation by the caller. Could the caller lock the state instead ? This
> > could possibly be done by wrapping the v4l2_subdev_call() calls in
> > dedicated helper functions.
> 
> Maybe, but it's easy to get into problems that way. One of the problems 
> is that subdev drivers already have locks for other things (say, 
> xyz-lock), and the locking order between state-lock and xyz-lock has to 
> be kept the same. And as we have subdev ops that don't get a state 
> (mainly s_stream), those have to do the locking themselves, and also 
> keep the order the same.

I thought about it previously, and wondered at what point we'll have to
introduce ww-mutex :-) Hopefully later, much later. I'm fine keeping the
lock in the drivers if it can help.

> I was hitting lockdep issues constantly when the v4l2-core was taking 
> the lock before calling the ops. I did sort some of those out, but 
> sorting some of those out cleanly wasn't trivial, and it felt like 
> swimming against the current. So I instead decided to go this way.
> 
> It might also introduce deadlocks in drivers. I don't have a real 
> example, but I'm not sure if it's too far fetched to imagine a case with 
> two subdev drivers, part of a single driver module, where, say, subdev A 
> gets a set_fmt call, it then calls set_fmt in subdev B, and B calls 
> get_fmt in subdev A. Yes, it's a contrived example, but are you sure 
> things like that are not done? =)

The best way to avoid this is to avoid subdev drivers calling each other
for anything else than .s_stream() and getting controls.

> I'm sure this can be sorted out, but I have a gut feeling it won't be easy.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 03/36] media: subdev: add 'which' to subdev state

[Laurent Pinchart]

Hi Tomi,

On Mon, Sep 27, 2021 at 11:55:38AM +0300, Tomi Valkeinen wrote:
> On 27/09/2021 03:48, Laurent Pinchart wrote:
> > On Thu, Sep 16, 2021 at 04:24:19PM +0300, Tomi Valkeinen wrote:
> >> On 16/09/2021 16:07, Jacopo Mondi wrote:
> >>
> >>> Also note that operations like s_stream do not take a state as
> >>> parameter. The driver has to fetch it from the subdev anyway
> >>> (this in reply to the idea of having the active state as parameter vs
> >>> retrieving it from the subdev if ACTIVE)
> >>>
> >>> While porting the R-Car drivers on top of this series I found myself
> >>> in the need to (in the s_stream call chain)
> >>>
> >>> static int rcsi2_start_receiver(struct rcar_csi2 *priv)
> >>> {
> >>> 	const struct v4l2_subdev_state *state = priv->subdev.state;
> >>> 	const struct v4l2_subdev_stream_configs *configs = &state->stream_configs;
> >>>
> >>>           ...
> >>>
> >>> 	/*
> >>> 	 * Configure field handling inspecting the formats of the
> >>> 	 * single sink pad streams.
> >>> 	 */
> >>> 	for (i = 0; i < configs->num_configs; ++i) {
> >>> 		const struct v4l2_subdev_stream_config *config = configs->configs;
> >>> 		if (config->pad != RCAR_CSI2_SINK)
> >>> 			continue;
> >>>
> >>> 		if (config->fmt.field != V4L2_FIELD_ALTERNATE)
> >>> 			continue;
> >>>
> >>> 		fld |= FLD_DET_SEL(1);
> >>> 		fld |= FLD_FLD_EN(config->stream);
> >>>
> >>> 		/* PAL vs NTSC. */
> >>> 		if (config->fmt.height == 240)
> >>> 			fld |= FLD_FLD_NUM(0);
> >>> 		else
> >>> 			fld |= FLD_FLD_NUM(1);
> >>> 	}
> >>>
> >>>           ...
> >>>
> >>> }
> >>>
> >>> Am I doing it wrong, or is this a case for the subdev to have to
> >>> directly access sd->state ?
> >>
> >> In s_stream path you should:
> >>
> >> 	state = v4l2_subdev_lock_active_state(sd);
> >>
> >> 	<do the work with the state>
> >>
> >> 	v4l2_subdev_unlock_state(state);
> >>
> >> If you already have the state, e.g. in set_fmt:
> >>
> >> 	state = v4l2_subdev_validate_and_lock_state(sd, state);
> >>
> >> 	<do the work with the state>
> >>
> >> 	v4l2_subdev_unlock_state(state);
> >>
> >> Accessing the stream_configs directly is fine but not that nice. I did
> >> think about some helpers, perhaps for_each_stream_config(), but I didn't
> >> add that as I didn't have the need.
> >>
> >> There's v4l2_state_get_stream_format() which can be used in many cases,
> >> but we probably need something else if you need to iterate over all the
> >> configs.
> > 
> > I really like forcing drivers to call functions that will lock the
> > state, at least until we can move the locks to the core (if ever). We
> > should move the fields of v4l2_subdev that drivers are not supposed to
> > access directly under a big PRIVATE comment.
> 
> Can you clarify what you mean here? Did you mean you like functions that 
> will _check_ the lock? Or did you just mean that you like 
> v4l2_subdev_lock_state() better than mutex_lock(state->lock)?

I like v4l2_subdev_lock_state().

> Well, in any case, I think my series does both =). I can add the private 
> comment to subdev. In fact, at one time I did not have sd->state, but 
> sd->_state, just to make sure no one accesses it.

I half recall there was a gcc attribute to mark private fields in a
structure, which then generates a warning if those fields are accessed
by a function that doesn't have another special attribute. Or maybe I
was dreaming :-)

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 06/36] media: subdev: Add v4l2_subdev_validate(_and_lock)_state()

[Tomi Valkeinen]

On 27/09/2021 04:45, Laurent Pinchart wrote:
> Hi Tomi,
> 
> Thank you for the patch.
> 
> On Mon, Aug 30, 2021 at 02:00:46PM +0300, Tomi Valkeinen wrote:
>> All suitable subdev ops are now passed either the TRY or the ACTIVE
>> state by the v4l2 core. However, other subdrivers can still call the ops
>> passing NULL as the state, implying the active case.
>>
>> Thus all subdev drivers supporting active state need to handle the NULL
>> state case.
> 
> Do they ? Can't we mandate that the callers pass the correct state ? Do
> you think that would make the transition too difficult ?

That would limit the use of the new drivers. E.g. the sensor driver I'm 
using works fine with CAL & co. without handling the NULL, but then the 
sensor driver couldn't be used with "old" drivers.

> The way I understand the issue, this would only be needed to facilitate
> the transition. Is there a reason why the drivers you've ported (CAL &
> co.) use v4l2_subdev_validate_and_lock_state() after completing the
> transition, or is the correct state always passed by the caller ?

The drivers I'm using only call non-state-aware ops in the other 
subdevs, so they should work fine without validate.

I think it's safer to just use the validate versions for now. They're 
trivial to convert to non-validate versions later.

We could just do the validate implicitly while locking the state, but it 
would make the code a bit odd:

state = v4l2_subdev_lock_state(state);

After the transition we want to do just:

v4l2_subdev_lock_state(state);

Or we could do v4l2_subdev_lock_state() with a macro, which changes the 
state variable, but... I think I'd rather have it clear and obvious with 
v4l2_subdev_validate_and_lock_state().

>> Additionally, the subdev drivers usually need to lock the
>> state.
>>
>> Add two helper functions to easen the transition to centrally managed
>> ACTIVE state. v4l2_subdev_validate_state() ensures that the state is not
>> NULL, and v4l2_subdev_validate_and_lock_state() does the same and
>> additionally locks the state.
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>>   include/media/v4l2-subdev.h | 41 +++++++++++++++++++++++++++++++++++++
>>   1 file changed, 41 insertions(+)
>>
>> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
>> index 52a725281b23..2290b5025fc0 100644
>> --- a/include/media/v4l2-subdev.h
>> +++ b/include/media/v4l2-subdev.h
>> @@ -1307,4 +1307,45 @@ void v4l2_subdev_lock_state(struct v4l2_subdev_state *state);
>>    */
>>   void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state);
>>   
>> +/**
>> + * v4l2_subdev_validate_state() - Gets the TRY or ACTIVE subdev state
>> + * @sd: subdevice
>> + * @state: subdevice state as passed to the subdev op
>> + *
>> + * Subdev ops used to be sometimes called with NULL as the state for ACTIVE
>> + * case. Even if the v4l2 core now passes proper state for both TRY and
>> + * ACTIVE cases, a subdev driver may call an op in another subdev driver,
>> + * passing NULL.
>> + *
>> + * This function can be used as a helper to get the state also for the ACTIVE
>> + * case. The subdev driver that supports ACTIVE state can use this function
>> + * as the first thing in its ops, ensuring that the state variable contains
>> + * either the TRY or ACTIVE state.
>> + */
>> +static inline struct v4l2_subdev_state *
>> +v4l2_subdev_validate_state(struct v4l2_subdev *sd,
>> +			   struct v4l2_subdev_state *state)
>> +{
>> +	return state ? state : sd->state;
>> +}
> 
> This doesn't seem to be used by the drivers you've ported, or by the
> R-Car and GMSL patches from Jacopo. Do we need this function ?

Probably not. If you need to validate, you most likely will lock the 
state right afterwards, so v4l2_subdev_validate_and_lock_state should be 
enough.

  Tomi

Re: [PATCH v8 02/36] media: subdev: add active state to struct v4l2_subdev

[Tomi Valkeinen]

On 27/09/2021 12:39, Laurent Pinchart wrote:

>>> I'd like to propose a third approach (or at least I believe it's a third
>>> one). I agree with Jacopo that the state structure should not have a
>>> which field, that's a layering violation. The state is a state,
>>> regardless of whether it holds TRY or ACTIVE data. What are the current
>>> blockers that would prevent that ?
>>
>> Oh, I agree with that too. I didn't add the 'which' field to state
>> because I thought it's good =). It is supposed to be temporary solution.
>> init_cfg() is the issue here.
>>
>> It think I had these options:
>>
>> - Change init_cfg() to take 'which' as a parameter
>> - Change init_cfg() implementations to not use 'which'
>> - Add new version for new drivers, init_cfg_state() or such, which
>> either gets the which as a parameter or doesn't use which.
>> - Add 'which' to state.
>>
>> I chose the fourth one as it's a very small change, and can be removed
>> easily in the future when the underlying problem is solved.
>>
>>> However, I think .init_cfg() should be used to initialize *all* states,
>>> both TRY and ACTIVE. That will simplify driver implementation (but
>>> centralizing the initialization in a single place) and ensure that the
>>> default value for the ACTIVE state always matches the default value for
>>> the TRY state, which I think is important.
>>
>> I agree.
> 
> That's the second option from you list above, right ?

Yes. I'm not 100% sure yet if it's possible to get rid of which from 
init_cfg, but I'll try it out.

> As ACTIVE state support is opt-in, it seems to me that we won't need to
> mass-fix drivers as part of this series if we go for this option. Am I

Yes, I think so. I'll be wiser after I've worked on this a bit =). I 
think the routing needs the biggest change, as the routing table 
contains 'which'. But routing won't affect the current drivers.

However, 'which' is quite ingrained to v4l2, I won't be too surprised if 
I keep finding new 'whiches' while removing it from the init_cfg call paths.

  Tomi

Re: [PATCH v8 06/36] media: subdev: Add v4l2_subdev_validate(_and_lock)_state()

[Laurent Pinchart]

Hi Tomi,

On Tue, Sep 28, 2021 at 08:02:18AM +0300, Tomi Valkeinen wrote:
> On 27/09/2021 04:45, Laurent Pinchart wrote:
> > On Mon, Aug 30, 2021 at 02:00:46PM +0300, Tomi Valkeinen wrote:
> >> All suitable subdev ops are now passed either the TRY or the ACTIVE
> >> state by the v4l2 core. However, other subdrivers can still call the ops
> >> passing NULL as the state, implying the active case.
> >>
> >> Thus all subdev drivers supporting active state need to handle the NULL
> >> state case.
> > 
> > Do they ? Can't we mandate that the callers pass the correct state ? Do
> > you think that would make the transition too difficult ?
> 
> That would limit the use of the new drivers. E.g. the sensor driver I'm 
> using works fine with CAL & co. without handling the NULL, but then the 
> sensor driver couldn't be used with "old" drivers.

I'm tempted to say that would be a good thing as it would accelerate the
transition :-)

> > The way I understand the issue, this would only be needed to facilitate
> > the transition. Is there a reason why the drivers you've ported (CAL &
> > co.) use v4l2_subdev_validate_and_lock_state() after completing the
> > transition, or is the correct state always passed by the caller ?
> 
> The drivers I'm using only call non-state-aware ops in the other 
> subdevs, so they should work fine without validate.
> 
> I think it's safer to just use the validate versions for now. They're 
> trivial to convert to non-validate versions later.
> 
> We could just do the validate implicitly while locking the state, but it 
> would make the code a bit odd:
> 
> state = v4l2_subdev_lock_state(state);
> 
> After the transition we want to do just:
> 
> v4l2_subdev_lock_state(state);
> 
> Or we could do v4l2_subdev_lock_state() with a macro, which changes the 
> state variable, but... I think I'd rather have it clear and obvious with 
> v4l2_subdev_validate_and_lock_state().
> 
> >> Additionally, the subdev drivers usually need to lock the
> >> state.
> >>
> >> Add two helper functions to easen the transition to centrally managed
> >> ACTIVE state. v4l2_subdev_validate_state() ensures that the state is not
> >> NULL, and v4l2_subdev_validate_and_lock_state() does the same and
> >> additionally locks the state.
> >>
> >> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> >> ---
> >>   include/media/v4l2-subdev.h | 41 +++++++++++++++++++++++++++++++++++++
> >>   1 file changed, 41 insertions(+)
> >>
> >> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> >> index 52a725281b23..2290b5025fc0 100644
> >> --- a/include/media/v4l2-subdev.h
> >> +++ b/include/media/v4l2-subdev.h
> >> @@ -1307,4 +1307,45 @@ void v4l2_subdev_lock_state(struct v4l2_subdev_state *state);
> >>    */
> >>   void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state);
> >>   
> >> +/**
> >> + * v4l2_subdev_validate_state() - Gets the TRY or ACTIVE subdev state
> >> + * @sd: subdevice
> >> + * @state: subdevice state as passed to the subdev op
> >> + *
> >> + * Subdev ops used to be sometimes called with NULL as the state for ACTIVE
> >> + * case. Even if the v4l2 core now passes proper state for both TRY and
> >> + * ACTIVE cases, a subdev driver may call an op in another subdev driver,
> >> + * passing NULL.
> >> + *
> >> + * This function can be used as a helper to get the state also for the ACTIVE
> >> + * case. The subdev driver that supports ACTIVE state can use this function
> >> + * as the first thing in its ops, ensuring that the state variable contains
> >> + * either the TRY or ACTIVE state.
> >> + */
> >> +static inline struct v4l2_subdev_state *
> >> +v4l2_subdev_validate_state(struct v4l2_subdev *sd,
> >> +			   struct v4l2_subdev_state *state)
> >> +{
> >> +	return state ? state : sd->state;
> >> +}
> > 
> > This doesn't seem to be used by the drivers you've ported, or by the
> > R-Car and GMSL patches from Jacopo. Do we need this function ?
> 
> Probably not. If you need to validate, you most likely will lock the 
> state right afterwards, so v4l2_subdev_validate_and_lock_state should be 
> enough.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 00/36] v4l: subdev internal routing and streams

[Jacopo Mondi]

Hi Tomi, Laurent

On Mon, Sep 27, 2021 at 04:24:53AM +0300, Laurent Pinchart wrote:
> Hi Tomi,
>
> On Mon, Sep 20, 2021 at 01:19:54PM +0300, Tomi Valkeinen wrote:
> > On 30/08/2021 14:00, Tomi Valkeinen wrote:
> > > Hi,
> > >
> > > This is v8 of the multiplexed streams series. v7 can be found from:
> > >
> > > https://lore.kernel.org/linux-media/20210524104408.599645-1-tomi.valkeinen@ideasonboard.com/
> > >
> > > The main change in this version is the implementation and use of
> > > centralized active state for subdevs.
> > >
> > > I have pushed my work branch to:
> > >
> > > git://git.kernel.org/pub/scm/linux/kernel/git/tomba/linux.git multistream/work-v8
> > >
> > > which contains the patches in this series, along with subdev drivers
> > > using multiplexed streams.
> > >
> > > Both this series and the branch above are based on top of today's
> > > git://linuxtv.org/media_tree.git master.
> > >
> > > The documentation still needs improving, but I hope the docs in this
> > > series, and the drivers in the work branch, are enough to give the
> > > reviewers enough information to do a review.
> > >
> > > As can be guessed from the work branch, I have been testing this series
> > > with TI's FPDLink setup. I have also done a "backwards compatibility"
> > > test by dropping all multiplexed streams patches from the CAL driver
> > > (the CSI-2 RX on the TI SoC), and using the FPDLink drivers with
> > > single-stream configuration.
> >
> > We've had good discussions with Jacopo about this series.
>

Thanks Tomi for summing it up here

> I hope my recent contribution was also useful to some extent :-) Up to
> patch 04/36, I like the direction this is taking and I'm quite confident
> that we'll reach an agreement. We need to get feedback from Sakari too
> though.
>
> > I chose the approaches in this series based on what I think the API
> > should be, even if the API has behaved differently before. And I think
> > I'm also leaning forward a bit, in the sense that the full benefit of
> > the API can only be had after more changes to the core and subdev
> > drivers (changes which may or may not happen).
> >
> > If I understood Jacopo correctly, his comments were essentially that my
> > approach is different than the current one, and as the current drivers
> > anyway do things the old way, this is very confusing. Basically I create
> > two different kinds of subdev drivers: the old and new ones, which
> > manage state differently.
> >

Most of my comments (I guess all except the addition of which to state
and a few other things) are about the quantity of helpers that have
been introduced.

Let me explain this:
I understand the benefits of relying on helpers to reduce the amount
of work the drivers have to do and enforce correctness, so I won't
argue about the fact they're helpful, but whenever I see something
like

        v4l2_subdev_validate_state(state)
        {
                return state ? state : sd->state;
        }

instead of having a driver doing

        void v4l2_subdev_op(sd, try_state, ...)
        {

                subdev_state *state;
                if (which == TRY)
                        state = try_state;
                else
                        state = sd->state;

        }

My immediate concern is the levels of obfuscation we introduce which
makes approaching the code a lot more difficult, at least for
me, and everytime I happen to jump into an helper which does something
trivial I'm left with the question "why an helper ? why a function
name I have to remember or jump into from my editor when it's so
simple to open code ?"

So yes, I guess you can call them tastes, I understand the benfit
and there seems to be a large consensus, so I'll just have to get to
like these extra levels of indirection.

(Talking with Tomi I said him "seems like reading DRM/KMS code, full
of helpers which do a single thing I could have open coded". I guess
it's a compliment).

> > I want to summarize two particular topics:
> >
> > 1) Active state & subdev ops
> >
> > In upstream we have v4l2_subdev_state which contains only the pad_config
> > array. This state is "try" state, it's allocated per file-handle, and
> > passed to the subdev drivers when executing subdev ioctls in try-mode
> > (which == V4L2_SUBDEV_FORMAT_TRY). This try-state is sometimes also
> > passed to the subdev drivers when executing in active-mode
> > (V4L2_SUBDEV_FORMAT_ACTIVE), but the drivers are supposed to ignore it.
> >
> > There is also an active-state, but it's driver-specific and
> > driver-internal. The drivers check the 'which' value, and either use the
> > passed try-state, or the internal state.
>
> To be very clear here, let's note that the driver-internal state is
> stored in a driver-specific format, which does not reuse the state
> structure used for the TRY state.
>

That's the current situation. I guess in the long term we should aim
to have as much as possible of the subdev state moved to the
'state' in the subdev.

> > What I did in this series aims to have both try- and active-states in
> > v4l2 core, and passing the correct state to subdevs so that they don't
> > (necessarily) need any internal state. There are some issues with it,
> > which have been discussed, but I believe those issues can be fixed.
> >
> > The subdev drivers need to be written to use this new active-state, so
> > it doesn't affect the current drivers.
> >
> > The question is, do we want to go that way?
>
> __     __  _______   ________
> \ \   / / |  _____| |  ______|
>  \ \ / /  | |       | |
>   \ v /   | |_____  | |______
>    | |    |  _____| |______  |
>    | |    | |              | |
>    | |    | |_____   ______| |
>    |_|    |_______| |________|
>
> (please let me know if you require additional clarification)
>
> > We could as well keep the
> > current behavior of subdev drivers only getting the try-state as a
> > parameter, and the drivers digging out the active state manually. This
> > active state could either be internal to the driver, or it could be in
> > the base struct v4l2_subdev (see also topic 2).
> >
> > 2) Shared subdev active-state
> >
> > The try-state is specific to a file-handle, and afaics have no real
> > race-issues as it's not really shared. Although I guess in theory an
> > application could call subdev ioctls from multiple threads using the
> > same fd.
>
> That's right. We could possibly serialize ioctl calls in v4l2-subdev.c.
>
> > In upstream the subdev drivers' internal state is managed fully by the
> > subdev drivers. The drivers are expected to handle necessary locking in
> > their subdev ops and interrupt handlers. If, say, v4l2 core needs to get
> > a format from the subdev, it calls a subdev op to get it.
>
> "supposed to" is the correct term here. Most of them don't (including
> drivers I have written myself), which I believe shows quite clearly that
> the API is wrong and that this shouldn't be left to drivers to handle.
>
> > In my series I aimed to a shared active-state. The state is located in a
> > known place, struct v4l2_subdev, and can be accessed without the subdev
> > driver's help. This requires locking, which I have implemented.
> >
> > At the moment the only real benefit with this is reading the routing
> > table while doing pipeline validation: Instead of having to dynamically
> > allocate memory and call the subdev op to create a copy of the routing
> > table (for each subdev, possibly multiple times), the validator can just
> > lock the state, and use it. And, in fact, there is no get_routing subdev
> > op at all.
> >

That's very nice, and I understand an helper here is useful, as
locking could be very well overlooked if subdev drivers are encouraged
to access their active state by just sd->state.

> > But this means that the subdev drivers that support this new
> > active-state have to handle locking for the active state, and the
> > "mindset" is different than previously.
>
> That's the right mindset I believe, and forcing drivers to use helper
> functions that ensure proper locking is the right way to go in my
> opinion.
>

Point taken. I'll learn how to love more those additional helpers.

Thanks
   j


> > So the question is, do we want to go that way? We could as well mandate
> > that the active-state can only be accessed via subdev's ops (and add the
> > get-routing, of course), and the subdev manages the locking internally.
>
> Been there, failed, let's not repeat the same mistake.
>
> --
> Regards,
>
> Laurent Pinchart

Re: [PATCH v8 02/36] media: subdev: add active state to struct v4l2_subdev

[Tomi Valkeinen]

On 28/09/2021 08:14, Tomi Valkeinen wrote:

> Yes. I'm not 100% sure yet if it's possible to get rid of which from 
> init_cfg, but I'll try it out.
> 
>> As ACTIVE state support is opt-in, it seems to me that we won't need to
>> mass-fix drivers as part of this series if we go for this option. Am I
> 
> Yes, I think so. I'll be wiser after I've worked on this a bit =). I 
> think the routing needs the biggest change, as the routing table 
> contains 'which'. But routing won't affect the current drivers.
> 
> However, 'which' is quite ingrained to v4l2, I won't be too surprised if 
> I keep finding new 'whiches' while removing it from the init_cfg call 
> paths.

It was rather easy to get rid of 'which'. I now have 
v4l2_subdev_routing, which is the struct used in the uAPI:

struct v4l2_subdev_routing {
	__u32 which;
	__u64 routes;
	__u32 num_routes;
	__u32 reserved[5];
};

Then I have v4l2_subdev_krouting, which is used when calling the subdev 
set_routing op:

struct v4l2_subdev_krouting {
	u32 which;
	struct v4l2_subdev_routing_table table;
};

And I have v4l2_subdev_routing_table, which is used in the various 
helper functions, and stored in the state:

struct v4l2_subdev_routing_table {
	unsigned int num_routes;
	struct v4l2_subdev_route *routes;
};

As the only use of v4l2_subdev_krouting is when calling the subdev's 
set_routing, I think we should just drop it and pass 'which' as a 
parameter to subdev's set_routing(). That is a different style compared 
to the other ops, which have the 'which' inside the passed struct.

Any opinions?

  Tomi

Re: [PATCH v8 06/36] media: subdev: Add v4l2_subdev_validate(_and_lock)_state()

[Tomi Valkeinen]

Hi Laurent,

On 28/09/2021 10:52, Laurent Pinchart wrote:
> Hi Tomi,
> 
> On Tue, Sep 28, 2021 at 08:02:18AM +0300, Tomi Valkeinen wrote:
>> On 27/09/2021 04:45, Laurent Pinchart wrote:
>>> On Mon, Aug 30, 2021 at 02:00:46PM +0300, Tomi Valkeinen wrote:
>>>> All suitable subdev ops are now passed either the TRY or the ACTIVE
>>>> state by the v4l2 core. However, other subdrivers can still call the ops
>>>> passing NULL as the state, implying the active case.
>>>>
>>>> Thus all subdev drivers supporting active state need to handle the NULL
>>>> state case.
>>>
>>> Do they ? Can't we mandate that the callers pass the correct state ? Do
>>> you think that would make the transition too difficult ?
>>
>> That would limit the use of the new drivers. E.g. the sensor driver I'm
>> using works fine with CAL & co. without handling the NULL, but then the
>> sensor driver couldn't be used with "old" drivers.
> 
> I'm tempted to say that would be a good thing as it would accelerate the
> transition :-)

I tested this, and as expected the drivers I'm using work fine without 
the "validate" version. From that perspective I'm fine with dropping the 
whole "validate" concept, and just require state to be valid for the new 
multiplexed-streams-enabled drivers.

But the thing is, all the drivers I use are new, and not used in other 
hardware platforms or configurations. However, if someone ports an 
existing driver to multiplexed streams, either he has to be sure no 
other setup is using that driver, or add some kind of "validate" system.

Maybe that's still ok. I don't expect people to be rushing to port the 
drivers to multiplexed streams =).

So if there are no complaints, I'll drop the validate functions. And we 
can always add them back later for the few drivers that need them, if 
the plan goes bad...

  Tomi

Re: [PATCH v8 06/36] media: subdev: Add v4l2_subdev_validate(_and_lock)_state()

[Laurent Pinchart]

Hi Tomi,

On Wed, Sep 29, 2021 at 06:35:30PM +0300, Tomi Valkeinen wrote:
> On 28/09/2021 10:52, Laurent Pinchart wrote:
> > On Tue, Sep 28, 2021 at 08:02:18AM +0300, Tomi Valkeinen wrote:
> >> On 27/09/2021 04:45, Laurent Pinchart wrote:
> >>> On Mon, Aug 30, 2021 at 02:00:46PM +0300, Tomi Valkeinen wrote:
> >>>> All suitable subdev ops are now passed either the TRY or the ACTIVE
> >>>> state by the v4l2 core. However, other subdrivers can still call the ops
> >>>> passing NULL as the state, implying the active case.
> >>>>
> >>>> Thus all subdev drivers supporting active state need to handle the NULL
> >>>> state case.
> >>>
> >>> Do they ? Can't we mandate that the callers pass the correct state ? Do
> >>> you think that would make the transition too difficult ?
> >>
> >> That would limit the use of the new drivers. E.g. the sensor driver I'm
> >> using works fine with CAL & co. without handling the NULL, but then the
> >> sensor driver couldn't be used with "old" drivers.
> > 
> > I'm tempted to say that would be a good thing as it would accelerate the
> > transition :-)
> 
> I tested this, and as expected the drivers I'm using work fine without 
> the "validate" version. From that perspective I'm fine with dropping the 
> whole "validate" concept, and just require state to be valid for the new 
> multiplexed-streams-enabled drivers.
> 
> But the thing is, all the drivers I use are new, and not used in other 
> hardware platforms or configurations. However, if someone ports an 
> existing driver to multiplexed streams, either he has to be sure no 
> other setup is using that driver, or add some kind of "validate" system.
> 
> Maybe that's still ok. I don't expect people to be rushing to port the 
> drivers to multiplexed streams =).
> 
> So if there are no complaints, I'll drop the validate functions. And we 
> can always add them back later for the few drivers that need them, if 
> the plan goes bad...

I tend to try and push for faster transitions, so I like this, but I
also understand your concern. For new drivers at least, I think we
shouldn't use the validate version. For existing drivers moved to the
new API, it may be more difficult to enforce that. I'd like to hear what
Hans and Sakari think about it, and I'll go with the majority.

Can you at least document the validate function as being temporary, and
not to be used for new drivers ?

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 02/36] media: subdev: add active state to struct v4l2_subdev

[Laurent Pinchart]

On Tue, Sep 28, 2021 at 03:33:18PM +0300, Tomi Valkeinen wrote:
> On 28/09/2021 08:14, Tomi Valkeinen wrote:
> 
> > Yes. I'm not 100% sure yet if it's possible to get rid of which from 
> > init_cfg, but I'll try it out.
> > 
> >> As ACTIVE state support is opt-in, it seems to me that we won't need to
> >> mass-fix drivers as part of this series if we go for this option. Am I
> > 
> > Yes, I think so. I'll be wiser after I've worked on this a bit =). I 
> > think the routing needs the biggest change, as the routing table 
> > contains 'which'. But routing won't affect the current drivers.
> > 
> > However, 'which' is quite ingrained to v4l2, I won't be too surprised if 
> > I keep finding new 'whiches' while removing it from the init_cfg call 
> > paths.
> 
> It was rather easy to get rid of 'which'.

Jee ! Olen iloinen :-)

> I now have 
> v4l2_subdev_routing, which is the struct used in the uAPI:
> 
> struct v4l2_subdev_routing {
> 	__u32 which;
> 	__u64 routes;
> 	__u32 num_routes;
> 	__u32 reserved[5];
> };
> 
> Then I have v4l2_subdev_krouting, which is used when calling the subdev 
> set_routing op:
> 
> struct v4l2_subdev_krouting {
> 	u32 which;
> 	struct v4l2_subdev_routing_table table;
> };
> 
> And I have v4l2_subdev_routing_table, which is used in the various 
> helper functions, and stored in the state:
> 
> struct v4l2_subdev_routing_table {
> 	unsigned int num_routes;
> 	struct v4l2_subdev_route *routes;
> };
> 
> As the only use of v4l2_subdev_krouting is when calling the subdev's 
> set_routing, I think we should just drop it and pass 'which' as a 
> parameter to subdev's set_routing(). That is a different style compared 
> to the other ops, which have the 'which' inside the passed struct.
> 
> Any opinions?

I don't mind, I was going to propose it when reading the above.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 28/36] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Dafna Hirschfeld]

Hi,

On 30.08.21 13:01, Tomi Valkeinen wrote:
> From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> 
> Add support for subdev internal routing. A route is defined as a single
> stream from a sink pad to a source pad.
> 
> The userspace can configure the routing via two new ioctls,
> VIDIOC_SUBDEV_G_ROUTING and VIDIOC_SUBDEV_S_ROUTING, and subdevs can
> implement the functionality with v4l2_subdev_pad_ops.set_routing().
> 
> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Signed-off-by: Michal Simek <michal.simek@xilinx.com>
> 
> - Add sink and source streams for multiplexed links
> - Copy the argument back in case of an error. This is needed to let the
>    caller know the number of routes.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> 
> - Expand and refine documentation.
> - Make the 'routes' pointer a __u64 __user pointer so that a compat32
>    version of the ioctl is not required.
> - Add struct v4l2_subdev_krouting to be used for subdevice operations.
> 
> Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
> 
> - Fix typecasing warnings
> - Check sink & source pad types
> - Add 'which' field
> - Add V4L2_SUBDEV_ROUTE_FL_SOURCE
> - Routing to subdev state
> - Dropped get_routing subdev op
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>   drivers/media/v4l2-core/v4l2-ioctl.c  | 25 ++++++++-
>   drivers/media/v4l2-core/v4l2-subdev.c | 75 +++++++++++++++++++++++++++
>   include/media/v4l2-subdev.h           | 24 +++++++++
>   include/uapi/linux/v4l2-subdev.h      | 57 ++++++++++++++++++++
>   4 files changed, 180 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index 05d5db3d85e5..8e9315ffcb99 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -16,6 +16,7 @@
>   #include <linux/kernel.h>
>   #include <linux/version.h>
>   
> +#include <linux/v4l2-subdev.h>
>   #include <linux/videodev2.h>
>   
>   #include <media/v4l2-common.h>
> @@ -3065,6 +3066,21 @@ static int check_array_args(unsigned int cmd, void *parg, size_t *array_size,
>   		ret = 1;
>   		break;
>   	}
> +
> +	case VIDIOC_SUBDEV_G_ROUTING:
> +	case VIDIOC_SUBDEV_S_ROUTING: {
> +		struct v4l2_subdev_routing *routing = parg;
> +
> +		if (routing->num_routes > 256)
> +			return -EINVAL;
> +
> +		*user_ptr = u64_to_user_ptr(routing->routes);
> +		*kernel_ptr = (void **)&routing->routes;
> +		*array_size = sizeof(struct v4l2_subdev_route)
> +			    * routing->num_routes;
> +		ret = 1;
> +		break;
> +	}
>   	}
>   
>   	return ret;
> @@ -3328,8 +3344,15 @@ video_usercopy(struct file *file, unsigned int orig_cmd, unsigned long arg,
>   	/*
>   	 * Some ioctls can return an error, but still have valid
>   	 * results that must be returned.
> +	 *
> +	 * FIXME: subdev IOCTLS are partially handled here and partially in
> +	 * v4l2-subdev.c and the 'always_copy' flag can only be set for IOCTLS
> +	 * defined here as part of the 'v4l2_ioctls' array. As
> +	 * VIDIOC_SUBDEV_G_ROUTING needs to return results to applications even
> +	 * in case of failure, but it is not defined here as part of the
> +	 * 'v4l2_ioctls' array, insert an ad-hoc check to address that.
>   	 */
> -	if (err < 0 && !always_copy)
> +	if (err < 0 && !always_copy && cmd != VIDIOC_SUBDEV_G_ROUTING)
>   		goto out;
>   
>   out_array_args:
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index b1e65488210d..0e1f325b3159 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -395,6 +395,12 @@ subdev_ioctl_get_state(struct v4l2_subdev *sd, struct v4l2_subdev_fh *subdev_fh,
>   		which = ((struct v4l2_subdev_selection *)arg)->which;
>   		break;
>   	}
> +
> +	case VIDIOC_SUBDEV_G_ROUTING:
> +	case VIDIOC_SUBDEV_S_ROUTING: {
> +		which = ((struct v4l2_subdev_routing *)arg)->which;
> +		break;
> +	}
>   	}
>   
>   	return which == V4L2_SUBDEV_FORMAT_TRY ?
> @@ -711,6 +717,74 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>   	case VIDIOC_SUBDEV_QUERYSTD:
>   		return v4l2_subdev_call(sd, video, querystd, arg);
>   
> +	case VIDIOC_SUBDEV_G_ROUTING: {
> +		struct v4l2_subdev_routing *routing = arg;
> +		struct v4l2_subdev_krouting *krouting;
> +
> +		if (!(sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED))
> +			return -ENOIOCTLCMD;
> +
> +		memset(routing->reserved, 0, sizeof(routing->reserved));
> +
> +		krouting = &state->routing;
> +
> +		if (routing->num_routes < krouting->num_routes) {
> +			routing->num_routes = krouting->num_routes;
> +			return -ENOSPC;
> +		}
> +
> +		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
> +		       krouting->routes,
> +		       krouting->num_routes * sizeof(*krouting->routes));
> +		routing->num_routes = krouting->num_routes;
> +
> +		return 0;
> +	}
> +
> +	case VIDIOC_SUBDEV_S_ROUTING: {
> +		struct v4l2_subdev_routing *routing = arg;
> +		struct v4l2_subdev_route *routes =
> +			(struct v4l2_subdev_route *)(uintptr_t)routing->routes;
> +		struct v4l2_subdev_krouting krouting = {};
> +		unsigned int i;
> +
> +		if (!(sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED))
> +			return -ENOIOCTLCMD;
> +
> +		if (routing->which != V4L2_SUBDEV_FORMAT_TRY && ro_subdev)
> +			return -EPERM;
> +
> +		memset(routing->reserved, 0, sizeof(routing->reserved));
> +
> +		for (i = 0; i < routing->num_routes; ++i) {
> +			const struct v4l2_subdev_route *route = &routes[i];
> +			const struct media_pad *pads = sd->entity.pads;
> +
> +			/* Do not check sink pad for source routes */
> +			if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE)) {
> +				if (route->sink_pad >= sd->entity.num_pads)
> +					return -EINVAL;
> +
> +				if (!(pads[route->sink_pad].flags &
> +				      MEDIA_PAD_FL_SINK))
> +					return -EINVAL;
> +			}
> +
> +			if (route->source_pad >= sd->entity.num_pads)
> +				return -EINVAL;
> +
> +			if (!(pads[route->source_pad].flags &
> +			      MEDIA_PAD_FL_SOURCE))
> +				return -EINVAL;
> +		}
> +
> +		krouting.which = routing->which;
> +		krouting.num_routes = routing->num_routes;
> +		krouting.routes = routes;
> +
> +		return v4l2_subdev_call(sd, pad, set_routing, state, &krouting);
> +	}
> +
>   	default:
>   		return v4l2_subdev_call(sd, core, ioctl, cmd, arg);
>   	}
> @@ -975,6 +1049,7 @@ void v4l2_free_subdev_state(struct v4l2_subdev_state *state)
>   
>   	mutex_destroy(&state->lock);
>   
> +	kvfree(state->routing.routes);
>   	kvfree(state->pads);
>   	kfree(state);
>   }
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 356901d8a948..cd6aad21ae0c 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -694,12 +694,29 @@ struct v4l2_subdev_pad_config {
>   	struct v4l2_rect try_compose;
>   };
>   
> +/**
> + * struct v4l2_subdev_krouting - subdev routing table
> + *
> + * @which: format type (from enum v4l2_subdev_format_whence)
> + * @routes: &struct v4l2_subdev_route
> + * @num_routes: number of routes
> + *
> + * This structure is used to translate arguments received from
> + * VIDIOC_SUBDEV_G/S_ROUTING() ioctl to subdev device drivers operations.
> + */
> +struct v4l2_subdev_krouting {
> +	u32 which;
> +	struct v4l2_subdev_route *routes;
> +	unsigned int num_routes;
> +};
> +
>   /**
>    * struct v4l2_subdev_state - Used for storing subdev state information.
>    *
>    * @lock: mutex for the state
>    * @which: state type (from enum v4l2_subdev_format_whence)
>    * @pads: &struct v4l2_subdev_pad_config array
> + * @routing: routing table for the subdev
>    *
>    * This structure only needs to be passed to the pad op if the 'which' field
>    * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
> @@ -709,6 +726,7 @@ struct v4l2_subdev_state {
>   	struct mutex lock;
>   	u32 which;
>   	struct v4l2_subdev_pad_config *pads;
> +	struct v4l2_subdev_krouting routing;
>   };
>   
>   /**
> @@ -772,6 +790,9 @@ struct v4l2_subdev_state {
>    *		     applied to the hardware. The operation shall fail if the
>    *		     pad index it has been called on is not valid or in case of
>    *		     unrecoverable failures.
> + *
> + * @set_routing: enable or disable data connection routes described in the
> + *		 subdevice routing table.
>    */
>   struct v4l2_subdev_pad_ops {
>   	int (*init_cfg)(struct v4l2_subdev *sd,
> @@ -816,6 +837,9 @@ struct v4l2_subdev_pad_ops {
>   			       struct v4l2_mbus_config *config);
>   	int (*set_mbus_config)(struct v4l2_subdev *sd, unsigned int pad,
>   			       struct v4l2_mbus_config *config);
> +	int (*set_routing)(struct v4l2_subdev *sd,
> +			   struct v4l2_subdev_state *state,
> +			   struct v4l2_subdev_krouting *route);
>   };
>   
>   /**
> diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
> index 658106f5b5dc..3aa623e0e5f9 100644
> --- a/include/uapi/linux/v4l2-subdev.h
> +++ b/include/uapi/linux/v4l2-subdev.h
> @@ -188,6 +188,61 @@ struct v4l2_subdev_capability {
>   /* The v4l2 sub-device video device node is registered in read-only mode. */
>   #define V4L2_SUBDEV_CAP_RO_SUBDEV		0x00000001
>   
> +/*
> + * Is the route active? An active route will start when streaming is enabled
> + * on a video node.
> + */
> +#define V4L2_SUBDEV_ROUTE_FL_ACTIVE		BIT(0)
> +
> +/*
> + * Is the route immutable, i.e. can it be activated and inactivated?
> + * Set by the driver.
> + */
> +#define V4L2_SUBDEV_ROUTE_FL_IMMUTABLE		BIT(1)
> +
> +/*
> + * Is the route a source endpoint? A source endpoint route refers to a stream
> + * generated internally by the subdevice (usually a sensor), and thus there
> + * is no sink-side endpoint for the route. The sink_pad and sink_stream
> + * fields are unused.
> + * Set by the driver.
> + */
> +#define V4L2_SUBDEV_ROUTE_FL_SOURCE		BIT(2)
> +
> +/**
> + * struct v4l2_subdev_route - A route inside a subdev
> + *
> + * @sink_pad: the sink pad index
> + * @sink_stream: the sink stream identifier
> + * @source_pad: the source pad index
> + * @source_stream: the source stream identifier
> + * @flags: route flags V4L2_SUBDEV_ROUTE_FL_*
> + * @reserved: drivers and applications must zero this array
> + */
> +struct v4l2_subdev_route {
> +	__u32 sink_pad;
> +	__u32 sink_stream;
> +	__u32 source_pad;
> +	__u32 source_stream;
> +	__u32 flags;
> +	__u32 reserved[5];
> +};
> +

I don't understand that struct, what is the meaning of the two sink_stream, source_stream fields?
What is the relation between sink_pad and source_pad? A 'route' between two pads means that 'streams' can flow through them?

If I have for example:

sink_pad = sink_stream = 0
source_pad = source_stream = 1

what does that mean?

Thanks,
Dafna

> +/**
> + * struct v4l2_subdev_routing - Subdev routing information
> + *
> + * @which: configuration type (from enum v4l2_subdev_format_whence)
> + * @routes: pointer to the routes array
> + * @num_routes: the total number of routes in the routes array
> + * @reserved: drivers and applications must zero this array
> + */
> +struct v4l2_subdev_routing {
> +	__u32 which;
> +	__u64 routes;
> +	__u32 num_routes;
> +	__u32 reserved[5];
> +};
> +
>   /* Backwards compatibility define --- to be removed */
>   #define v4l2_subdev_edid v4l2_edid
>   
> @@ -203,6 +258,8 @@ struct v4l2_subdev_capability {
>   #define VIDIOC_SUBDEV_S_CROP			_IOWR('V', 60, struct v4l2_subdev_crop)
>   #define VIDIOC_SUBDEV_G_SELECTION		_IOWR('V', 61, struct v4l2_subdev_selection)
>   #define VIDIOC_SUBDEV_S_SELECTION		_IOWR('V', 62, struct v4l2_subdev_selection)
> +#define VIDIOC_SUBDEV_G_ROUTING			_IOWR('V', 38, struct v4l2_subdev_routing)
> +#define VIDIOC_SUBDEV_S_ROUTING			_IOWR('V', 39, struct v4l2_subdev_routing)
>   /* The following ioctls are identical to the ioctls in videodev2.h */
>   #define VIDIOC_SUBDEV_G_STD			_IOR('V', 23, v4l2_std_id)
>   #define VIDIOC_SUBDEV_S_STD			_IOW('V', 24, v4l2_std_id)
> 

Re: [PATCH v8 28/36] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Tomi Valkeinen]

Hi,

On 03/10/2021 22:52, Dafna Hirschfeld wrote:

>> +/**
>> + * struct v4l2_subdev_route - A route inside a subdev
>> + *
>> + * @sink_pad: the sink pad index
>> + * @sink_stream: the sink stream identifier
>> + * @source_pad: the source pad index
>> + * @source_stream: the source stream identifier
>> + * @flags: route flags V4L2_SUBDEV_ROUTE_FL_*
>> + * @reserved: drivers and applications must zero this array
>> + */
>> +struct v4l2_subdev_route {
>> +    __u32 sink_pad;
>> +    __u32 sink_stream;
>> +    __u32 source_pad;
>> +    __u32 source_stream;
>> +    __u32 flags;
>> +    __u32 reserved[5];
>> +};
>> +
> 
> I don't understand that struct, what is the meaning of the two 
> sink_stream, source_stream fields?
> What is the relation between sink_pad and source_pad? A 'route' between 
> two pads means that 'streams' can flow through them?
> 
> If I have for example:
> 
> sink_pad = sink_stream = 0
> source_pad = source_stream = 1
> 
> what does that mean?

It means that a stream with stream ID 0 that comes to the subdev's sink 
pad 0 will be routed to the subdev's source pad 1 with stream ID 1.

So the sink_pad and source_pad tell via which pads the streams flows.

Stream IDs are numbers used to uniquely identify the stream for the 
specific pad. In other words, for the subdev's pad 0, each stream ID 
must be unique. That pad 0 is linked (media link) to another subdev's 
pad, and on that pad there must be streams with the matching IDs.

  Tomi

Re: [PATCH v8 28/36] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Dafna Hirschfeld]

On 04.10.21 07:15, Tomi Valkeinen wrote:
> Hi,
> 
> On 03/10/2021 22:52, Dafna Hirschfeld wrote:
> 
>>> +/**
>>> + * struct v4l2_subdev_route - A route inside a subdev
>>> + *
>>> + * @sink_pad: the sink pad index
>>> + * @sink_stream: the sink stream identifier
>>> + * @source_pad: the source pad index
>>> + * @source_stream: the source stream identifier
>>> + * @flags: route flags V4L2_SUBDEV_ROUTE_FL_*
>>> + * @reserved: drivers and applications must zero this array
>>> + */
>>> +struct v4l2_subdev_route {
>>> +    __u32 sink_pad;
>>> +    __u32 sink_stream;
>>> +    __u32 source_pad;
>>> +    __u32 source_stream;
>>> +    __u32 flags;
>>> +    __u32 reserved[5];
>>> +};
>>> +
>>
>> I don't understand that struct, what is the meaning of the two sink_stream, source_stream fields?
>> What is the relation between sink_pad and source_pad? A 'route' between two pads means that 'streams' can flow through them?
>>
>> If I have for example:
>>
>> sink_pad = sink_stream = 0
>> source_pad = source_stream = 1
>>
>> what does that mean?
> 
> It means that a stream with stream ID 0 that comes to the subdev's sink pad 0 will be routed to the subdev's source pad 1 with stream ID 1.
> 
> So the sink_pad and source_pad tell via which pads the streams flows.
> 
> Stream IDs are numbers used to uniquely identify the stream for the specific pad. In other words, for the subdev's pad 0, each stream ID must be unique. That pad 0 is linked (media link) to another subdev's pad, and on that pad there must be streams with the matching IDs.

Ok, so the stream ID for the same stream changes from link to link.

So different streams can have the same ID on different pads?

Won't it be more simple if the stream ID is forced to stay the same along all the links and pads?

> 
>   Tomi

Re: [PATCH v8 28/36] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Tomi Valkeinen]

Hi,

On 05/10/2021 13:19, Dafna Hirschfeld wrote:
> 
> 
> On 04.10.21 07:15, Tomi Valkeinen wrote:
>> Hi,
>>
>> On 03/10/2021 22:52, Dafna Hirschfeld wrote:
>>
>>>> +/**
>>>> + * struct v4l2_subdev_route - A route inside a subdev
>>>> + *
>>>> + * @sink_pad: the sink pad index
>>>> + * @sink_stream: the sink stream identifier
>>>> + * @source_pad: the source pad index
>>>> + * @source_stream: the source stream identifier
>>>> + * @flags: route flags V4L2_SUBDEV_ROUTE_FL_*
>>>> + * @reserved: drivers and applications must zero this array
>>>> + */
>>>> +struct v4l2_subdev_route {
>>>> +    __u32 sink_pad;
>>>> +    __u32 sink_stream;
>>>> +    __u32 source_pad;
>>>> +    __u32 source_stream;
>>>> +    __u32 flags;
>>>> +    __u32 reserved[5];
>>>> +};
>>>> +
>>>
>>> I don't understand that struct, what is the meaning of the two 
>>> sink_stream, source_stream fields?
>>> What is the relation between sink_pad and source_pad? A 'route' 
>>> between two pads means that 'streams' can flow through them?
>>>
>>> If I have for example:
>>>
>>> sink_pad = sink_stream = 0
>>> source_pad = source_stream = 1
>>>
>>> what does that mean?
>>
>> It means that a stream with stream ID 0 that comes to the subdev's 
>> sink pad 0 will be routed to the subdev's source pad 1 with stream ID 1.
>>
>> So the sink_pad and source_pad tell via which pads the streams flows.
>>
>> Stream IDs are numbers used to uniquely identify the stream for the 
>> specific pad. In other words, for the subdev's pad 0, each stream ID 
>> must be unique. That pad 0 is linked (media link) to another subdev's 
>> pad, and on that pad there must be streams with the matching IDs.

Btw, I've just sent v9 of the series, which includes documentation that 
covers this.

> Ok, so the stream ID for the same stream changes from link to link.

Not necessarily, but yes, it may.

> So different streams can have the same ID on different pads?

True.

> Won't it be more simple if the stream ID is forced to stay the same 
> along all the links and pads?

I don't think so.

For one, it would be difficult, or at least laborious, to verify that 
the above holds true.

Also, streams can appear and disappear, e.g. a single stream in a 
subdev's input side might be split or duplicated into multiple streams 
on the output side.

And I think that it makes sense that e.g. a sensor that outputs two 
streams (pixel & metadata) can have a hardcoded routing, with pad 0 and 
streams 0 and 1. Stream 0 is always the pixel data, and stream 1 is 
always the metadata, and the user knows this so he can connect those 
streams appropriately. If you have multiple cameras, such a scheme would 
not be possible.

What would be the benefit of such rule? Afaik, it would be just for the 
benefit of the human who writes the configuration for the pipeline. And 
with the current series, nothing prevents the user from assigning 
"unique" stream IDs for the pipeline being configured. You can pick 
stream ID 6001 for pixel stream from camera 1, and keep that stream ID 
for the whole pipeline for that stream (assuming the stream won't be split).

That said, I agree that "stream ID" kind of sounds like something used 
to identify the stream, regardless where in the pipeline we look. But 
that is not the case. As I write in the docs: "A stream ID (often just 
"stream") is a media link-local identifier for a stream".

  Tomi