[PATCH v15 00/19] v4l: routing and streams support

[PATCH v15 00/19] v4l: routing and streams support

[Tomi Valkeinen]

Hi,

This is v15 of the streams series. The v14 can be found from:

https://lore.kernel.org/all/20220831141357.1396081-1-tomi.valkeinen@ideasonboard.com/

My work branch with additional drivers can be found from:

git://git.kernel.org/pub/scm/linux/kernel/git/tomba/linux.git streams/work-v15

And there's also the v4l-utils series v2 to add support to v4l2-ctl and
media-ctl:

https://lore.kernel.org/all/20220714132116.132498-1-tomi.valkeinen@ideasonboard.com/

This series is based on top of linux-media-stage, and this series is now
smaller as many patches from v14 have been merged.

Changes in v15:

media: v4l2-subdev: Sort includes
- New patch

media: add V4L2_SUBDEV_CAP_STREAMS
- Small refactor to use 'streams_subdev' variable

media: Documentation: Add GS_ROUTING documentation
- Improve error code descriptions

media: subdev: Add [GS]_ROUTING subdev ioctls and operations
- Return E2BIG when there are too many routes in VIDIOC_SUBDEV_S_ROUTING
- Use plain bitshift in the uapi header instead of _BITUL()
- V4L2_SUBDEV_ROUTE_FL_SOURCE is now bit 1, so if you have userspace which uses V4L2_SUBDEV_ROUTE_FL_SOURCE, you need to recompile

media: subdev: add v4l2_subdev_set_routing helper()
- Check for overflow in v4l2_subdev_set_routing()

media: subdev: add stream based configuration
- Fix compilation for !CONFIG_VIDEO_V4L2_SUBDEV_API or !CONFIG_MEDIA_CONTROLLER

media: subdev: use streams in v4l2_subdev_link_validate()
- Modify link validation to use 64bit streams-mask instead of collecting stream IDs into allocated arrays. This simplifies the functions quite a bit.

 Tomi


Jacopo Mondi (2):
  media: Documentation: Add GS_ROUTING documentation
  media: subdev: Add for_each_active_route() macro

Laurent Pinchart (5):
  media: subdev: Add [GS]_ROUTING subdev ioctls and operations
  media: subdev: add v4l2_subdev_routing_validate() helper
  media: v4l2-subdev: Add v4l2_subdev_state_xlate_streams() helper
  media: v4l2-subdev: Add subdev .(enable|disable)_streams() operations
  media: v4l2-subdev: Add v4l2_subdev_s_stream_helper() function

Sakari Ailus (1):
  media: Add stream to frame descriptor

Tomi Valkeinen (11):
  media: v4l2-subdev: Sort includes
  media: add V4L2_SUBDEV_FL_STREAMS
  media: add V4L2_SUBDEV_CAP_STREAMS
  media: subdev: add v4l2_subdev_has_pad_interdep()
  media: subdev: add v4l2_subdev_set_routing helper()
  media: Documentation: add multiplexed streams documentation
  media: subdev: add stream based configuration
  media: subdev: use streams in v4l2_subdev_link_validate()
  media: subdev: add "opposite" stream helper funcs
  media: subdev: add streams to v4l2_subdev_get_fmt() helper function
  media: subdev: add v4l2_subdev_set_routing_with_fmt() helper

 .clang-format                                 |   1 +
 .../driver-api/media/v4l2-subdev.rst          |   8 +
 .../userspace-api/media/v4l/dev-subdev.rst    | 175 ++++
 .../userspace-api/media/v4l/user-func.rst     |   1 +
 .../v4l/vidioc-subdev-enum-frame-interval.rst |   5 +-
 .../v4l/vidioc-subdev-enum-frame-size.rst     |   5 +-
 .../v4l/vidioc-subdev-enum-mbus-code.rst      |   5 +-
 .../media/v4l/vidioc-subdev-g-crop.rst        |   5 +-
 .../media/v4l/vidioc-subdev-g-fmt.rst         |   5 +-
 .../v4l/vidioc-subdev-g-frame-interval.rst    |   5 +-
 .../media/v4l/vidioc-subdev-g-routing.rst     | 155 +++
 .../media/v4l/vidioc-subdev-g-selection.rst   |   5 +-
 drivers/media/v4l2-core/v4l2-ioctl.c          |  25 +-
 drivers/media/v4l2-core/v4l2-subdev.c         | 991 +++++++++++++++++-
 include/media/v4l2-subdev.h                   | 377 +++++++
 include/uapi/linux/v4l2-subdev.h              |  83 +-
 16 files changed, 1801 insertions(+), 50 deletions(-)
 create mode 100644 Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst

-- 
2.34.1

[PATCH v15 01/19] media: v4l2-subdev: Sort includes

[Tomi Valkeinen]

Sort the includes alphabetically.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 5c27bac772ea..ca5b764d796d 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -8,20 +8,20 @@
  *	    Sakari Ailus <sakari.ailus@iki.fi>
  */
 
+#include <linux/export.h>
 #include <linux/ioctl.h>
 #include <linux/mm.h>
 #include <linux/module.h>
 #include <linux/slab.h>
 #include <linux/types.h>
-#include <linux/videodev2.h>
-#include <linux/export.h>
 #include <linux/version.h>
+#include <linux/videodev2.h>
 
 #include <media/v4l2-ctrls.h>
 #include <media/v4l2-device.h>
-#include <media/v4l2-ioctl.h>
-#include <media/v4l2-fh.h>
 #include <media/v4l2-event.h>
+#include <media/v4l2-fh.h>
+#include <media/v4l2-ioctl.h>
 
 #if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
 static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
-- 
2.34.1

[PATCH v15 02/19] media: add V4L2_SUBDEV_FL_STREAMS

[Tomi Valkeinen]

Add subdev flag V4L2_SUBDEV_FL_STREAMS. It is used to indicate that the
subdev supports the new API with multiplexed streams (routing, stream
configs).

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 include/media/v4l2-subdev.h | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 2f80c9c818ed..4be0a590c7c7 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -879,6 +879,17 @@ struct v4l2_subdev_internal_ops {
  * should set this flag.
  */
 #define V4L2_SUBDEV_FL_HAS_EVENTS		(1U << 3)
+/*
+ * Set this flag if this subdev supports multiplexed streams. This means
+ * that the driver supports routing and handles the stream parameter in its
+ * v4l2_subdev_pad_ops handlers. More specifically, this means:
+ *
+ * - Centrally managed subdev active state is enabled
+ * - Legacy pad config is _not_ supported (state->pads is NULL)
+ * - Routing ioctls are available
+ * - Multiple streams per pad are supported
+ */
+#define V4L2_SUBDEV_FL_STREAMS			(1U << 4)
 
 struct regulator_bulk_data;
 
-- 
2.34.1

[PATCH v15 03/19] media: add V4L2_SUBDEV_CAP_STREAMS

[Tomi Valkeinen]

Add a subdev capability flag to expose to userspace if a subdev supports
multiplexed streams.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 5 ++++-
 include/uapi/linux/v4l2-subdev.h      | 3 +++
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index ca5b764d796d..8983d33fdb4b 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -431,6 +431,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 	struct v4l2_subdev *sd = vdev_to_v4l2_subdev(vdev);
 	struct v4l2_fh *vfh = file->private_data;
 	bool ro_subdev = test_bit(V4L2_FL_SUBDEV_RO_DEVNODE, &vdev->flags);
+	bool streams_subdev = sd->flags & V4L2_SUBDEV_FL_STREAMS;
 	int rval;
 
 	switch (cmd) {
@@ -439,7 +440,9 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 
 		memset(cap->reserved, 0, sizeof(cap->reserved));
 		cap->version = LINUX_VERSION_CODE;
-		cap->capabilities = ro_subdev ? V4L2_SUBDEV_CAP_RO_SUBDEV : 0;
+		cap->capabilities =
+			(ro_subdev ? V4L2_SUBDEV_CAP_RO_SUBDEV : 0) |
+			(streams_subdev ? V4L2_SUBDEV_CAP_STREAMS : 0);
 
 		return 0;
 	}
diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
index 658106f5b5dc..89af27f50a41 100644
--- a/include/uapi/linux/v4l2-subdev.h
+++ b/include/uapi/linux/v4l2-subdev.h
@@ -188,6 +188,9 @@ struct v4l2_subdev_capability {
 /* The v4l2 sub-device video device node is registered in read-only mode. */
 #define V4L2_SUBDEV_CAP_RO_SUBDEV		0x00000001
 
+/* The v4l2 sub-device supports routing and multiplexed streams. */
+#define V4L2_SUBDEV_CAP_STREAMS			0x00000002
+
 /* Backwards compatibility define --- to be removed */
 #define v4l2_subdev_edid v4l2_edid
 
-- 
2.34.1

[PATCH v15 04/19] media: Documentation: Add GS_ROUTING documentation

[Tomi Valkeinen]

From: Jacopo Mondi <jacopo+renesas@jmondi.org>

Add documentation for VIDIOC_SUBDEV_G/S_ROUTING ioctl and add
description of multiplexed media pads and internal routing to the
V4L2-subdev documentation section.

Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
---
 .../userspace-api/media/v4l/dev-subdev.rst    |   2 +
 .../userspace-api/media/v4l/user-func.rst     |   1 +
 .../media/v4l/vidioc-subdev-g-routing.rst     | 155 ++++++++++++++++++
 3 files changed, 158 insertions(+)
 create mode 100644 Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst

diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index fd1de0a73a9f..a67c2749089a 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -29,6 +29,8 @@ will feature a character device node on which ioctls can be called to
 
 -  negotiate image formats on individual pads
 
+-  inspect and modify internal data routing between pads of the same entity
+
 Sub-device character device nodes, conventionally named
 ``/dev/v4l-subdev*``, use major number 81.
 
diff --git a/Documentation/userspace-api/media/v4l/user-func.rst b/Documentation/userspace-api/media/v4l/user-func.rst
index 53e604bd7d60..228c1521f190 100644
--- a/Documentation/userspace-api/media/v4l/user-func.rst
+++ b/Documentation/userspace-api/media/v4l/user-func.rst
@@ -70,6 +70,7 @@ Function Reference
     vidioc-subdev-g-crop
     vidioc-subdev-g-fmt
     vidioc-subdev-g-frame-interval
+    vidioc-subdev-g-routing
     vidioc-subdev-g-selection
     vidioc-subdev-querycap
     vidioc-subscribe-event
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
new file mode 100644
index 000000000000..6d8fc3b11352
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
@@ -0,0 +1,155 @@
+.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
+.. c:namespace:: V4L
+
+.. _VIDIOC_SUBDEV_G_ROUTING:
+
+******************************************************
+ioctl VIDIOC_SUBDEV_G_ROUTING, VIDIOC_SUBDEV_S_ROUTING
+******************************************************
+
+Name
+====
+
+VIDIOC_SUBDEV_G_ROUTING - VIDIOC_SUBDEV_S_ROUTING - Get or set routing between streams of media pads in a media entity.
+
+
+Synopsis
+========
+
+.. c:function:: int ioctl( int fd, VIDIOC_SUBDEV_G_ROUTING, struct v4l2_subdev_routing *argp )
+    :name: VIDIOC_SUBDEV_G_ROUTING
+
+.. c:function:: int ioctl( int fd, VIDIOC_SUBDEV_S_ROUTING, struct v4l2_subdev_routing *argp )
+    :name: VIDIOC_SUBDEV_S_ROUTING
+
+
+Arguments
+=========
+
+``fd``
+    File descriptor returned by :ref:`open() <func-open>`.
+
+``argp``
+    Pointer to struct :c:type:`v4l2_subdev_routing`.
+
+
+Description
+===========
+
+These ioctls are used to get and set the routing in a media entity.
+The routing configuration determines the flows of data inside an entity.
+
+Drivers report their current routing tables using the
+``VIDIOC_SUBDEV_G_ROUTING`` ioctl and application may enable or disable routes
+with the ``VIDIOC_SUBDEV_S_ROUTING`` ioctl, by adding or removing routes and
+setting or clearing flags of the  ``flags`` field of a
+struct :c:type:`v4l2_subdev_route`.
+
+All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called. This
+means that the userspace mut reconfigure all streams after calling the ioctl
+with e.g. ``VIDIOC_SUBDEV_S_FMT``.
+
+A special case for routing are routes marked with
+``V4L2_SUBDEV_ROUTE_FL_SOURCE`` flag. These routes are used to describe
+source endpoints on sensors and the sink fields are unused.
+
+When inspecting routes through ``VIDIOC_SUBDEV_G_ROUTING`` and the application
+provided ``num_routes`` is not big enough to contain all the available routes
+the subdevice exposes, drivers return the ENOSPC error code and adjust the
+value of the ``num_routes`` field. Application should then reserve enough memory
+for all the route entries and call ``VIDIOC_SUBDEV_G_ROUTING`` again.
+
+.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
+
+.. c:type:: v4l2_subdev_routing
+
+.. flat-table:: struct v4l2_subdev_routing
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u32
+      - ``which``
+      - Format to modified, from enum
+        :ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
+    * - struct :c:type:`v4l2_subdev_route`
+      - ``routes[]``
+      - Array of struct :c:type:`v4l2_subdev_route` entries
+    * - __u32
+      - ``num_routes``
+      - Number of entries of the routes array
+    * - __u32
+      - ``reserved``\ [5]
+      - Reserved for future extensions. Applications and drivers must set
+	the array to zero.
+
+.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
+
+.. c:type:: v4l2_subdev_route
+
+.. flat-table:: struct v4l2_subdev_route
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u32
+      - ``sink_pad``
+      - Sink pad number.
+    * - __u32
+      - ``sink_stream``
+      - Sink pad stream number.
+    * - __u32
+      - ``source_pad``
+      - Source pad number.
+    * - __u32
+      - ``source_stream``
+      - Source pad stream number.
+    * - __u32
+      - ``flags``
+      - Route enable/disable flags
+	:ref:`v4l2_subdev_routing_flags <v4l2-subdev-routing-flags>`.
+    * - __u32
+      - ``reserved``\ [5]
+      - Reserved for future extensions. Applications and drivers must set
+	the array to zero.
+
+.. tabularcolumns:: |p{6.6cm}|p{2.2cm}|p{8.7cm}|
+
+.. _v4l2-subdev-routing-flags:
+
+.. flat-table:: enum v4l2_subdev_routing_flags
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       3 1 4
+
+    * - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - 0
+      - The route is enabled. Set by applications.
+    * - V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
+      - 1
+      - The route is immutable. Set by the driver.
+    * - V4L2_SUBDEV_ROUTE_FL_SOURCE
+      - 2
+      - The route is a source route, and the ``sink_pad`` and ``sink_stream``
+        fields are unused. Set by the driver.
+
+Return Value
+============
+
+On success 0 is returned, on error -1 and the ``errno`` variable is set
+appropriately. The generic error codes are described at the
+:ref:`Generic Error Codes <gen-errors>` chapter.
+
+ENOSPC
+   The application provided ``num_routes`` is not big enough to contain
+   all the available routes the subdevice exposes.
+
+EINVAL
+   The sink or source pad identifiers reference a non-existing pad, or reference
+   pads of different types (ie. the sink_pad identifiers refers to a source pad)
+   or the sink or source stream identifiers reference a non-existing stream on
+   the sink or source pad.
+
+E2BIG
+   The application provided ``num_routes`` for ``VIDIOC_SUBDEV_S_ROUTING`` is
+   larger than the number of routes the driver can handle.
-- 
2.34.1

[PATCH v15 05/19] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Add support for subdev internal routing. A route is defined as a single
stream from a sink pad to a source pad.

The userspace can configure the routing via two new ioctls,
VIDIOC_SUBDEV_G_ROUTING and VIDIOC_SUBDEV_S_ROUTING, and subdevs can
implement the functionality with v4l2_subdev_pad_ops.set_routing().

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Michal Simek <michal.simek@xilinx.com>

- Add sink and source streams for multiplexed links
- Copy the argument back in case of an error. This is needed to let the
  caller know the number of routes.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>

- Expand and refine documentation.
- Make the 'routes' pointer a __u64 __user pointer so that a compat32
  version of the ioctl is not required.
- Add struct v4l2_subdev_krouting to be used for subdevice operations.

Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>

- Fix typecasing warnings
- Check sink & source pad types
- Add 'which' field
- Add V4L2_SUBDEV_ROUTE_FL_SOURCE
- Routing to subdev state
- Dropped get_routing subdev op

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-ioctl.c  | 25 +++++++-
 drivers/media/v4l2-core/v4l2-subdev.c | 87 +++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 22 +++++++
 include/uapi/linux/v4l2-subdev.h      | 52 ++++++++++++++++
 4 files changed, 185 insertions(+), 1 deletion(-)

diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index fddba75d9074..26b9a9626c96 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -16,6 +16,7 @@
 #include <linux/kernel.h>
 #include <linux/version.h>
 
+#include <linux/v4l2-subdev.h>
 #include <linux/videodev2.h>
 
 #include <media/media-device.h> /* for media_set_bus_info() */
@@ -3151,6 +3152,21 @@ static int check_array_args(unsigned int cmd, void *parg, size_t *array_size,
 		ret = 1;
 		break;
 	}
+
+	case VIDIOC_SUBDEV_G_ROUTING:
+	case VIDIOC_SUBDEV_S_ROUTING: {
+		struct v4l2_subdev_routing *routing = parg;
+
+		if (routing->num_routes > 256)
+			return -E2BIG;
+
+		*user_ptr = u64_to_user_ptr(routing->routes);
+		*kernel_ptr = (void **)&routing->routes;
+		*array_size = sizeof(struct v4l2_subdev_route)
+			    * routing->num_routes;
+		ret = 1;
+		break;
+	}
 	}
 
 	return ret;
@@ -3397,8 +3413,15 @@ video_usercopy(struct file *file, unsigned int orig_cmd, unsigned long arg,
 	/*
 	 * Some ioctls can return an error, but still have valid
 	 * results that must be returned.
+	 *
+	 * FIXME: subdev IOCTLS are partially handled here and partially in
+	 * v4l2-subdev.c and the 'always_copy' flag can only be set for IOCTLS
+	 * defined here as part of the 'v4l2_ioctls' array. As
+	 * VIDIOC_SUBDEV_G_ROUTING needs to return results to applications even
+	 * in case of failure, but it is not defined here as part of the
+	 * 'v4l2_ioctls' array, insert an ad-hoc check to address that.
 	 */
-	if (err < 0 && !always_copy)
+	if (err < 0 && !always_copy && cmd != VIDIOC_SUBDEV_G_ROUTING)
 		goto out;
 
 	if (has_array_args) {
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 8983d33fdb4b..1a451351554b 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -23,6 +23,16 @@
 #include <media/v4l2-fh.h>
 #include <media/v4l2-ioctl.h>
 
+/*
+ * Maximum stream ID is 63 for now, as we use u64 bitmask to represent a set
+ * of streams.
+ *
+ * Note that V4L2_FRAME_DESC_ENTRY_MAX is related: V4L2_FRAME_DESC_ENTRY_MAX
+ * restricts the total number of streams in a pad, although the stream ID is
+ * not restricted.
+ */
+#define V4L2_SUBDEV_MAX_STREAM_ID 63
+
 #if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
 static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
 {
@@ -417,6 +427,10 @@ subdev_ioctl_get_state(struct v4l2_subdev *sd, struct v4l2_subdev_fh *subdev_fh,
 	case VIDIOC_SUBDEV_S_SELECTION:
 		which = ((struct v4l2_subdev_selection *)arg)->which;
 		break;
+	case VIDIOC_SUBDEV_G_ROUTING:
+	case VIDIOC_SUBDEV_S_ROUTING:
+		which = ((struct v4l2_subdev_routing *)arg)->which;
+		break;
 	}
 
 	return which == V4L2_SUBDEV_FORMAT_TRY ?
@@ -733,6 +747,78 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 	case VIDIOC_SUBDEV_QUERYSTD:
 		return v4l2_subdev_call(sd, video, querystd, arg);
 
+	case VIDIOC_SUBDEV_G_ROUTING: {
+		struct v4l2_subdev_routing *routing = arg;
+		struct v4l2_subdev_krouting *krouting;
+
+		if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS))
+			return -ENOIOCTLCMD;
+
+		memset(routing->reserved, 0, sizeof(routing->reserved));
+
+		krouting = &state->routing;
+
+		if (routing->num_routes < krouting->num_routes) {
+			routing->num_routes = krouting->num_routes;
+			return -ENOSPC;
+		}
+
+		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
+		       krouting->routes,
+		       krouting->num_routes * sizeof(*krouting->routes));
+		routing->num_routes = krouting->num_routes;
+
+		return 0;
+	}
+
+	case VIDIOC_SUBDEV_S_ROUTING: {
+		struct v4l2_subdev_routing *routing = arg;
+		struct v4l2_subdev_route *routes =
+			(struct v4l2_subdev_route *)(uintptr_t)routing->routes;
+		struct v4l2_subdev_krouting krouting = {};
+		unsigned int i;
+
+		if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS))
+			return -ENOIOCTLCMD;
+
+		if (routing->which != V4L2_SUBDEV_FORMAT_TRY && ro_subdev)
+			return -EPERM;
+
+		memset(routing->reserved, 0, sizeof(routing->reserved));
+
+		for (i = 0; i < routing->num_routes; ++i) {
+			const struct v4l2_subdev_route *route = &routes[i];
+			const struct media_pad *pads = sd->entity.pads;
+
+			if (route->sink_stream > V4L2_SUBDEV_MAX_STREAM_ID ||
+			    route->source_stream > V4L2_SUBDEV_MAX_STREAM_ID)
+				return -EINVAL;
+
+			/* Do not check sink pad for source routes */
+			if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE)) {
+				if (route->sink_pad >= sd->entity.num_pads)
+					return -EINVAL;
+
+				if (!(pads[route->sink_pad].flags &
+				      MEDIA_PAD_FL_SINK))
+					return -EINVAL;
+			}
+
+			if (route->source_pad >= sd->entity.num_pads)
+				return -EINVAL;
+
+			if (!(pads[route->source_pad].flags &
+			      MEDIA_PAD_FL_SOURCE))
+				return -EINVAL;
+		}
+
+		krouting.num_routes = routing->num_routes;
+		krouting.routes = routes;
+
+		return v4l2_subdev_call(sd, pad, set_routing, state,
+					routing->which, &krouting);
+	}
+
 	default:
 		return v4l2_subdev_call(sd, core, ioctl, cmd, arg);
 	}
@@ -1016,6 +1102,7 @@ void __v4l2_subdev_state_free(struct v4l2_subdev_state *state)
 
 	mutex_destroy(&state->_lock);
 
+	kfree(state->routing.routes);
 	kvfree(state->pads);
 	kfree(state);
 }
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 4be0a590c7c7..4934dc9468a8 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -695,12 +695,26 @@ struct v4l2_subdev_pad_config {
 	struct v4l2_rect try_compose;
 };
 
+/**
+ * struct v4l2_subdev_krouting - subdev routing table
+ *
+ * @num_routes: number of routes
+ * @routes: &struct v4l2_subdev_route
+ *
+ * This structure contains the routing table for a subdev.
+ */
+struct v4l2_subdev_krouting {
+	unsigned int num_routes;
+	struct v4l2_subdev_route *routes;
+};
+
 /**
  * struct v4l2_subdev_state - Used for storing subdev state information.
  *
  * @_lock: default for 'lock'
  * @lock: mutex for the state. May be replaced by the user.
  * @pads: &struct v4l2_subdev_pad_config array
+ * @routing: routing table for the subdev
  *
  * This structure only needs to be passed to the pad op if the 'which' field
  * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
@@ -711,6 +725,7 @@ struct v4l2_subdev_state {
 	struct mutex _lock;
 	struct mutex *lock;
 	struct v4l2_subdev_pad_config *pads;
+	struct v4l2_subdev_krouting routing;
 };
 
 /**
@@ -763,6 +778,9 @@ struct v4l2_subdev_state {
  *		     this operation as close as possible to stream on time. The
  *		     operation shall fail if the pad index it has been called on
  *		     is not valid or in case of unrecoverable failures.
+ *
+ * @set_routing: enable or disable data connection routes described in the
+ *		 subdevice routing table.
  */
 struct v4l2_subdev_pad_ops {
 	int (*init_cfg)(struct v4l2_subdev *sd,
@@ -805,6 +823,10 @@ struct v4l2_subdev_pad_ops {
 			      struct v4l2_mbus_frame_desc *fd);
 	int (*get_mbus_config)(struct v4l2_subdev *sd, unsigned int pad,
 			       struct v4l2_mbus_config *config);
+	int (*set_routing)(struct v4l2_subdev *sd,
+			   struct v4l2_subdev_state *state,
+			   enum v4l2_subdev_format_whence which,
+			   struct v4l2_subdev_krouting *route);
 };
 
 /**
diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
index 89af27f50a41..db4af236d7ed 100644
--- a/include/uapi/linux/v4l2-subdev.h
+++ b/include/uapi/linux/v4l2-subdev.h
@@ -24,6 +24,7 @@
 #ifndef __LINUX_V4L2_SUBDEV_H
 #define __LINUX_V4L2_SUBDEV_H
 
+#include <linux/const.h>
 #include <linux/ioctl.h>
 #include <linux/types.h>
 #include <linux/v4l2-common.h>
@@ -191,6 +192,55 @@ struct v4l2_subdev_capability {
 /* The v4l2 sub-device supports routing and multiplexed streams. */
 #define V4L2_SUBDEV_CAP_STREAMS			0x00000002
 
+/*
+ * Is the route active? An active route will start when streaming is enabled
+ * on a video node.
+ */
+#define V4L2_SUBDEV_ROUTE_FL_ACTIVE		(1U << 0)
+
+/*
+ * Is the route a source endpoint? A source endpoint route refers to a stream
+ * generated by the subdevice (usually a sensor), and thus there is no
+ * sink-side endpoint for the route. The sink_pad and sink_stream fields are
+ * unused.
+ * Set by the driver.
+ */
+#define V4L2_SUBDEV_ROUTE_FL_SOURCE		(1U << 1)
+
+/**
+ * struct v4l2_subdev_route - A route inside a subdev
+ *
+ * @sink_pad: the sink pad index
+ * @sink_stream: the sink stream identifier
+ * @source_pad: the source pad index
+ * @source_stream: the source stream identifier
+ * @flags: route flags V4L2_SUBDEV_ROUTE_FL_*
+ * @reserved: drivers and applications must zero this array
+ */
+struct v4l2_subdev_route {
+	__u32 sink_pad;
+	__u32 sink_stream;
+	__u32 source_pad;
+	__u32 source_stream;
+	__u32 flags;
+	__u32 reserved[5];
+};
+
+/**
+ * struct v4l2_subdev_routing - Subdev routing information
+ *
+ * @which: configuration type (from enum v4l2_subdev_format_whence)
+ * @num_routes: the total number of routes in the routes array
+ * @routes: pointer to the routes array
+ * @reserved: drivers and applications must zero this array
+ */
+struct v4l2_subdev_routing {
+	__u32 which;
+	__u32 num_routes;
+	__u64 routes;
+	__u32 reserved[6];
+};
+
 /* Backwards compatibility define --- to be removed */
 #define v4l2_subdev_edid v4l2_edid
 
@@ -206,6 +256,8 @@ struct v4l2_subdev_capability {
 #define VIDIOC_SUBDEV_S_CROP			_IOWR('V', 60, struct v4l2_subdev_crop)
 #define VIDIOC_SUBDEV_G_SELECTION		_IOWR('V', 61, struct v4l2_subdev_selection)
 #define VIDIOC_SUBDEV_S_SELECTION		_IOWR('V', 62, struct v4l2_subdev_selection)
+#define VIDIOC_SUBDEV_G_ROUTING			_IOWR('V', 38, struct v4l2_subdev_routing)
+#define VIDIOC_SUBDEV_S_ROUTING			_IOWR('V', 39, struct v4l2_subdev_routing)
 /* The following ioctls are identical to the ioctls in videodev2.h */
 #define VIDIOC_SUBDEV_G_STD			_IOR('V', 23, v4l2_std_id)
 #define VIDIOC_SUBDEV_S_STD			_IOW('V', 24, v4l2_std_id)
-- 
2.34.1

[PATCH v15 06/19] media: subdev: add v4l2_subdev_has_pad_interdep()

[Tomi Valkeinen]

Add a v4l2_subdev_has_pad_interdep() helper function which can be used
for media_entity_operations.has_pad_interdep op.

It considers two pads interdependent if there is an active route between
pad0 and pad1.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 31 +++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 18 ++++++++++++++++
 2 files changed, 49 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 1a451351554b..fff17b8536fc 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1046,6 +1046,37 @@ int v4l2_subdev_link_validate(struct media_link *link)
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
 
+bool v4l2_subdev_has_pad_interdep(struct media_entity *entity,
+				  unsigned int pad0, unsigned int pad1)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct v4l2_subdev_krouting *routing;
+	struct v4l2_subdev_state *state;
+	unsigned int i;
+
+	state = v4l2_subdev_lock_and_get_active_state(sd);
+
+	routing = &state->routing;
+
+	for (i = 0; i < routing->num_routes; ++i) {
+		struct v4l2_subdev_route *route = &routing->routes[i];
+
+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
+			continue;
+
+		if ((route->sink_pad == pad0 && route->source_pad == pad1) ||
+		    (route->source_pad == pad0 && route->sink_pad == pad1)) {
+			v4l2_subdev_unlock_state(state);
+			return true;
+		}
+	}
+
+	v4l2_subdev_unlock_state(state);
+
+	return false;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_has_pad_interdep);
+
 struct v4l2_subdev_state *
 __v4l2_subdev_state_alloc(struct v4l2_subdev *sd, const char *lock_name,
 			  struct lock_class_key *lock_key)
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 4934dc9468a8..45c41f4d6a2b 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1241,6 +1241,24 @@ int v4l2_subdev_link_validate_default(struct v4l2_subdev *sd,
  */
 int v4l2_subdev_link_validate(struct media_link *link);
 
+/**
+ * v4l2_subdev_has_pad_interdep - MC has_pad_interdep implementation for subdevs
+ *
+ * @entity: pointer to &struct media_entity
+ * @pad0: pad number for the first pad
+ * @pad1: pad number for the second pad
+ *
+ * This function is an implementation of the
+ * media_entity_operations.has_pad_interdep operation for subdevs that
+ * implement the multiplexed streams API (as indicated by the
+ * V4L2_SUBDEV_FL_STREAMS subdev flag).
+ *
+ * It considers two pads interdependent if there is an active route between pad0
+ * and pad1.
+ */
+bool v4l2_subdev_has_pad_interdep(struct media_entity *entity,
+				  unsigned int pad0, unsigned int pad1);
+
 /**
  * __v4l2_subdev_state_alloc - allocate v4l2_subdev_state
  *
-- 
2.34.1

[PATCH v15 07/19] media: subdev: add v4l2_subdev_set_routing helper()

[Tomi Valkeinen]

Add a helper function to set the subdev routing. The helper can be used
from subdev driver's set_routing op to store the routing table.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 31 +++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 16 ++++++++++++++
 2 files changed, 47 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index fff17b8536fc..3ae4f39a50e4 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -12,6 +12,7 @@
 #include <linux/ioctl.h>
 #include <linux/mm.h>
 #include <linux/module.h>
+#include <linux/overflow.h>
 #include <linux/slab.h>
 #include <linux/types.h>
 #include <linux/version.h>
@@ -1181,6 +1182,36 @@ int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_get_fmt);
 
+int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_state *state,
+			    const struct v4l2_subdev_krouting *routing)
+{
+	struct v4l2_subdev_krouting *dst = &state->routing;
+	const struct v4l2_subdev_krouting *src = routing;
+	struct v4l2_subdev_krouting new_routing = { 0 };
+	size_t bytes;
+
+	if (unlikely(check_mul_overflow(src->num_routes, sizeof(*src->routes),
+					&bytes)))
+		return -EOVERFLOW;
+
+	lockdep_assert_held(state->lock);
+
+	if (src->num_routes > 0) {
+		new_routing.routes = kmemdup(src->routes, bytes, GFP_KERNEL);
+		if (!new_routing.routes)
+			return -ENOMEM;
+	}
+
+	new_routing.num_routes = src->num_routes;
+
+	kfree(dst->routes);
+	*dst = new_routing;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 45c41f4d6a2b..7962e6572bda 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1419,6 +1419,22 @@ v4l2_subdev_lock_and_get_active_state(struct v4l2_subdev *sd)
 int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
 			struct v4l2_subdev_format *format);
 
+/**
+ * v4l2_subdev_set_routing() - Set given routing to subdev state
+ * @sd: The subdevice
+ * @state: The subdevice state
+ * @routing: Routing that will be copied to subdev state
+ *
+ * This will release old routing table (if any) from the state, allocate
+ * enough space for the given routing, and copy the routing.
+ *
+ * This can be used from the subdev driver's set_routing op, after validating
+ * the routing.
+ */
+int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_state *state,
+			    const struct v4l2_subdev_krouting *routing);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
-- 
2.34.1

[PATCH v15 08/19] media: subdev: Add for_each_active_route() macro

[Tomi Valkeinen]

From: Jacopo Mondi <jacopo+renesas@jmondi.org>

Add a for_each_active_route() macro to replace the repeated pattern
of iterating on the active routes of a routing table.

Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 .clang-format                         |  1 +
 drivers/media/v4l2-core/v4l2-subdev.c | 20 ++++++++++++++++++++
 include/media/v4l2-subdev.h           | 13 +++++++++++++
 3 files changed, 34 insertions(+)

diff --git a/.clang-format b/.clang-format
index 1247d54f9e49..31f39ae78f7b 100644
--- a/.clang-format
+++ b/.clang-format
@@ -190,6 +190,7 @@ ForEachMacros:
   - 'for_each_active_dev_scope'
   - 'for_each_active_drhd_unit'
   - 'for_each_active_iommu'
+  - 'for_each_active_route'
   - 'for_each_aggr_pgid'
   - 'for_each_available_child_of_node'
   - 'for_each_bench'
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 3ae4f39a50e4..1049c07d2e49 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1212,6 +1212,26 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing);
 
+struct v4l2_subdev_route *
+__v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting *routing,
+				struct v4l2_subdev_route *route)
+{
+	if (route)
+		++route;
+	else
+		route = &routing->routes[0];
+
+	for (; route < routing->routes + routing->num_routes; ++route) {
+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
+			continue;
+
+		return route;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(__v4l2_subdev_next_active_route);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 7962e6572bda..89e58208e330 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1435,6 +1435,19 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
 			    struct v4l2_subdev_state *state,
 			    const struct v4l2_subdev_krouting *routing);
 
+struct v4l2_subdev_route *
+__v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting *routing,
+				struct v4l2_subdev_route *route);
+
+/**
+ * for_each_active_route - iterate on all active routes of a routing table
+ * @routing: The routing table
+ * @route: The route iterator
+ */
+#define for_each_active_route(routing, route) \
+	for ((route) = NULL;                  \
+	     ((route) = __v4l2_subdev_next_active_route((routing), (route)));)
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
-- 
2.34.1

[PATCH v15 09/19] media: Documentation: add multiplexed streams documentation

[Tomi Valkeinen]

Add documentation related to multiplexed streams.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 .../driver-api/media/v4l2-subdev.rst          |   8 +
 .../userspace-api/media/v4l/dev-subdev.rst    | 173 ++++++++++++++++++
 2 files changed, 181 insertions(+)

diff --git a/Documentation/driver-api/media/v4l2-subdev.rst b/Documentation/driver-api/media/v4l2-subdev.rst
index 6f8d79926aa5..260cfa8c3f3d 100644
--- a/Documentation/driver-api/media/v4l2-subdev.rst
+++ b/Documentation/driver-api/media/v4l2-subdev.rst
@@ -593,6 +593,14 @@ before calling v4l2_subdev_init_finalize():
 
 This shares the driver's private mutex between the controls and the states.
 
+Streams, multiplexed media pads and internal routing
+----------------------------------------------------
+
+A subdevice driver can implement support for multiplexed streams by setting
+the V4L2_SUBDEV_FL_STREAMS subdev flag and implementing support for
+centrally managed subdev active state, routing and stream based
+configuration.
+
 V4L2 sub-device functions and data structures
 ---------------------------------------------
 
diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index a67c2749089a..5075b1828b32 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -503,3 +503,176 @@ source pads.
     :maxdepth: 1
 
     subdev-formats
+
+Streams, multiplexed media pads and internal routing
+----------------------------------------------------
+
+Commonly V4L2 subdevices support only separate video streams, that is, only a
+single stream can pass through a media link and a media pad. Thus each pad
+contains a format configuration for that single stream. In some cases a subdev
+can do stream processing and split a stream into two or compose two streams
+into one, but the inputs and outputs for the subdev are still a single stream
+per pad.
+
+Some hardware, e.g. MIPI CSI-2, support multiplexed streams, that is, multiple
+data streams are transmitted on the same bus, which is represented by a media
+link connecting a transmitter source pad with a sink pad on the receiver. For
+example, a camera sensor can produce two distinct streams, a pixel stream and a
+metadata stream, which are transmitted on the multiplexed data bus, represented
+by a media link which connects the single sensor's source pad with the receiver
+sink pad. The stream-aware receiver will de-multiplex the streams received on
+the its sink pad and allows to route them individually to one of its source
+pads.
+
+Subdevice drivers that support multiplexed streams are compatible with
+non-multiplexed subdev drivers, but, of course, require a routing configuration
+where the link between those two types of drivers contains only a single
+stream.
+
+Understanding streams
+^^^^^^^^^^^^^^^^^^^^^
+
+A stream is a stream of content (e.g. pixel data or metadata) flowing through
+the media pipeline from a source (e.g. a sensor) towards the final sink (e.g. a
+receiver and demultiplexer in a SoC). Each media link carries all the enabled
+streams from one end of the link to the other, and subdevices have routing
+tables which describe how the incoming streams from sink pads are routed to the
+source pads.
+
+A stream ID (often just "stream") is a media link-local identifier for a stream.
+In other words, a particular stream ID must exist on both sides of a media
+link, but another stream ID can be used for the same stream at the other side
+of the subdevice.
+
+A stream at a specific point in the media pipeline is identified with the
+subdev and a (pad, stream) pair. For subdevices that do not support
+multiplexed streams the 'stream' is always 0.
+
+Configuring streams
+^^^^^^^^^^^^^^^^^^^
+
+The configuration of the streams is done individually for each subdevice and
+the validity of the streams between subdevices is validated when the pipeline
+is started.
+
+There are three steps in configuring the streams:
+
+1) Set up links. Connect the pads between subdevices using the :ref:`Media
+Controller API <media_controller>`
+
+2) Routing. The routing table for the subdevice must be set with
+:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that
+setting the routing table will reset all the stream configurations in a media
+entity.
+
+3) Configure streams. Each route endpoint must be configured
+with :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>`.
+
+Multiplexed streams setup example
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A simple example of a multiplexed stream setup might be as follows:
+
+- Two identical sensors (Sensor A and Sensor B). Each sensor has a single source
+  pad (pad 0) which carries two streams, pixel data stream and metadata
+  stream.
+
+- Multiplexer bridge (Bridge). The bridge has two sink pads, connected to the
+  sensors (pads 0, 1), and one source pad (pad 2), which outputs all 4
+  streams.
+
+- Receiver in the SoC (Receiver). The receiver has a single sink pad (pad 0),
+  connected to the bridge, and four source pads (pads 1-4), going to the DMA
+  engine. The receiver demultiplexes the incoming streams to the four source
+  pads.
+
+- Four DMA Engines in the SoC (DMA Engine). Each DMA engine is connected to a
+  single source pad in the receiver.
+
+The sensors, the bridge and the receiver are modeled as V4L2 subdevices,
+exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are
+modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
+
+To configure this pipeline, the userspace must take the following steps:
+
+1) Set up media links between entities: connect the sensors to the bridge,
+bridge to the receiver, and the receiver to the DMA engines. This step does
+not differ from normal non-multiplexed media controller setup.
+
+2) Configure routing.
+
+.. flat-table:: Sensor routing table (identical on both sensors)
+    :header-rows:  1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0 (unused)
+      - 0/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE | V4L2_SUBDEV_ROUTE_FL_SOURCE
+      - Pixel data stream. Source route, i.e. the sink fields are unused.
+    * - 0/0 (unused)
+      - 0/1
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE | V4L2_SUBDEV_ROUTE_FL_SOURCE
+      - Metadata stream. Source route, i.e. the sink fields are unused.
+
+.. flat-table:: Bridge routing table
+    :header-rows:  1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0
+      - 2/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from Sensor A
+    * - 0/1
+      - 2/1
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from Sensor A
+    * - 1/0
+      - 2/2
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from Sensor B
+    * - 1/1
+      - 2/3
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from Sensor B
+
+.. flat-table:: Receiver routing table
+    :header-rows:  1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0
+      - 1/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from Sensor A
+    * - 0/1
+      - 2/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from Sensor A
+    * - 0/2
+      - 3/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from Sensor B
+    * - 0/3
+      - 4/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from Sensor B
+
+3) Configure streams
+
+After configuring the routing table, the next step is configuring the streams.
+This step is similar to configuring the pads in a non-multiplexed streams
+setup, with the difference that we need to configure each (pad, stream) pair
+(i.e. route endpoint) instead of just a pad.
+
+A common way to accomplish this is to start from the sensors and propagate the
+configurations along the stream towards the receiver,
+using :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each
+stream endpoint in each subdev.
-- 
2.34.1

[PATCH v15 10/19] media: subdev: add stream based configuration

[Tomi Valkeinen]

Add support to manage configurations (format, crop, compose) per stream,
instead of per pad. This is accomplished with data structures that hold
an array of all subdev's stream configurations.

The number of streams can vary at runtime based on routing. Every time
the routing is changed, the stream configurations need to be
re-initialized.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 .../v4l/vidioc-subdev-enum-frame-interval.rst |   5 +-
 .../v4l/vidioc-subdev-enum-frame-size.rst     |   5 +-
 .../v4l/vidioc-subdev-enum-mbus-code.rst      |   5 +-
 .../media/v4l/vidioc-subdev-g-crop.rst        |   5 +-
 .../media/v4l/vidioc-subdev-g-fmt.rst         |   5 +-
 .../v4l/vidioc-subdev-g-frame-interval.rst    |   5 +-
 .../media/v4l/vidioc-subdev-g-selection.rst   |   5 +-
 drivers/media/v4l2-core/v4l2-subdev.c         | 154 +++++++++++++++++-
 include/media/v4l2-subdev.h                   |  79 +++++++++
 include/uapi/linux/v4l2-subdev.h              |  28 +++-
 10 files changed, 275 insertions(+), 21 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
index 3703943b412f..8def4c05d3da 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
@@ -92,7 +92,10 @@ multiple pads of the same sub-device is not defined.
       - Frame intervals to be enumerated, from enum
 	:ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
index c25a9896df0e..3ef361c0dca7 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
@@ -97,7 +97,10 @@ information about try formats.
       - Frame sizes to be enumerated, from enum
 	:ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
index 417f1a19bcc4..248f6f9ee7c5 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
@@ -73,7 +73,10 @@ information about the try formats.
       - ``flags``
       - See :ref:`v4l2-subdev-mbus-code-flags`
     * - __u32
-      - ``reserved``\ [7]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [6]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
index bd15c0a5a66b..1d267f7e7991 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
@@ -96,7 +96,10 @@ modified format should be as close as possible to the original request.
       - ``rect``
       - Crop rectangle boundaries, in pixels.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
index 7acdbb939d89..ed253a1e44b7 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
@@ -102,7 +102,10 @@ should be as close as possible to the original request.
       - Definition of an image format, see :c:type:`v4l2_mbus_framefmt` for
 	details.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
index d7fe7543c506..842f962d2aea 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
@@ -90,7 +90,10 @@ the same sub-device is not defined.
       - ``interval``
       - Period, in seconds, between consecutive video frames.
     * - __u32
-      - ``reserved``\ [9]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [8]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
index f9172a42f036..6b629c19168c 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
@@ -94,7 +94,10 @@ Selection targets and flags are documented in
       - ``r``
       - Selection rectangle, in pixels.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 1049c07d2e49..be778e619694 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -159,8 +159,22 @@ static inline int check_pad(struct v4l2_subdev *sd, u32 pad)
 	return 0;
 }
 
-static int check_state_pads(u32 which, struct v4l2_subdev_state *state)
+static int check_state(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
+		       u32 which, u32 pad, u32 stream)
 {
+	if (sd->flags & V4L2_SUBDEV_FL_STREAMS) {
+#if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
+		if (!v4l2_subdev_state_get_stream_format(state, pad, stream))
+			return -EINVAL;
+		return 0;
+#else
+		return -EINVAL;
+#endif
+	}
+
+	if (stream != 0)
+		return -EINVAL;
+
 	if (which == V4L2_SUBDEV_FORMAT_TRY && (!state || !state->pads))
 		return -EINVAL;
 
@@ -175,7 +189,7 @@ static inline int check_format(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	return check_which(format->which) ? : check_pad(sd, format->pad) ? :
-	       check_state_pads(format->which, state);
+	       check_state(sd, state, format->which, format->pad, format->stream);
 }
 
 static int call_get_fmt(struct v4l2_subdev *sd,
@@ -202,7 +216,7 @@ static int call_enum_mbus_code(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	return check_which(code->which) ? : check_pad(sd, code->pad) ? :
-	       check_state_pads(code->which, state) ? :
+	       check_state(sd, state, code->which, code->pad, code->stream) ? :
 	       sd->ops->pad->enum_mbus_code(sd, state, code);
 }
 
@@ -214,7 +228,7 @@ static int call_enum_frame_size(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	return check_which(fse->which) ? : check_pad(sd, fse->pad) ? :
-	       check_state_pads(fse->which, state) ? :
+	       check_state(sd, state, fse->which, fse->pad, fse->stream) ? :
 	       sd->ops->pad->enum_frame_size(sd, state, fse);
 }
 
@@ -249,7 +263,7 @@ static int call_enum_frame_interval(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	return check_which(fie->which) ? : check_pad(sd, fie->pad) ? :
-	       check_state_pads(fie->which, state) ? :
+	       check_state(sd, state, fie->which, fie->pad, fie->stream) ? :
 	       sd->ops->pad->enum_frame_interval(sd, state, fie);
 }
 
@@ -261,7 +275,7 @@ static inline int check_selection(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	return check_which(sel->which) ? : check_pad(sd, sel->pad) ? :
-	       check_state_pads(sel->which, state);
+	       check_state(sd, state, sel->which, sel->pad, sel->stream);
 }
 
 static int call_get_selection(struct v4l2_subdev *sd,
@@ -1095,7 +1109,8 @@ __v4l2_subdev_state_alloc(struct v4l2_subdev *sd, const char *lock_name,
 	else
 		state->lock = &state->_lock;
 
-	if (sd->entity.num_pads) {
+	/* Drivers that support streams do not need the legacy pad config */
+	if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS) && sd->entity.num_pads) {
 		state->pads = kvcalloc(sd->entity.num_pads,
 				       sizeof(*state->pads), GFP_KERNEL);
 		if (!state->pads) {
@@ -1135,6 +1150,7 @@ void __v4l2_subdev_state_free(struct v4l2_subdev_state *state)
 	mutex_destroy(&state->_lock);
 
 	kfree(state->routing.routes);
+	kvfree(state->stream_configs.configs);
 	kvfree(state->pads);
 	kfree(state);
 }
@@ -1164,6 +1180,59 @@ EXPORT_SYMBOL_GPL(v4l2_subdev_cleanup);
 
 #if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
 
+static int
+v4l2_subdev_init_stream_configs(struct v4l2_subdev_stream_configs *stream_configs,
+				const struct v4l2_subdev_krouting *routing)
+{
+	struct v4l2_subdev_stream_configs new_configs = { 0 };
+	struct v4l2_subdev_route *route;
+	u32 format_idx = 0;
+
+	/* Count number of formats needed */
+	for_each_active_route(routing, route) {
+		/*
+		 * Each route needs a format on both ends of the route, except
+		 * for source streams which only need one format.
+		 */
+		new_configs.num_configs +=
+			(route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE) ? 1 : 2;
+	}
+
+	if (new_configs.num_configs) {
+		new_configs.configs = kvcalloc(new_configs.num_configs,
+					       sizeof(*new_configs.configs),
+					       GFP_KERNEL);
+
+		if (!new_configs.configs)
+			return -ENOMEM;
+	}
+
+	/*
+	 * Fill in the 'pad' and stream' value for each item in the array from
+	 * the routing table
+	 */
+	for_each_active_route(routing, route) {
+		u32 idx;
+
+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE)) {
+			idx = format_idx++;
+
+			new_configs.configs[idx].pad = route->sink_pad;
+			new_configs.configs[idx].stream = route->sink_stream;
+		}
+
+		idx = format_idx++;
+
+		new_configs.configs[idx].pad = route->source_pad;
+		new_configs.configs[idx].stream = route->source_stream;
+	}
+
+	kvfree(stream_configs->configs);
+	*stream_configs = new_configs;
+
+	return 0;
+}
+
 int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
 			struct v4l2_subdev_format *format)
 {
@@ -1190,6 +1259,7 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
 	const struct v4l2_subdev_krouting *src = routing;
 	struct v4l2_subdev_krouting new_routing = { 0 };
 	size_t bytes;
+	int r;
 
 	if (unlikely(check_mul_overflow(src->num_routes, sizeof(*src->routes),
 					&bytes)))
@@ -1205,6 +1275,13 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
 
 	new_routing.num_routes = src->num_routes;
 
+	r = v4l2_subdev_init_stream_configs(&state->stream_configs,
+					    &new_routing);
+	if (r) {
+		kfree(new_routing.routes);
+		return r;
+	}
+
 	kfree(dst->routes);
 	*dst = new_routing;
 
@@ -1232,6 +1309,69 @@ __v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting *routing,
 }
 EXPORT_SYMBOL_GPL(__v4l2_subdev_next_active_route);
 
+struct v4l2_mbus_framefmt *
+v4l2_subdev_state_get_stream_format(struct v4l2_subdev_state *state,
+				    unsigned int pad, u32 stream)
+{
+	struct v4l2_subdev_stream_configs *stream_configs;
+	unsigned int i;
+
+	lockdep_assert_held(state->lock);
+
+	stream_configs = &state->stream_configs;
+
+	for (i = 0; i < stream_configs->num_configs; ++i) {
+		if (stream_configs->configs[i].pad == pad &&
+		    stream_configs->configs[i].stream == stream)
+			return &stream_configs->configs[i].fmt;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_stream_format);
+
+struct v4l2_rect *
+v4l2_subdev_state_get_stream_crop(struct v4l2_subdev_state *state,
+				  unsigned int pad, u32 stream)
+{
+	struct v4l2_subdev_stream_configs *stream_configs;
+	unsigned int i;
+
+	lockdep_assert_held(state->lock);
+
+	stream_configs = &state->stream_configs;
+
+	for (i = 0; i < stream_configs->num_configs; ++i) {
+		if (stream_configs->configs[i].pad == pad &&
+		    stream_configs->configs[i].stream == stream)
+			return &stream_configs->configs[i].crop;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_stream_crop);
+
+struct v4l2_rect *
+v4l2_subdev_state_get_stream_compose(struct v4l2_subdev_state *state,
+				     unsigned int pad, u32 stream)
+{
+	struct v4l2_subdev_stream_configs *stream_configs;
+	unsigned int i;
+
+	lockdep_assert_held(state->lock);
+
+	stream_configs = &state->stream_configs;
+
+	for (i = 0; i < stream_configs->num_configs; ++i) {
+		if (stream_configs->configs[i].pad == pad &&
+		    stream_configs->configs[i].stream == stream)
+			return &stream_configs->configs[i].compose;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_stream_compose);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 89e58208e330..d6273ad2eea8 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -695,6 +695,37 @@ struct v4l2_subdev_pad_config {
 	struct v4l2_rect try_compose;
 };
 
+/**
+ * struct v4l2_subdev_stream_config - Used for storing stream configuration.
+ *
+ * @pad: pad number
+ * @stream: stream number
+ * @fmt: &struct v4l2_mbus_framefmt
+ * @crop: &struct v4l2_rect to be used for crop
+ * @compose: &struct v4l2_rect to be used for compose
+ *
+ * This structure stores configuration for a stream.
+ */
+struct v4l2_subdev_stream_config {
+	u32 pad;
+	u32 stream;
+
+	struct v4l2_mbus_framefmt fmt;
+	struct v4l2_rect crop;
+	struct v4l2_rect compose;
+};
+
+/**
+ * struct v4l2_subdev_stream_configs - A collection of stream configs.
+ *
+ * @num_configs: number of entries in @config.
+ * @configs: an array of &struct v4l2_subdev_stream_configs.
+ */
+struct v4l2_subdev_stream_configs {
+	u32 num_configs;
+	struct v4l2_subdev_stream_config *configs;
+};
+
 /**
  * struct v4l2_subdev_krouting - subdev routing table
  *
@@ -715,6 +746,7 @@ struct v4l2_subdev_krouting {
  * @lock: mutex for the state. May be replaced by the user.
  * @pads: &struct v4l2_subdev_pad_config array
  * @routing: routing table for the subdev
+ * @stream_configs: stream configurations (only for V4L2_SUBDEV_FL_STREAMS)
  *
  * This structure only needs to be passed to the pad op if the 'which' field
  * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
@@ -726,6 +758,7 @@ struct v4l2_subdev_state {
 	struct mutex *lock;
 	struct v4l2_subdev_pad_config *pads;
 	struct v4l2_subdev_krouting routing;
+	struct v4l2_subdev_stream_configs stream_configs;
 };
 
 /**
@@ -1448,6 +1481,52 @@ __v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting *routing,
 	for ((route) = NULL;                  \
 	     ((route) = __v4l2_subdev_next_active_route((routing), (route)));)
 
+/**
+ * v4l2_subdev_state_get_stream_format() - Get pointer to a stream format
+ * @state: subdevice state
+ * @pad: pad id
+ * @stream: stream id
+ *
+ * This returns a pointer to &struct v4l2_mbus_framefmt for the given pad +
+ * stream in the subdev state.
+ *
+ * If the state does not contain the given pad + stream, NULL is returned.
+ */
+struct v4l2_mbus_framefmt *
+v4l2_subdev_state_get_stream_format(struct v4l2_subdev_state *state,
+				    unsigned int pad, u32 stream);
+
+/**
+ * v4l2_subdev_state_get_stream_crop() - Get pointer to a stream crop rectangle
+ * @state: subdevice state
+ * @pad: pad id
+ * @stream: stream id
+ *
+ * This returns a pointer to crop rectangle for the given pad + stream in the
+ * subdev state.
+ *
+ * If the state does not contain the given pad + stream, NULL is returned.
+ */
+struct v4l2_rect *
+v4l2_subdev_state_get_stream_crop(struct v4l2_subdev_state *state,
+				  unsigned int pad, u32 stream);
+
+/**
+ * v4l2_subdev_state_get_stream_compose() - Get pointer to a stream compose
+ *					    rectangle
+ * @state: subdevice state
+ * @pad: pad id
+ * @stream: stream id
+ *
+ * This returns a pointer to compose rectangle for the given pad + stream in the
+ * subdev state.
+ *
+ * If the state does not contain the given pad + stream, NULL is returned.
+ */
+struct v4l2_rect *
+v4l2_subdev_state_get_stream_compose(struct v4l2_subdev_state *state,
+				     unsigned int pad, u32 stream);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
index db4af236d7ed..3ba76d0c7a50 100644
--- a/include/uapi/linux/v4l2-subdev.h
+++ b/include/uapi/linux/v4l2-subdev.h
@@ -45,13 +45,15 @@ enum v4l2_subdev_format_whence {
  * @which: format type (from enum v4l2_subdev_format_whence)
  * @pad: pad number, as reported by the media API
  * @format: media bus format (format code and frame size)
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_format {
 	__u32 which;
 	__u32 pad;
 	struct v4l2_mbus_framefmt format;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 /**
@@ -59,13 +61,15 @@ struct v4l2_subdev_format {
  * @which: format type (from enum v4l2_subdev_format_whence)
  * @pad: pad number, as reported by the media API
  * @rect: pad crop rectangle boundaries
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_crop {
 	__u32 which;
 	__u32 pad;
 	struct v4l2_rect rect;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 #define V4L2_SUBDEV_MBUS_CODE_CSC_COLORSPACE	0x00000001
@@ -81,6 +85,7 @@ struct v4l2_subdev_crop {
  * @code: format code (MEDIA_BUS_FMT_ definitions)
  * @which: format type (from enum v4l2_subdev_format_whence)
  * @flags: flags set by the driver, (V4L2_SUBDEV_MBUS_CODE_*)
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_mbus_code_enum {
@@ -89,7 +94,8 @@ struct v4l2_subdev_mbus_code_enum {
 	__u32 code;
 	__u32 which;
 	__u32 flags;
-	__u32 reserved[7];
+	__u32 stream;
+	__u32 reserved[6];
 };
 
 /**
@@ -102,6 +108,7 @@ struct v4l2_subdev_mbus_code_enum {
  * @min_height: minimum frame height, in pixels
  * @max_height: maximum frame height, in pixels
  * @which: format type (from enum v4l2_subdev_format_whence)
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_frame_size_enum {
@@ -113,19 +120,22 @@ struct v4l2_subdev_frame_size_enum {
 	__u32 min_height;
 	__u32 max_height;
 	__u32 which;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 /**
  * struct v4l2_subdev_frame_interval - Pad-level frame rate
  * @pad: pad number, as reported by the media API
  * @interval: frame interval in seconds
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_frame_interval {
 	__u32 pad;
 	struct v4l2_fract interval;
-	__u32 reserved[9];
+	__u32 stream;
+	__u32 reserved[8];
 };
 
 /**
@@ -137,6 +147,7 @@ struct v4l2_subdev_frame_interval {
  * @height: frame height in pixels
  * @interval: frame interval in seconds
  * @which: format type (from enum v4l2_subdev_format_whence)
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_frame_interval_enum {
@@ -147,7 +158,8 @@ struct v4l2_subdev_frame_interval_enum {
 	__u32 height;
 	struct v4l2_fract interval;
 	__u32 which;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 /**
@@ -159,6 +171,7 @@ struct v4l2_subdev_frame_interval_enum {
  *	    defined in v4l2-common.h; V4L2_SEL_TGT_* .
  * @flags: constraint flags, defined in v4l2-common.h; V4L2_SEL_FLAG_*.
  * @r: coordinates of the selection window
+ * @stream: stream number, defined in subdev routing
  * @reserved: for future use, set to zero for now
  *
  * Hardware may use multiple helper windows to process a video stream.
@@ -171,7 +184,8 @@ struct v4l2_subdev_selection {
 	__u32 target;
 	__u32 flags;
 	struct v4l2_rect r;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 /**
-- 
2.34.1

[PATCH v15 11/19] media: subdev: use streams in v4l2_subdev_link_validate()

[Tomi Valkeinen]

Update v4l2_subdev_link_validate() to use routing and streams for
validation.

Instead of just looking at the format on the pad on both ends of the
link, the routing tables are used to collect all the streams going from
the source to the sink over the link, and the streams' formats on both
ends of the link are verified.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 182 +++++++++++++++++++++++---
 1 file changed, 162 insertions(+), 20 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index be778e619694..1cea6ec750c0 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1014,7 +1014,7 @@ int v4l2_subdev_link_validate_default(struct v4l2_subdev *sd,
 EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate_default);
 
 static int
-v4l2_subdev_link_validate_get_format(struct media_pad *pad,
+v4l2_subdev_link_validate_get_format(struct media_pad *pad, u32 stream,
 				     struct v4l2_subdev_format *fmt)
 {
 	if (is_media_entity_v4l2_subdev(pad->entity)) {
@@ -1023,7 +1023,11 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
 
 		fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
 		fmt->pad = pad->index;
-		return v4l2_subdev_call_state_active(sd, pad, get_fmt, fmt);
+		fmt->stream = stream;
+
+		return v4l2_subdev_call(sd, pad, get_fmt,
+					v4l2_subdev_get_locked_active_state(sd),
+					fmt);
 	}
 
 	WARN(pad->entity->function != MEDIA_ENT_F_IO_V4L,
@@ -1033,31 +1037,169 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
 	return -EINVAL;
 }
 
+#if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
+
+static void __v4l2_link_validate_get_streams(struct media_pad *pad,
+					     u64 *streams_mask)
+{
+	struct v4l2_subdev_route *route;
+	struct v4l2_subdev_state *state;
+	struct v4l2_subdev *subdev;
+
+	subdev = media_entity_to_v4l2_subdev(pad->entity);
+
+	*streams_mask = 0;
+
+	state = v4l2_subdev_get_locked_active_state(subdev);
+	if (WARN_ON(!state))
+		return;
+
+	for_each_active_route(&state->routing, route) {
+		u32 route_pad;
+		u32 route_stream;
+
+		if (pad->flags & MEDIA_PAD_FL_SOURCE) {
+			route_pad = route->source_pad;
+			route_stream = route->source_stream;
+		} else {
+			route_pad = route->sink_pad;
+			route_stream = route->sink_stream;
+		}
+
+		if (route_pad != pad->index)
+			continue;
+
+		*streams_mask |= BIT_ULL(route_stream);
+	}
+}
+
+#endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
+
+static void v4l2_link_validate_get_streams(struct media_pad *pad,
+					   u64 *streams_mask)
+{
+	struct v4l2_subdev *subdev = media_entity_to_v4l2_subdev(pad->entity);
+
+	if (!(subdev->flags & V4L2_SUBDEV_FL_STREAMS)) {
+		/* Non-streams subdevs have an implicit stream 0 */
+		*streams_mask = BIT_ULL(0);
+		return;
+	}
+
+#if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
+	__v4l2_link_validate_get_streams(pad, streams_mask);
+#else
+	/* This shouldn't happen */
+	*streams_mask = 0;
+#endif
+}
+
+static int v4l2_subdev_link_validate_locked(struct media_link *link)
+{
+	struct v4l2_subdev *sink_subdev =
+		media_entity_to_v4l2_subdev(link->sink->entity);
+	struct device *dev = sink_subdev->entity.graph_obj.mdev->dev;
+	u64 source_streams_mask;
+	u64 sink_streams_mask;
+	u64 dangling_sink_streams;
+	u32 stream;
+	int ret;
+
+	dev_dbg(dev, "validating link \"%s\":%u -> \"%s\":%u\n",
+		link->source->entity->name, link->source->index,
+		link->sink->entity->name, link->sink->index);
+
+	v4l2_link_validate_get_streams(link->source, &source_streams_mask);
+	v4l2_link_validate_get_streams(link->sink, &sink_streams_mask);
+
+	/*
+	 * It is ok to have more source streams than sink streams as extra
+	 * source streams can just be ignored by the receiver, but having extra
+	 * sink streams is an error as streams must have a source.
+	 */
+	dangling_sink_streams = (source_streams_mask ^ sink_streams_mask) &
+				sink_streams_mask;
+	if (dangling_sink_streams) {
+		dev_err(dev, "Dangling sink streams: mask %#llx\n",
+			dangling_sink_streams);
+		return -EINVAL;
+	}
+
+	/* Validate source and sink stream formats */
+
+	for_each_set_bit(stream, (void *)&sink_streams_mask, 64) {
+		struct v4l2_subdev_format sink_fmt, source_fmt;
+
+		dev_dbg(dev, "validating stream \"%s\":%u:%u -> \"%s\":%u:%u\n",
+			link->source->entity->name, link->source->index, stream,
+			link->sink->entity->name, link->sink->index, stream);
+
+		ret = v4l2_subdev_link_validate_get_format(link->source, stream,
+							   &source_fmt);
+		if (ret < 0) {
+			dev_dbg(dev,
+				"Failed to get format for \"%s\":%u:%u (but that's ok)\n",
+				link->source->entity->name, link->source->index,
+				stream);
+			continue;
+		}
+
+		ret = v4l2_subdev_link_validate_get_format(link->sink, stream,
+							   &sink_fmt);
+		if (ret < 0) {
+			dev_dbg(dev,
+				"Failed to get format for \"%s\":%u:%u (but that's ok)\n",
+				link->sink->entity->name, link->sink->index,
+				stream);
+			continue;
+		}
+
+		/* TODO: add stream number to link_validate() */
+		ret = v4l2_subdev_call(sink_subdev, pad, link_validate, link,
+				       &source_fmt, &sink_fmt);
+		if (!ret)
+			continue;
+
+		if (ret != -ENOIOCTLCMD)
+			return ret;
+
+		ret = v4l2_subdev_link_validate_default(sink_subdev, link,
+							&source_fmt, &sink_fmt);
+
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
+
 int v4l2_subdev_link_validate(struct media_link *link)
 {
-	struct v4l2_subdev *sink;
-	struct v4l2_subdev_format sink_fmt, source_fmt;
-	int rval;
+	struct v4l2_subdev *source_sd, *sink_sd;
+	struct v4l2_subdev_state *source_state, *sink_state;
+	int ret;
 
-	rval = v4l2_subdev_link_validate_get_format(
-		link->source, &source_fmt);
-	if (rval < 0)
-		return 0;
+	sink_sd = media_entity_to_v4l2_subdev(link->sink->entity);
+	source_sd = media_entity_to_v4l2_subdev(link->source->entity);
 
-	rval = v4l2_subdev_link_validate_get_format(
-		link->sink, &sink_fmt);
-	if (rval < 0)
-		return 0;
+	sink_state = v4l2_subdev_get_unlocked_active_state(sink_sd);
+	source_state = v4l2_subdev_get_unlocked_active_state(source_sd);
 
-	sink = media_entity_to_v4l2_subdev(link->sink->entity);
+	if (sink_state)
+		v4l2_subdev_lock_state(sink_state);
 
-	rval = v4l2_subdev_call(sink, pad, link_validate, link,
-				&source_fmt, &sink_fmt);
-	if (rval != -ENOIOCTLCMD)
-		return rval;
+	if (source_state)
+		v4l2_subdev_lock_state(source_state);
 
-	return v4l2_subdev_link_validate_default(
-		sink, link, &source_fmt, &sink_fmt);
+	ret = v4l2_subdev_link_validate_locked(link);
+
+	if (sink_state)
+		v4l2_subdev_unlock_state(sink_state);
+
+	if (source_state)
+		v4l2_subdev_unlock_state(source_state);
+
+	return ret;
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
 
-- 
2.34.1

[PATCH v15 12/19] media: subdev: add "opposite" stream helper funcs

[Tomi Valkeinen]

Add two helper functions to make dealing with streams easier:

v4l2_subdev_routing_find_opposite_end - given a routing table and a pad
+ stream, return the pad + stream on the opposite side of the subdev.

v4l2_subdev_state_get_opposite_stream_format - return a pointer to the
format on the pad + stream on the opposite side from the given pad +
stream.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
Reviewed-by: Jacopo Mondi <jacopo@jmondi.org>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 49 +++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 36 ++++++++++++++++++++
 2 files changed, 85 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 1cea6ec750c0..7d2d8e8d3aea 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1514,6 +1514,55 @@ v4l2_subdev_state_get_stream_compose(struct v4l2_subdev_state *state,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_stream_compose);
 
+int v4l2_subdev_routing_find_opposite_end(const struct v4l2_subdev_krouting *routing,
+					  u32 pad, u32 stream, u32 *other_pad,
+					  u32 *other_stream)
+{
+	unsigned int i;
+
+	for (i = 0; i < routing->num_routes; ++i) {
+		struct v4l2_subdev_route *route = &routing->routes[i];
+
+		if (route->source_pad == pad &&
+		    route->source_stream == stream) {
+			if (other_pad)
+				*other_pad = route->sink_pad;
+			if (other_stream)
+				*other_stream = route->sink_stream;
+			return 0;
+		}
+
+		if (route->sink_pad == pad && route->sink_stream == stream) {
+			if (other_pad)
+				*other_pad = route->source_pad;
+			if (other_stream)
+				*other_stream = route->source_stream;
+			return 0;
+		}
+	}
+
+	return -EINVAL;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_routing_find_opposite_end);
+
+struct v4l2_mbus_framefmt *
+v4l2_subdev_state_get_opposite_stream_format(struct v4l2_subdev_state *state,
+					     u32 pad, u32 stream)
+{
+	u32 other_pad, other_stream;
+	int ret;
+
+	ret = v4l2_subdev_routing_find_opposite_end(&state->routing,
+						    pad, stream,
+						    &other_pad, &other_stream);
+	if (ret)
+		return NULL;
+
+	return v4l2_subdev_state_get_stream_format(state, other_pad,
+						   other_stream);
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_opposite_stream_format);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index d6273ad2eea8..6f4719e28ad1 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1527,6 +1527,42 @@ struct v4l2_rect *
 v4l2_subdev_state_get_stream_compose(struct v4l2_subdev_state *state,
 				     unsigned int pad, u32 stream);
 
+/**
+ * v4l2_subdev_routing_find_opposite_end() - Find the opposite stream
+ * @routing: routing used to find the opposite side
+ * @pad: pad id
+ * @stream: stream id
+ * @other_pad: pointer used to return the opposite pad
+ * @other_stream: pointer used to return the opposite stream
+ *
+ * This function uses the routing table to find the pad + stream which is
+ * opposite the given pad + stream.
+ *
+ * @other_pad and/or @other_stream can be NULL if the caller does not need the
+ * value.
+ *
+ * Returns 0 on success, or -EINVAL if no matching route is found.
+ */
+int v4l2_subdev_routing_find_opposite_end(const struct v4l2_subdev_krouting *routing,
+					  u32 pad, u32 stream, u32 *other_pad,
+					  u32 *other_stream);
+
+/**
+ * v4l2_subdev_state_get_opposite_stream_format() - Get pointer to opposite
+ *                                                  stream format
+ * @state: subdevice state
+ * @pad: pad id
+ * @stream: stream id
+ *
+ * This returns a pointer to &struct v4l2_mbus_framefmt for the pad + stream
+ * that is opposite the given pad + stream in the subdev state.
+ *
+ * If the state does not contain the given pad + stream, NULL is returned.
+ */
+struct v4l2_mbus_framefmt *
+v4l2_subdev_state_get_opposite_stream_format(struct v4l2_subdev_state *state,
+					     u32 pad, u32 stream);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
-- 
2.34.1

[PATCH v15 13/19] media: subdev: add streams to v4l2_subdev_get_fmt() helper function

[Tomi Valkeinen]

Add streams support to v4l2_subdev_get_fmt() helper function. Subdev
drivers that do not need to do anything special in their get_fmt op can
use this helper directly for v4l2_subdev_pad_ops.get_fmt.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 7d2d8e8d3aea..2f2ede00e60f 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1380,10 +1380,14 @@ int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
 {
 	struct v4l2_mbus_framefmt *fmt;
 
-	if (format->pad >= sd->entity.num_pads)
-		return -EINVAL;
+	if (sd->flags & V4L2_SUBDEV_FL_STREAMS)
+		fmt = v4l2_subdev_state_get_stream_format(state, format->pad,
+							  format->stream);
+	else if (format->pad < sd->entity.num_pads && format->stream == 0)
+		fmt = v4l2_subdev_get_pad_format(sd, state, format->pad);
+	else
+		fmt = NULL;
 
-	fmt = v4l2_subdev_get_pad_format(sd, state, format->pad);
 	if (!fmt)
 		return -EINVAL;
 
-- 
2.34.1

[PATCH v15 14/19] media: subdev: add v4l2_subdev_set_routing_with_fmt() helper

[Tomi Valkeinen]

v4l2_subdev_set_routing_with_fmt() is the same as
v4l2_subdev_set_routing(), but additionally initializes all the streams
with the given format.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
Reviewed-by: Jacopo Mondi <jacopo@jmondi.org>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 22 ++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 16 ++++++++++++++++
 2 files changed, 38 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 2f2ede00e60f..42cd95488f97 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1455,6 +1455,28 @@ __v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting *routing,
 }
 EXPORT_SYMBOL_GPL(__v4l2_subdev_next_active_route);
 
+int v4l2_subdev_set_routing_with_fmt(struct v4l2_subdev *sd,
+				     struct v4l2_subdev_state *state,
+				     struct v4l2_subdev_krouting *routing,
+				     const struct v4l2_mbus_framefmt *fmt)
+{
+	struct v4l2_subdev_stream_configs *stream_configs;
+	unsigned int i;
+	int ret;
+
+	ret = v4l2_subdev_set_routing(sd, state, routing);
+	if (ret)
+		return ret;
+
+	stream_configs = &state->stream_configs;
+
+	for (i = 0; i < stream_configs->num_configs; ++i)
+		stream_configs->configs[i].fmt = *fmt;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing_with_fmt);
+
 struct v4l2_mbus_framefmt *
 v4l2_subdev_state_get_stream_format(struct v4l2_subdev_state *state,
 				    unsigned int pad, u32 stream)
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 6f4719e28ad1..020ad79182cc 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1481,6 +1481,22 @@ __v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting *routing,
 	for ((route) = NULL;                  \
 	     ((route) = __v4l2_subdev_next_active_route((routing), (route)));)
 
+/**
+ * v4l2_subdev_set_routing_with_fmt() - Set given routing and format to subdev
+ *					state
+ * @sd: The subdevice
+ * @state: The subdevice state
+ * @routing: Routing that will be copied to subdev state
+ * @fmt: Format used to initialize all the streams
+ *
+ * This is the same as v4l2_subdev_set_routing, but additionally initializes
+ * all the streams using the given format.
+ */
+int v4l2_subdev_set_routing_with_fmt(struct v4l2_subdev *sd,
+				     struct v4l2_subdev_state *state,
+				     struct v4l2_subdev_krouting *routing,
+				     const struct v4l2_mbus_framefmt *fmt);
+
 /**
  * v4l2_subdev_state_get_stream_format() - Get pointer to a stream format
  * @state: subdevice state
-- 
2.34.1

[PATCH v15 15/19] media: subdev: add v4l2_subdev_routing_validate() helper

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Add a v4l2_subdev_routing_validate() helper for verifying routing for
common cases like only allowing non-overlapping 1-to-1 streams.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Jacopo Mondi <jacopo@jmondi.org>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 102 ++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           |  39 ++++++++++
 2 files changed, 141 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 42cd95488f97..7fabb2079b3f 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1589,6 +1589,108 @@ v4l2_subdev_state_get_opposite_stream_format(struct v4l2_subdev_state *state,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_opposite_stream_format);
 
+int v4l2_subdev_routing_validate(struct v4l2_subdev *sd,
+				 const struct v4l2_subdev_krouting *routing,
+				 enum v4l2_subdev_routing_restriction disallow)
+{
+	u32 *remote_pads = NULL;
+	unsigned int i, j;
+	int ret = -EINVAL;
+
+	if (disallow & V4L2_SUBDEV_ROUTING_NO_STREAM_MIX) {
+		remote_pads = kcalloc(sd->entity.num_pads, sizeof(*remote_pads),
+				      GFP_KERNEL);
+		if (!remote_pads)
+			return -ENOMEM;
+
+		for (i = 0; i < sd->entity.num_pads; ++i)
+			remote_pads[i] = U32_MAX;
+	}
+
+	for (i = 0; i < routing->num_routes; ++i) {
+		const struct v4l2_subdev_route *route = &routing->routes[i];
+
+		/* Validate the sink and source pad numbers. */
+		if (route->sink_pad >= sd->entity.num_pads ||
+		    !(sd->entity.pads[route->sink_pad].flags & MEDIA_PAD_FL_SINK)) {
+			dev_dbg(sd->dev, "route %u sink (%u) is not a sink pad\n",
+				i, route->sink_pad);
+			goto out;
+		}
+
+		if (route->source_pad >= sd->entity.num_pads ||
+		    !(sd->entity.pads[route->source_pad].flags & MEDIA_PAD_FL_SOURCE)) {
+			dev_dbg(sd->dev, "route %u source (%u) is not a source pad\n",
+				i, route->source_pad);
+			goto out;
+		}
+
+		/*
+		 * V4L2_SUBDEV_ROUTING_NO_STREAM_MIX: Streams on the same pad
+		 * may not be routed to streams on different pads.
+		 */
+		if (disallow & V4L2_SUBDEV_ROUTING_NO_STREAM_MIX) {
+			if (remote_pads[route->sink_pad] != U32_MAX &&
+			    remote_pads[route->sink_pad] != route->source_pad) {
+				dev_dbg(sd->dev,
+					"route %u attempts to mix %s streams\n",
+					i, "sink");
+				goto out;
+			}
+
+			if (remote_pads[route->source_pad] != U32_MAX &&
+			    remote_pads[route->source_pad] != route->sink_pad) {
+				dev_dbg(sd->dev,
+					"route %u attempts to mix %s streams\n",
+					i, "source");
+				goto out;
+			}
+
+			remote_pads[route->sink_pad] = route->source_pad;
+			remote_pads[route->source_pad] = route->sink_pad;
+		}
+
+		for (j = i + 1; j < routing->num_routes; ++j) {
+			const struct v4l2_subdev_route *r = &routing->routes[j];
+
+			/*
+			 * V4L2_SUBDEV_ROUTING_NO_1_TO_N: No two routes can
+			 * originate from the same (sink) stream.
+			 */
+			if ((disallow & V4L2_SUBDEV_ROUTING_NO_1_TO_N) &&
+			    route->sink_pad == r->sink_pad &&
+			    route->sink_stream == r->sink_stream) {
+				dev_dbg(sd->dev,
+					"routes %u and %u originate from same sink (%u/%u)\n",
+					i, j, route->sink_pad,
+					route->sink_stream);
+				goto out;
+			}
+
+			/*
+			 * V4L2_SUBDEV_ROUTING_NO_N_TO_1: No two routes can end
+			 * at the same (source) stream.
+			 */
+			if ((disallow & V4L2_SUBDEV_ROUTING_NO_N_TO_1) &&
+			    route->source_pad == r->source_pad &&
+			    route->source_stream == r->source_stream) {
+				dev_dbg(sd->dev,
+					"routes %u and %u end at same source (%u/%u)\n",
+					i, j, route->source_pad,
+					route->source_stream);
+				goto out;
+			}
+		}
+	}
+
+	ret = 0;
+
+out:
+	kfree(remote_pads);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_routing_validate);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 020ad79182cc..6661887536bf 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1579,6 +1579,45 @@ struct v4l2_mbus_framefmt *
 v4l2_subdev_state_get_opposite_stream_format(struct v4l2_subdev_state *state,
 					     u32 pad, u32 stream);
 
+/**
+ * enum v4l2_subdev_routing_restriction - Subdevice internal routing restrictions
+ *
+ * @V4L2_SUBDEV_ROUTING_NO_1_TO_N:
+ *	an input stream may not be routed to multiple output streams (stream
+ *	duplication)
+ * @V4L2_SUBDEV_ROUTING_NO_N_TO_1:
+ *	multiple input streams may not be routed to the same output stream
+ *	(stream merging)
+ * @V4L2_SUBDEV_ROUTING_NO_STREAM_MIX:
+ *	streams on the same pad may not be routed to streams on different pads
+ * @V4L2_SUBDEV_ROUTING_ONLY_1_TO_1:
+ *	only non-overlapping 1-to-1 stream routing is allowed (a combination of
+ *	@V4L2_SUBDEV_ROUTING_NO_1_TO_N and @V4L2_SUBDEV_ROUTING_NO_N_TO_1)
+ */
+enum v4l2_subdev_routing_restriction {
+	V4L2_SUBDEV_ROUTING_NO_1_TO_N = BIT(0),
+	V4L2_SUBDEV_ROUTING_NO_N_TO_1 = BIT(1),
+	V4L2_SUBDEV_ROUTING_NO_STREAM_MIX = BIT(2),
+	V4L2_SUBDEV_ROUTING_ONLY_1_TO_1 =
+		V4L2_SUBDEV_ROUTING_NO_1_TO_N |
+		V4L2_SUBDEV_ROUTING_NO_N_TO_1,
+};
+
+/**
+ * v4l2_subdev_routing_validate() - Verify that routes comply with driver
+ *				    constraints
+ * @sd: The subdevice
+ * @routing: Routing to verify
+ * @disallow: Restrictions on routes
+ *
+ * This verifies that the given routing complies with the @disallow constraints.
+ *
+ * Returns 0 on success, error value otherwise.
+ */
+int v4l2_subdev_routing_validate(struct v4l2_subdev *sd,
+				 const struct v4l2_subdev_krouting *routing,
+				 enum v4l2_subdev_routing_restriction disallow);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
-- 
2.34.1

[PATCH v15 16/19] media: v4l2-subdev: Add v4l2_subdev_state_xlate_streams() helper

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Add a helper function to translate streams between two pads of a subdev,
using the subdev's internal routing table.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 26 ++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 23 +++++++++++++++++++++++
 2 files changed, 49 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 7fabb2079b3f..cc8666c5069b 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1589,6 +1589,32 @@ v4l2_subdev_state_get_opposite_stream_format(struct v4l2_subdev_state *state,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_opposite_stream_format);
 
+u64 v4l2_subdev_state_xlate_streams(const struct v4l2_subdev_state *state,
+				    u32 pad0, u32 pad1, u64 *streams)
+{
+	const struct v4l2_subdev_krouting *routing = &state->routing;
+	struct v4l2_subdev_route *route;
+	u64 streams0 = 0;
+	u64 streams1 = 0;
+
+	for_each_active_route(routing, route) {
+		if (route->sink_pad == pad0 && route->source_pad == pad1 &&
+		    (*streams & BIT_ULL(route->sink_stream))) {
+			streams0 |= BIT_ULL(route->sink_stream);
+			streams1 |= BIT_ULL(route->source_stream);
+		}
+		if (route->source_pad == pad0 && route->sink_pad == pad1 &&
+		    (*streams & BIT_ULL(route->source_stream))) {
+			streams0 |= BIT_ULL(route->source_stream);
+			streams1 |= BIT_ULL(route->sink_stream);
+		}
+	}
+
+	*streams = streams0;
+	return streams1;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_state_xlate_streams);
+
 int v4l2_subdev_routing_validate(struct v4l2_subdev *sd,
 				 const struct v4l2_subdev_krouting *routing,
 				 enum v4l2_subdev_routing_restriction disallow)
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 6661887536bf..8c002d65e08e 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1579,6 +1579,29 @@ struct v4l2_mbus_framefmt *
 v4l2_subdev_state_get_opposite_stream_format(struct v4l2_subdev_state *state,
 					     u32 pad, u32 stream);
 
+/**
+ * v4l2_subdev_state_xlate_streams() - Translate streams from one pad to another
+ *
+ * @state: Subdevice state
+ * @pad0: The first pad
+ * @pad1: The second pad
+ * @streams: Streams bitmask on the first pad
+ *
+ * Streams on sink pads of a subdev are routed to source pads as expressed in
+ * the subdev state routing table. Stream numbers don't necessarily match on
+ * the sink and source side of a route. This function translates stream numbers
+ * on @pad0, expressed as a bitmask in @streams, to the corresponding streams
+ * on @pad1 using the routing table from the @state. It returns the stream mask
+ * on @pad1, and updates @streams with the streams that have been found in the
+ * routing table.
+ *
+ * @pad0 and @pad1 must be a sink and a source, in any order.
+ *
+ * Return: The bitmask of streams of @pad1 that are routed to @streams on @pad0.
+ */
+u64 v4l2_subdev_state_xlate_streams(const struct v4l2_subdev_state *state,
+				    u32 pad0, u32 pad1, u64 *streams);
+
 /**
  * enum v4l2_subdev_routing_restriction - Subdevice internal routing restrictions
  *
-- 
2.34.1

[PATCH v15 17/19] media: v4l2-subdev: Add subdev .(enable|disable)_streams() operations

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Add two new subdev pad operations, .enable_streams() and
.disable_streams(), to allow control of individual streams per pad. This
is a superset of what the video .s_stream() operation implements.

To help with handling of backward compatibility, add two wrapper
functions around those operations, and require their usage in drivers.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Jacopo Mondi <jacopo@jmondi.org>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 224 ++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           |  85 ++++++++++
 2 files changed, 309 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index cc8666c5069b..101c1cfc0123 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1717,6 +1717,230 @@ int v4l2_subdev_routing_validate(struct v4l2_subdev *sd,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_routing_validate);
 
+static int v4l2_subdev_enable_streams_fallback(struct v4l2_subdev *sd, u32 pad,
+					       u64 streams_mask)
+{
+	struct device *dev = sd->entity.graph_obj.mdev->dev;
+	unsigned int i;
+	int ret;
+
+	/*
+	 * The subdev doesn't implement pad-based stream enable, fall back
+	 * on the .s_stream() operation. This can only be done for subdevs that
+	 * have a single source pad, as sd->enabled_streams is global to the
+	 * subdev.
+	 */
+	if (!(sd->entity.pads[pad].flags & MEDIA_PAD_FL_SOURCE))
+		return -EOPNOTSUPP;
+
+	for (i = 0; i < sd->entity.num_pads; ++i) {
+		if (i != pad && sd->entity.pads[i].flags & MEDIA_PAD_FL_SOURCE)
+			return -EOPNOTSUPP;
+	}
+
+	if (sd->enabled_streams & streams_mask) {
+		dev_dbg(dev, "set of streams %#llx already enabled on %s:%u\n",
+			streams_mask, sd->entity.name, pad);
+		return -EALREADY;
+	}
+
+	/* Start streaming when the first streams are enabled. */
+	if (!sd->enabled_streams) {
+		ret = v4l2_subdev_call(sd, video, s_stream, 1);
+		if (ret)
+			return ret;
+	}
+
+	sd->enabled_streams |= streams_mask;
+
+	return 0;
+}
+
+int v4l2_subdev_enable_streams(struct v4l2_subdev *sd, u32 pad,
+			       u64 streams_mask)
+{
+	struct device *dev = sd->entity.graph_obj.mdev->dev;
+	struct v4l2_subdev_state *state;
+	u64 found_streams = 0;
+	unsigned int i;
+	int ret;
+
+	/* A few basic sanity checks first. */
+	if (pad >= sd->entity.num_pads)
+		return -EINVAL;
+
+	if (!streams_mask)
+		return 0;
+
+	/* Fallback on .s_stream() if .enable_streams() isn't available. */
+	if (!sd->ops->pad || !sd->ops->pad->enable_streams)
+		return v4l2_subdev_enable_streams_fallback(sd, pad,
+							   streams_mask);
+
+	state = v4l2_subdev_lock_and_get_active_state(sd);
+
+	/*
+	 * Verify that the requested streams exist and that they are not
+	 * already enabled.
+	 */
+	for (i = 0; i < state->stream_configs.num_configs; ++i) {
+		struct v4l2_subdev_stream_config *cfg =
+			&state->stream_configs.configs[i];
+
+		if (cfg->pad != pad || !(streams_mask & BIT_ULL(cfg->stream)))
+			continue;
+
+		found_streams |= BIT_ULL(cfg->stream);
+
+		if (cfg->enabled) {
+			dev_dbg(dev, "stream %u already enabled on %s:%u\n",
+				cfg->stream, sd->entity.name, pad);
+			ret = -EALREADY;
+			goto done;
+		}
+	}
+
+	if (found_streams != streams_mask) {
+		dev_dbg(dev, "streams 0x%llx not found on %s:%u\n",
+			streams_mask & ~found_streams, sd->entity.name, pad);
+		ret = -EINVAL;
+		goto done;
+	}
+
+	/* Call the .enable_streams() operation. */
+	ret = v4l2_subdev_call(sd, pad, enable_streams, state, pad,
+			       streams_mask);
+	if (ret)
+		goto done;
+
+	/* Mark the streams as enabled. */
+	for (i = 0; i < state->stream_configs.num_configs; ++i) {
+		struct v4l2_subdev_stream_config *cfg =
+			&state->stream_configs.configs[i];
+
+		if (cfg->pad == pad && (streams_mask & BIT_ULL(cfg->stream)))
+			cfg->enabled = true;
+	}
+
+done:
+	v4l2_subdev_unlock_state(state);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_enable_streams);
+
+static int v4l2_subdev_disable_streams_fallback(struct v4l2_subdev *sd, u32 pad,
+						u64 streams_mask)
+{
+	struct device *dev = sd->entity.graph_obj.mdev->dev;
+	unsigned int i;
+	int ret;
+
+	/*
+	 * If the subdev doesn't implement pad-based stream enable, fall  back
+	 * on the .s_stream() operation. This can only be done for subdevs that
+	 * have a single source pad, as sd->enabled_streams is global to the
+	 * subdev.
+	 */
+	if (!(sd->entity.pads[pad].flags & MEDIA_PAD_FL_SOURCE))
+		return -EOPNOTSUPP;
+
+	for (i = 0; i < sd->entity.num_pads; ++i) {
+		if (i != pad && sd->entity.pads[i].flags & MEDIA_PAD_FL_SOURCE)
+			return -EOPNOTSUPP;
+	}
+
+	if ((sd->enabled_streams & streams_mask) != streams_mask) {
+		dev_dbg(dev, "set of streams %#llx already disabled on %s:%u\n",
+			streams_mask, sd->entity.name, pad);
+		return -EALREADY;
+	}
+
+	/* Stop streaming when the last streams are disabled. */
+	if (!(sd->enabled_streams & ~streams_mask)) {
+		ret = v4l2_subdev_call(sd, video, s_stream, 0);
+		if (ret)
+			return ret;
+	}
+
+	sd->enabled_streams &= ~streams_mask;
+
+	return 0;
+}
+
+int v4l2_subdev_disable_streams(struct v4l2_subdev *sd, u32 pad,
+				u64 streams_mask)
+{
+	struct device *dev = sd->entity.graph_obj.mdev->dev;
+	struct v4l2_subdev_state *state;
+	u64 found_streams = 0;
+	unsigned int i;
+	int ret;
+
+	/* A few basic sanity checks first. */
+	if (pad >= sd->entity.num_pads)
+		return -EINVAL;
+
+	if (!streams_mask)
+		return 0;
+
+	/* Fallback on .s_stream() if .disable_streams() isn't available. */
+	if (!sd->ops->pad || !sd->ops->pad->disable_streams)
+		return v4l2_subdev_disable_streams_fallback(sd, pad,
+							    streams_mask);
+
+	state = v4l2_subdev_lock_and_get_active_state(sd);
+
+	/*
+	 * Verify that the requested streams exist and that they are not
+	 * already disabled.
+	 */
+	for (i = 0; i < state->stream_configs.num_configs; ++i) {
+		struct v4l2_subdev_stream_config *cfg =
+			&state->stream_configs.configs[i];
+
+		if (cfg->pad != pad || !(streams_mask & BIT_ULL(cfg->stream)))
+			continue;
+
+		found_streams |= BIT_ULL(cfg->stream);
+
+		if (!cfg->enabled) {
+			dev_dbg(dev, "stream %u already disabled on %s:%u\n",
+				cfg->stream, sd->entity.name, pad);
+			ret = -EALREADY;
+			goto done;
+		}
+	}
+
+	if (found_streams != streams_mask) {
+		dev_dbg(dev, "streams 0x%llx not found on %s:%u\n",
+			streams_mask & ~found_streams, sd->entity.name, pad);
+		ret = -EINVAL;
+		goto done;
+	}
+
+	/* Call the .disable_streams() operation. */
+	ret = v4l2_subdev_call(sd, pad, disable_streams, state, pad,
+			       streams_mask);
+	if (ret)
+		goto done;
+
+	/* Mark the streams as disabled. */
+	for (i = 0; i < state->stream_configs.num_configs; ++i) {
+		struct v4l2_subdev_stream_config *cfg =
+			&state->stream_configs.configs[i];
+
+		if (cfg->pad == pad && (streams_mask & BIT_ULL(cfg->stream)))
+			cfg->enabled = false;
+	}
+
+done:
+	v4l2_subdev_unlock_state(state);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_disable_streams);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 8c002d65e08e..bff824367e0b 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -700,6 +700,7 @@ struct v4l2_subdev_pad_config {
  *
  * @pad: pad number
  * @stream: stream number
+ * @enabled: has the stream been enabled with v4l2_subdev_enable_stream()
  * @fmt: &struct v4l2_mbus_framefmt
  * @crop: &struct v4l2_rect to be used for crop
  * @compose: &struct v4l2_rect to be used for compose
@@ -709,6 +710,7 @@ struct v4l2_subdev_pad_config {
 struct v4l2_subdev_stream_config {
 	u32 pad;
 	u32 stream;
+	bool enabled;
 
 	struct v4l2_mbus_framefmt fmt;
 	struct v4l2_rect crop;
@@ -814,6 +816,18 @@ struct v4l2_subdev_state {
  *
  * @set_routing: enable or disable data connection routes described in the
  *		 subdevice routing table.
+ *
+ * @enable_streams: Enable the streams defined in streams_mask on the given
+ *	source pad. Subdevs that implement this operation must use the active
+ *	state management provided by the subdev core (enabled through a call to
+ *	v4l2_subdev_init_finalize() at initialization time). Do not call
+ *	directly, use v4l2_subdev_enable_streams() instead.
+ *
+ * @disable_streams: Disable the streams defined in streams_mask on the given
+ *	source pad. Subdevs that implement this operation must use the active
+ *	state management provided by the subdev core (enabled through a call to
+ *	v4l2_subdev_init_finalize() at initialization time). Do not call
+ *	directly, use v4l2_subdev_disable_streams() instead.
  */
 struct v4l2_subdev_pad_ops {
 	int (*init_cfg)(struct v4l2_subdev *sd,
@@ -860,6 +874,12 @@ struct v4l2_subdev_pad_ops {
 			   struct v4l2_subdev_state *state,
 			   enum v4l2_subdev_format_whence which,
 			   struct v4l2_subdev_krouting *route);
+	int (*enable_streams)(struct v4l2_subdev *sd,
+			      struct v4l2_subdev_state *state, u32 pad,
+			      u64 streams_mask);
+	int (*disable_streams)(struct v4l2_subdev *sd,
+			       struct v4l2_subdev_state *state, u32 pad,
+			       u64 streams_mask);
 };
 
 /**
@@ -1005,6 +1025,10 @@ struct v4l2_subdev_platform_data {
  * @active_state: Active state for the subdev (NULL for subdevs tracking the
  *		  state internally). Initialized by calling
  *		  v4l2_subdev_init_finalize().
+ * @enabled_streams: Bitmask of enabled streams used by
+ *		     v4l2_subdev_enable_streams() and
+ *		     v4l2_subdev_disable_streams() helper functions for fallback
+ *		     cases.
  *
  * Each instance of a subdev driver should create this struct, either
  * stand-alone or embedded in a larger struct.
@@ -1050,6 +1074,7 @@ struct v4l2_subdev {
 	 * doesn't support it.
 	 */
 	struct v4l2_subdev_state *active_state;
+	u64 enabled_streams;
 };
 
 
@@ -1641,6 +1666,66 @@ int v4l2_subdev_routing_validate(struct v4l2_subdev *sd,
 				 const struct v4l2_subdev_krouting *routing,
 				 enum v4l2_subdev_routing_restriction disallow);
 
+/**
+ * v4l2_subdev_enable_streams() - Enable streams on a pad
+ * @sd: The subdevice
+ * @pad: The pad
+ * @streams_mask: Bitmask of streams to enable
+ *
+ * This function enables streams on a source @pad of a subdevice. The pad is
+ * identified by its index, while the streams are identified by the
+ * @streams_mask bitmask. This allows enabling multiple streams on a pad at
+ * once.
+ *
+ * Enabling a stream that is already enabled isn't allowed. If @streams_mask
+ * contains an already enabled stream, this function returns -EALREADY without
+ * performing any operation.
+ *
+ * Per-stream enable is only available for subdevs that implement the
+ * .enable_streams() and .disable_streams() operations. For other subdevs, this
+ * function implements a best-effort compatibility by calling the .s_stream()
+ * operation, limited to subdevs that have a single source pad.
+ *
+ * Return:
+ * * 0: Success
+ * * -EALREADY: One of the streams in streams_mask is already enabled
+ * * -EINVAL: The pad index is invalid, or doesn't correspond to a source pad
+ * * -EOPNOTSUPP: Falling back to the legacy .s_stream() operation is
+ *   impossible because the subdev has multiple source pads
+ */
+int v4l2_subdev_enable_streams(struct v4l2_subdev *sd, u32 pad,
+			       u64 streams_mask);
+
+/**
+ * v4l2_subdev_disable_streams() - Disable streams on a pad
+ * @sd: The subdevice
+ * @pad: The pad
+ * @streams_mask: Bitmask of streams to disable
+ *
+ * This function disables streams on a source @pad of a subdevice. The pad is
+ * identified by its index, while the streams are identified by the
+ * @streams_mask bitmask. This allows disabling multiple streams on a pad at
+ * once.
+ *
+ * Disabling a streams that is not enabled isn't allowed. If @streams_mask
+ * contains a disabled stream, this function returns -EALREADY without
+ * performing any operation.
+ *
+ * Per-stream disable is only available for subdevs that implement the
+ * .enable_streams() and .disable_streams() operations. For other subdevs, this
+ * function implements a best-effort compatibility by calling the .s_stream()
+ * operation, limited to subdevs that have a single source pad.
+ *
+ * Return:
+ * * 0: Success
+ * * -EALREADY: One of the streams in streams_mask is not enabled
+ * * -EINVAL: The pad index is invalid, or doesn't correspond to a source pad
+ * * -EOPNOTSUPP: Falling back to the legacy .s_stream() operation is
+ *   impossible because the subdev has multiple source pads
+ */
+int v4l2_subdev_disable_streams(struct v4l2_subdev *sd, u32 pad,
+				u64 streams_mask);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
-- 
2.34.1

[PATCH v15 18/19] media: v4l2-subdev: Add v4l2_subdev_s_stream_helper() function

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

The v4l2_subdev_s_stream_helper() helper can be used by subdevs that
implement the stream-aware .enable_streams() and .disable_streams()
operations to implement .s_stream(). This is limited to subdevs that
have a single source pad.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Jacopo Mondi <jacopo@jmondi.org>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 40 +++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 17 ++++++++++++
 2 files changed, 57 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 101c1cfc0123..1da35bf9af01 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1941,6 +1941,46 @@ int v4l2_subdev_disable_streams(struct v4l2_subdev *sd, u32 pad,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_disable_streams);
 
+int v4l2_subdev_s_stream_helper(struct v4l2_subdev *sd, int enable)
+{
+	struct v4l2_subdev_state *state;
+	struct v4l2_subdev_route *route;
+	struct media_pad *pad;
+	u64 source_mask = 0;
+	int pad_index = -1;
+
+	/*
+	 * Find the source pad. This helper is meant for subdevs that have a
+	 * single source pad, so failures shouldn't happen, but catch them
+	 * loudly nonetheless as they indicate a driver bug.
+	 */
+	media_entity_for_each_pad(&sd->entity, pad) {
+		if (pad->flags & MEDIA_PAD_FL_SOURCE) {
+			pad_index = pad->index;
+			break;
+		}
+	}
+
+	if (WARN_ON(pad_index == -1))
+		return -EINVAL;
+
+	/*
+	 * As there's a single source pad, just collect all the source streams.
+	 */
+	state = v4l2_subdev_lock_and_get_active_state(sd);
+
+	for_each_active_route(&state->routing, route)
+		source_mask |= BIT_ULL(route->source_stream);
+
+	v4l2_subdev_unlock_state(state);
+
+	if (enable)
+		return v4l2_subdev_enable_streams(sd, pad_index, source_mask);
+	else
+		return v4l2_subdev_disable_streams(sd, pad_index, source_mask);
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_s_stream_helper);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index bff824367e0b..b21af00b1884 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1726,6 +1726,23 @@ int v4l2_subdev_enable_streams(struct v4l2_subdev *sd, u32 pad,
 int v4l2_subdev_disable_streams(struct v4l2_subdev *sd, u32 pad,
 				u64 streams_mask);
 
+/**
+ * v4l2_subdev_s_stream_helper() - Helper to implement the subdev s_stream
+ *	operation using enable_streams and disable_streams
+ * @sd: The subdevice
+ * @enable: Enable or disable streaming
+ *
+ * Subdevice drivers that implement the streams-aware
+ * &v4l2_subdev_pad_ops.enable_streams and &v4l2_subdev_pad_ops.disable_streams
+ * operations can use this helper to implement the legacy
+ * &v4l2_subdev_video_ops.s_stream operation.
+ *
+ * This helper can only be used by subdevs that have a single source pad.
+ *
+ * Return: 0 on success, or a negative error code otherwise.
+ */
+int v4l2_subdev_s_stream_helper(struct v4l2_subdev *sd, int enable);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
-- 
2.34.1

[PATCH v15 19/19] media: Add stream to frame descriptor

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

The stream field identifies the stream this frame descriptor applies to in
routing configuration across a multiplexed link.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 include/media/v4l2-subdev.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index b21af00b1884..6a77aa9bb1da 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -342,6 +342,7 @@ enum v4l2_mbus_frame_desc_flags {
  * struct v4l2_mbus_frame_desc_entry - media bus frame description structure
  *
  * @flags:	bitmask flags, as defined by &enum v4l2_mbus_frame_desc_flags.
+ * @stream:	stream in routing configuration
  * @pixelcode:	media bus pixel code, valid if @flags
  *		%FRAME_DESC_FL_BLOB is not set.
  * @length:	number of octets per frame, valid if @flags
@@ -351,6 +352,7 @@ enum v4l2_mbus_frame_desc_flags {
  */
 struct v4l2_mbus_frame_desc_entry {
 	enum v4l2_mbus_frame_desc_flags flags;
+	u32 stream;
 	u32 pixelcode;
 	u32 length;
 	union {
-- 
2.34.1

Re: [PATCH v15 05/19] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Hans Verkuil]

Hi Tomi,

On 10/3/22 14:18, Tomi Valkeinen wrote:
> From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> 
> Add support for subdev internal routing. A route is defined as a single
> stream from a sink pad to a source pad.
> 
> The userspace can configure the routing via two new ioctls,
> VIDIOC_SUBDEV_G_ROUTING and VIDIOC_SUBDEV_S_ROUTING, and subdevs can
> implement the functionality with v4l2_subdev_pad_ops.set_routing().
> 
> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Signed-off-by: Michal Simek <michal.simek@xilinx.com>
> 
> - Add sink and source streams for multiplexed links
> - Copy the argument back in case of an error. This is needed to let the
>   caller know the number of routes.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> 
> - Expand and refine documentation.
> - Make the 'routes' pointer a __u64 __user pointer so that a compat32
>   version of the ioctl is not required.
> - Add struct v4l2_subdev_krouting to be used for subdevice operations.
> 
> Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
> 
> - Fix typecasing warnings
> - Check sink & source pad types
> - Add 'which' field
> - Add V4L2_SUBDEV_ROUTE_FL_SOURCE
> - Routing to subdev state
> - Dropped get_routing subdev op
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/v4l2-core/v4l2-ioctl.c  | 25 +++++++-
>  drivers/media/v4l2-core/v4l2-subdev.c | 87 +++++++++++++++++++++++++++
>  include/media/v4l2-subdev.h           | 22 +++++++
>  include/uapi/linux/v4l2-subdev.h      | 52 ++++++++++++++++
>  4 files changed, 185 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index fddba75d9074..26b9a9626c96 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -16,6 +16,7 @@
>  #include <linux/kernel.h>
>  #include <linux/version.h>
>  
> +#include <linux/v4l2-subdev.h>
>  #include <linux/videodev2.h>
>  
>  #include <media/media-device.h> /* for media_set_bus_info() */
> @@ -3151,6 +3152,21 @@ static int check_array_args(unsigned int cmd, void *parg, size_t *array_size,
>  		ret = 1;
>  		break;
>  	}
> +
> +	case VIDIOC_SUBDEV_G_ROUTING:
> +	case VIDIOC_SUBDEV_S_ROUTING: {
> +		struct v4l2_subdev_routing *routing = parg;
> +
> +		if (routing->num_routes > 256)
> +			return -E2BIG;
> +
> +		*user_ptr = u64_to_user_ptr(routing->routes);
> +		*kernel_ptr = (void **)&routing->routes;
> +		*array_size = sizeof(struct v4l2_subdev_route)
> +			    * routing->num_routes;
> +		ret = 1;
> +		break;
> +	}
>  	}
>  
>  	return ret;
> @@ -3397,8 +3413,15 @@ video_usercopy(struct file *file, unsigned int orig_cmd, unsigned long arg,
>  	/*
>  	 * Some ioctls can return an error, but still have valid
>  	 * results that must be returned.
> +	 *
> +	 * FIXME: subdev IOCTLS are partially handled here and partially in
> +	 * v4l2-subdev.c and the 'always_copy' flag can only be set for IOCTLS
> +	 * defined here as part of the 'v4l2_ioctls' array. As
> +	 * VIDIOC_SUBDEV_G_ROUTING needs to return results to applications even
> +	 * in case of failure, but it is not defined here as part of the
> +	 * 'v4l2_ioctls' array, insert an ad-hoc check to address that.
>  	 */
> -	if (err < 0 && !always_copy)
> +	if (err < 0 && !always_copy && cmd != VIDIOC_SUBDEV_G_ROUTING)
>  		goto out;
>  
>  	if (has_array_args) {
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 8983d33fdb4b..1a451351554b 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -23,6 +23,16 @@
>  #include <media/v4l2-fh.h>
>  #include <media/v4l2-ioctl.h>
>  
> +/*
> + * Maximum stream ID is 63 for now, as we use u64 bitmask to represent a set
> + * of streams.
> + *
> + * Note that V4L2_FRAME_DESC_ENTRY_MAX is related: V4L2_FRAME_DESC_ENTRY_MAX
> + * restricts the total number of streams in a pad, although the stream ID is
> + * not restricted.
> + */
> +#define V4L2_SUBDEV_MAX_STREAM_ID 63
> +
>  #if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
>  static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
>  {
> @@ -417,6 +427,10 @@ subdev_ioctl_get_state(struct v4l2_subdev *sd, struct v4l2_subdev_fh *subdev_fh,
>  	case VIDIOC_SUBDEV_S_SELECTION:
>  		which = ((struct v4l2_subdev_selection *)arg)->which;
>  		break;
> +	case VIDIOC_SUBDEV_G_ROUTING:
> +	case VIDIOC_SUBDEV_S_ROUTING:
> +		which = ((struct v4l2_subdev_routing *)arg)->which;
> +		break;
>  	}
>  
>  	return which == V4L2_SUBDEV_FORMAT_TRY ?
> @@ -733,6 +747,78 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>  	case VIDIOC_SUBDEV_QUERYSTD:
>  		return v4l2_subdev_call(sd, video, querystd, arg);
>  
> +	case VIDIOC_SUBDEV_G_ROUTING: {
> +		struct v4l2_subdev_routing *routing = arg;
> +		struct v4l2_subdev_krouting *krouting;
> +
> +		if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS))
> +			return -ENOIOCTLCMD;
> +
> +		memset(routing->reserved, 0, sizeof(routing->reserved));
> +
> +		krouting = &state->routing;
> +
> +		if (routing->num_routes < krouting->num_routes) {
> +			routing->num_routes = krouting->num_routes;
> +			return -ENOSPC;
> +		}
> +
> +		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
> +		       krouting->routes,
> +		       krouting->num_routes * sizeof(*krouting->routes));
> +		routing->num_routes = krouting->num_routes;
> +
> +		return 0;
> +	}
> +
> +	case VIDIOC_SUBDEV_S_ROUTING: {
> +		struct v4l2_subdev_routing *routing = arg;
> +		struct v4l2_subdev_route *routes =
> +			(struct v4l2_subdev_route *)(uintptr_t)routing->routes;
> +		struct v4l2_subdev_krouting krouting = {};
> +		unsigned int i;
> +
> +		if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS))
> +			return -ENOIOCTLCMD;
> +
> +		if (routing->which != V4L2_SUBDEV_FORMAT_TRY && ro_subdev)
> +			return -EPERM;
> +
> +		memset(routing->reserved, 0, sizeof(routing->reserved));
> +
> +		for (i = 0; i < routing->num_routes; ++i) {
> +			const struct v4l2_subdev_route *route = &routes[i];
> +			const struct media_pad *pads = sd->entity.pads;
> +
> +			if (route->sink_stream > V4L2_SUBDEV_MAX_STREAM_ID ||
> +			    route->source_stream > V4L2_SUBDEV_MAX_STREAM_ID)
> +				return -EINVAL;
> +
> +			/* Do not check sink pad for source routes */
> +			if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE)) {
> +				if (route->sink_pad >= sd->entity.num_pads)
> +					return -EINVAL;
> +
> +				if (!(pads[route->sink_pad].flags &
> +				      MEDIA_PAD_FL_SINK))
> +					return -EINVAL;
> +			}
> +
> +			if (route->source_pad >= sd->entity.num_pads)
> +				return -EINVAL;
> +
> +			if (!(pads[route->source_pad].flags &
> +			      MEDIA_PAD_FL_SOURCE))
> +				return -EINVAL;
> +		}
> +
> +		krouting.num_routes = routing->num_routes;
> +		krouting.routes = routes;
> +
> +		return v4l2_subdev_call(sd, pad, set_routing, state,
> +					routing->which, &krouting);
> +	}
> +
>  	default:
>  		return v4l2_subdev_call(sd, core, ioctl, cmd, arg);
>  	}
> @@ -1016,6 +1102,7 @@ void __v4l2_subdev_state_free(struct v4l2_subdev_state *state)
>  
>  	mutex_destroy(&state->_lock);
>  
> +	kfree(state->routing.routes);
>  	kvfree(state->pads);
>  	kfree(state);
>  }
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 4be0a590c7c7..4934dc9468a8 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -695,12 +695,26 @@ struct v4l2_subdev_pad_config {
>  	struct v4l2_rect try_compose;
>  };
>  
> +/**
> + * struct v4l2_subdev_krouting - subdev routing table
> + *
> + * @num_routes: number of routes
> + * @routes: &struct v4l2_subdev_route
> + *
> + * This structure contains the routing table for a subdev.
> + */
> +struct v4l2_subdev_krouting {
> +	unsigned int num_routes;
> +	struct v4l2_subdev_route *routes;
> +};
> +
>  /**
>   * struct v4l2_subdev_state - Used for storing subdev state information.
>   *
>   * @_lock: default for 'lock'
>   * @lock: mutex for the state. May be replaced by the user.
>   * @pads: &struct v4l2_subdev_pad_config array
> + * @routing: routing table for the subdev
>   *
>   * This structure only needs to be passed to the pad op if the 'which' field
>   * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
> @@ -711,6 +725,7 @@ struct v4l2_subdev_state {
>  	struct mutex _lock;
>  	struct mutex *lock;
>  	struct v4l2_subdev_pad_config *pads;
> +	struct v4l2_subdev_krouting routing;
>  };
>  
>  /**
> @@ -763,6 +778,9 @@ struct v4l2_subdev_state {
>   *		     this operation as close as possible to stream on time. The
>   *		     operation shall fail if the pad index it has been called on
>   *		     is not valid or in case of unrecoverable failures.
> + *
> + * @set_routing: enable or disable data connection routes described in the
> + *		 subdevice routing table.
>   */
>  struct v4l2_subdev_pad_ops {
>  	int (*init_cfg)(struct v4l2_subdev *sd,
> @@ -805,6 +823,10 @@ struct v4l2_subdev_pad_ops {
>  			      struct v4l2_mbus_frame_desc *fd);
>  	int (*get_mbus_config)(struct v4l2_subdev *sd, unsigned int pad,
>  			       struct v4l2_mbus_config *config);
> +	int (*set_routing)(struct v4l2_subdev *sd,
> +			   struct v4l2_subdev_state *state,
> +			   enum v4l2_subdev_format_whence which,
> +			   struct v4l2_subdev_krouting *route);
>  };
>  
>  /**
> diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
> index 89af27f50a41..db4af236d7ed 100644
> --- a/include/uapi/linux/v4l2-subdev.h
> +++ b/include/uapi/linux/v4l2-subdev.h
> @@ -24,6 +24,7 @@
>  #ifndef __LINUX_V4L2_SUBDEV_H
>  #define __LINUX_V4L2_SUBDEV_H
>  
> +#include <linux/const.h>
>  #include <linux/ioctl.h>
>  #include <linux/types.h>
>  #include <linux/v4l2-common.h>
> @@ -191,6 +192,55 @@ struct v4l2_subdev_capability {
>  /* The v4l2 sub-device supports routing and multiplexed streams. */
>  #define V4L2_SUBDEV_CAP_STREAMS			0x00000002
>  
> +/*
> + * Is the route active? An active route will start when streaming is enabled
> + * on a video node.
> + */
> +#define V4L2_SUBDEV_ROUTE_FL_ACTIVE		(1U << 0)
> +
> +/*
> + * Is the route a source endpoint? A source endpoint route refers to a stream
> + * generated by the subdevice (usually a sensor), and thus there is no
> + * sink-side endpoint for the route. The sink_pad and sink_stream fields are
> + * unused.
> + * Set by the driver.
> + */
> +#define V4L2_SUBDEV_ROUTE_FL_SOURCE		(1U << 1)

Can we rename this to _FL_INTERNAL_SOURCE? Just 'SOURCE' is very confusing
IMHO. The name 'INTERNAL_SOURCE' makes it clear that it is generated internally,
and so does not come from an external sink-side endpoint.

I also think that the documentation for this flag in patch 04/19 is very vague,
I'll comment on that patch as well.

Regards,

	Hans

> +
> +/**
> + * struct v4l2_subdev_route - A route inside a subdev
> + *
> + * @sink_pad: the sink pad index
> + * @sink_stream: the sink stream identifier
> + * @source_pad: the source pad index
> + * @source_stream: the source stream identifier
> + * @flags: route flags V4L2_SUBDEV_ROUTE_FL_*
> + * @reserved: drivers and applications must zero this array
> + */
> +struct v4l2_subdev_route {
> +	__u32 sink_pad;
> +	__u32 sink_stream;
> +	__u32 source_pad;
> +	__u32 source_stream;
> +	__u32 flags;
> +	__u32 reserved[5];
> +};
> +
> +/**
> + * struct v4l2_subdev_routing - Subdev routing information
> + *
> + * @which: configuration type (from enum v4l2_subdev_format_whence)
> + * @num_routes: the total number of routes in the routes array
> + * @routes: pointer to the routes array
> + * @reserved: drivers and applications must zero this array
> + */
> +struct v4l2_subdev_routing {
> +	__u32 which;
> +	__u32 num_routes;
> +	__u64 routes;
> +	__u32 reserved[6];
> +};
> +
>  /* Backwards compatibility define --- to be removed */
>  #define v4l2_subdev_edid v4l2_edid
>  
> @@ -206,6 +256,8 @@ struct v4l2_subdev_capability {
>  #define VIDIOC_SUBDEV_S_CROP			_IOWR('V', 60, struct v4l2_subdev_crop)
>  #define VIDIOC_SUBDEV_G_SELECTION		_IOWR('V', 61, struct v4l2_subdev_selection)
>  #define VIDIOC_SUBDEV_S_SELECTION		_IOWR('V', 62, struct v4l2_subdev_selection)
> +#define VIDIOC_SUBDEV_G_ROUTING			_IOWR('V', 38, struct v4l2_subdev_routing)
> +#define VIDIOC_SUBDEV_S_ROUTING			_IOWR('V', 39, struct v4l2_subdev_routing)
>  /* The following ioctls are identical to the ioctls in videodev2.h */
>  #define VIDIOC_SUBDEV_G_STD			_IOR('V', 23, v4l2_std_id)
>  #define VIDIOC_SUBDEV_S_STD			_IOW('V', 24, v4l2_std_id)

Re: [PATCH v15 04/19] media: Documentation: Add GS_ROUTING documentation

[Hans Verkuil]

Hi Tomi,

On 10/3/22 14:18, Tomi Valkeinen wrote:
> From: Jacopo Mondi <jacopo+renesas@jmondi.org>
> 
> Add documentation for VIDIOC_SUBDEV_G/S_ROUTING ioctl and add
> description of multiplexed media pads and internal routing to the
> V4L2-subdev documentation section.
> 
> Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
> ---
>  .../userspace-api/media/v4l/dev-subdev.rst    |   2 +
>  .../userspace-api/media/v4l/user-func.rst     |   1 +
>  .../media/v4l/vidioc-subdev-g-routing.rst     | 155 ++++++++++++++++++
>  3 files changed, 158 insertions(+)
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> 
> diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> index fd1de0a73a9f..a67c2749089a 100644
> --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> @@ -29,6 +29,8 @@ will feature a character device node on which ioctls can be called to
>  
>  -  negotiate image formats on individual pads
>  
> +-  inspect and modify internal data routing between pads of the same entity
> +
>  Sub-device character device nodes, conventionally named
>  ``/dev/v4l-subdev*``, use major number 81.
>  
> diff --git a/Documentation/userspace-api/media/v4l/user-func.rst b/Documentation/userspace-api/media/v4l/user-func.rst
> index 53e604bd7d60..228c1521f190 100644
> --- a/Documentation/userspace-api/media/v4l/user-func.rst
> +++ b/Documentation/userspace-api/media/v4l/user-func.rst
> @@ -70,6 +70,7 @@ Function Reference
>      vidioc-subdev-g-crop
>      vidioc-subdev-g-fmt
>      vidioc-subdev-g-frame-interval
> +    vidioc-subdev-g-routing
>      vidioc-subdev-g-selection
>      vidioc-subdev-querycap
>      vidioc-subscribe-event
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> new file mode 100644
> index 000000000000..6d8fc3b11352
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> @@ -0,0 +1,155 @@
> +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
> +.. c:namespace:: V4L
> +
> +.. _VIDIOC_SUBDEV_G_ROUTING:
> +
> +******************************************************
> +ioctl VIDIOC_SUBDEV_G_ROUTING, VIDIOC_SUBDEV_S_ROUTING
> +******************************************************
> +
> +Name
> +====
> +
> +VIDIOC_SUBDEV_G_ROUTING - VIDIOC_SUBDEV_S_ROUTING - Get or set routing between streams of media pads in a media entity.
> +
> +
> +Synopsis
> +========
> +
> +.. c:function:: int ioctl( int fd, VIDIOC_SUBDEV_G_ROUTING, struct v4l2_subdev_routing *argp )
> +    :name: VIDIOC_SUBDEV_G_ROUTING
> +
> +.. c:function:: int ioctl( int fd, VIDIOC_SUBDEV_S_ROUTING, struct v4l2_subdev_routing *argp )
> +    :name: VIDIOC_SUBDEV_S_ROUTING
> +
> +
> +Arguments
> +=========
> +
> +``fd``
> +    File descriptor returned by :ref:`open() <func-open>`.
> +
> +``argp``
> +    Pointer to struct :c:type:`v4l2_subdev_routing`.
> +
> +
> +Description
> +===========
> +
> +These ioctls are used to get and set the routing in a media entity.
> +The routing configuration determines the flows of data inside an entity.
> +
> +Drivers report their current routing tables using the
> +``VIDIOC_SUBDEV_G_ROUTING`` ioctl and application may enable or disable routes
> +with the ``VIDIOC_SUBDEV_S_ROUTING`` ioctl, by adding or removing routes and
> +setting or clearing flags of the  ``flags`` field of a
> +struct :c:type:`v4l2_subdev_route`.
> +
> +All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called. This
> +means that the userspace mut reconfigure all streams after calling the ioctl

typo: mut -> must

> +with e.g. ``VIDIOC_SUBDEV_S_FMT``.
> +
> +A special case for routing are routes marked with
> +``V4L2_SUBDEV_ROUTE_FL_SOURCE`` flag. These routes are used to describe
> +source endpoints on sensors and the sink fields are unused.

This is very vague. As mentioned in my review for 05/19 I think this flag should
be renamed to INTERNAL_SOURCE. Then the last sentence can change to:

"These routes are used to describe source endpoints where the stream is internally
created (such as a sensor) and so the sink fields are unused."

> +
> +When inspecting routes through ``VIDIOC_SUBDEV_G_ROUTING`` and the application
> +provided ``num_routes`` is not big enough to contain all the available routes
> +the subdevice exposes, drivers return the ENOSPC error code and adjust the
> +value of the ``num_routes`` field. Application should then reserve enough memory
> +for all the route entries and call ``VIDIOC_SUBDEV_G_ROUTING`` again.
> +
> +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
> +
> +.. c:type:: v4l2_subdev_routing
> +
> +.. flat-table:: struct v4l2_subdev_routing
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths:       1 1 2
> +
> +    * - __u32
> +      - ``which``
> +      - Format to modified, from enum
> +        :ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
> +    * - struct :c:type:`v4l2_subdev_route`
> +      - ``routes[]``
> +      - Array of struct :c:type:`v4l2_subdev_route` entries
> +    * - __u32
> +      - ``num_routes``
> +      - Number of entries of the routes array
> +    * - __u32
> +      - ``reserved``\ [5]
> +      - Reserved for future extensions. Applications and drivers must set
> +	the array to zero.
> +
> +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
> +
> +.. c:type:: v4l2_subdev_route
> +
> +.. flat-table:: struct v4l2_subdev_route
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths:       1 1 2
> +
> +    * - __u32
> +      - ``sink_pad``
> +      - Sink pad number.
> +    * - __u32
> +      - ``sink_stream``
> +      - Sink pad stream number.
> +    * - __u32
> +      - ``source_pad``
> +      - Source pad number.
> +    * - __u32
> +      - ``source_stream``
> +      - Source pad stream number.
> +    * - __u32
> +      - ``flags``
> +      - Route enable/disable flags
> +	:ref:`v4l2_subdev_routing_flags <v4l2-subdev-routing-flags>`.
> +    * - __u32
> +      - ``reserved``\ [5]
> +      - Reserved for future extensions. Applications and drivers must set
> +	the array to zero.
> +
> +.. tabularcolumns:: |p{6.6cm}|p{2.2cm}|p{8.7cm}|
> +
> +.. _v4l2-subdev-routing-flags:
> +
> +.. flat-table:: enum v4l2_subdev_routing_flags
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths:       3 1 4
> +
> +    * - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +      - 0
> +      - The route is enabled. Set by applications.
> +    * - V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
> +      - 1
> +      - The route is immutable. Set by the driver.
> +    * - V4L2_SUBDEV_ROUTE_FL_SOURCE
> +      - 2
> +      - The route is a source route, and the ``sink_pad`` and ``sink_stream``
> +        fields are unused. Set by the driver.

Same issue as above, it's very vague.

"Used to describe a route source endpoint where the stream is internally
created (such as a sensor) and so the sink fields are unused."

Regards,

	Hans

> +
> +Return Value
> +============
> +
> +On success 0 is returned, on error -1 and the ``errno`` variable is set
> +appropriately. The generic error codes are described at the
> +:ref:`Generic Error Codes <gen-errors>` chapter.
> +
> +ENOSPC
> +   The application provided ``num_routes`` is not big enough to contain
> +   all the available routes the subdevice exposes.
> +
> +EINVAL
> +   The sink or source pad identifiers reference a non-existing pad, or reference
> +   pads of different types (ie. the sink_pad identifiers refers to a source pad)
> +   or the sink or source stream identifiers reference a non-existing stream on
> +   the sink or source pad.
> +
> +E2BIG
> +   The application provided ``num_routes`` for ``VIDIOC_SUBDEV_S_ROUTING`` is
> +   larger than the number of routes the driver can handle.

Re: [PATCH v15 01/19] media: v4l2-subdev: Sort includes

[Laurent Pinchart]

Hi Tomi,

Thank you for the patch.

On Mon, Oct 03, 2022 at 03:18:34PM +0300, Tomi Valkeinen wrote:
> Sort the includes alphabetically.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>

Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 5c27bac772ea..ca5b764d796d 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -8,20 +8,20 @@
>   *	    Sakari Ailus <sakari.ailus@iki.fi>
>   */
>  
> +#include <linux/export.h>
>  #include <linux/ioctl.h>
>  #include <linux/mm.h>
>  #include <linux/module.h>
>  #include <linux/slab.h>
>  #include <linux/types.h>
> -#include <linux/videodev2.h>
> -#include <linux/export.h>
>  #include <linux/version.h>
> +#include <linux/videodev2.h>
>  
>  #include <media/v4l2-ctrls.h>
>  #include <media/v4l2-device.h>
> -#include <media/v4l2-ioctl.h>
> -#include <media/v4l2-fh.h>
>  #include <media/v4l2-event.h>
> +#include <media/v4l2-fh.h>
> +#include <media/v4l2-ioctl.h>
>  
>  #if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
>  static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)

-- 
Regards,

Laurent Pinchart

Re: [PATCH v15 05/19] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Sakari Ailus]

Hi Hans,

On Mon, Oct 03, 2022 at 04:26:32PM +0200, Hans Verkuil wrote:
> > +#define V4L2_SUBDEV_ROUTE_FL_SOURCE		(1U << 1)
> 
> Can we rename this to _FL_INTERNAL_SOURCE? Just 'SOURCE' is very confusing
> IMHO. The name 'INTERNAL_SOURCE' makes it clear that it is generated internally,
> and so does not come from an external sink-side endpoint.
> 
> I also think that the documentation for this flag in patch 04/19 is very vague,
> I'll comment on that patch as well.

Having descriptive names is important but I think "SOURCE" as such is fine
for the purpose. Adding "INTERNAL_" adds 9 characters to what is already a
very long name, making the flag very clumsy to use in code. That's why I
would prefer to keep it as-is.

-- 
Kind regards,

Sakari Ailus

Re: [PATCH v15 05/19] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Hans Verkuil]

Hi Sakari,

On 10/4/22 00:01, Sakari Ailus wrote:
> Hi Hans,
> 
> On Mon, Oct 03, 2022 at 04:26:32PM +0200, Hans Verkuil wrote:
>>> +#define V4L2_SUBDEV_ROUTE_FL_SOURCE		(1U << 1)
>>
>> Can we rename this to _FL_INTERNAL_SOURCE? Just 'SOURCE' is very confusing
>> IMHO. The name 'INTERNAL_SOURCE' makes it clear that it is generated internally,
>> and so does not come from an external sink-side endpoint.
>>
>> I also think that the documentation for this flag in patch 04/19 is very vague,
>> I'll comment on that patch as well.
> 
> Having descriptive names is important but I think "SOURCE" as such is fine
> for the purpose. Adding "INTERNAL_" adds 9 characters to what is already a
> very long name, making the flag very clumsy to use in code. That's why I
> would prefer to keep it as-is.
> 

_FL_SOURCE is meaningless (at least to me): there are so many 'source' and 'sink'
references, that just plain 'SOURCE' doesn't help me understand what the flag
means. I did consider INT_SOURCE, but I thought 'INT' is too close to 'interrupt'.
I'm OK with that, though.

Another alternative would be _FL_NO_SINK: that clearly conveys that 1) there is
no sink, and implies that 2) the source is internally generated.

Or perhaps: _FL_SOURCE_ONLY?

But let's avoid V4L2_SUBDEV_ROUTE_FL_SOURCE: to me that just says that the route
has a source, but that's true for all routes. There is nothing in the flag name
to indicate that the route has *only* a source and no sink.

Regards,

	Hans

Re: [PATCH v15 05/19] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Sakari Ailus]

Hi Hans,

On Tue, Oct 04, 2022 at 10:43:55AM +0200, Hans Verkuil wrote:
> Hi Sakari,
> 
> On 10/4/22 00:01, Sakari Ailus wrote:
> > Hi Hans,
> > 
> > On Mon, Oct 03, 2022 at 04:26:32PM +0200, Hans Verkuil wrote:
> >>> +#define V4L2_SUBDEV_ROUTE_FL_SOURCE		(1U << 1)
> >>
> >> Can we rename this to _FL_INTERNAL_SOURCE? Just 'SOURCE' is very confusing
> >> IMHO. The name 'INTERNAL_SOURCE' makes it clear that it is generated internally,
> >> and so does not come from an external sink-side endpoint.
> >>
> >> I also think that the documentation for this flag in patch 04/19 is very vague,
> >> I'll comment on that patch as well.
> > 
> > Having descriptive names is important but I think "SOURCE" as such is fine
> > for the purpose. Adding "INTERNAL_" adds 9 characters to what is already a
> > very long name, making the flag very clumsy to use in code. That's why I
> > would prefer to keep it as-is.
> > 
> 
> _FL_SOURCE is meaningless (at least to me): there are so many 'source' and 'sink'
> references, that just plain 'SOURCE' doesn't help me understand what the flag
> means. I did consider INT_SOURCE, but I thought 'INT' is too close to 'interrupt'.
> I'm OK with that, though.
> 
> Another alternative would be _FL_NO_SINK: that clearly conveys that 1) there is
> no sink, and implies that 2) the source is internally generated.
> 
> Or perhaps: _FL_SOURCE_ONLY?

This appears as the best compromise IMO. NO_SINK is shorter but conveying
the meaning through negation is what I don't like too much.

-- 
Kind regards,

Sakari Ailus

Re: [PATCH v15 08/19] media: subdev: Add for_each_active_route() macro

[Dafna Hirschfeld]

On 03.10.2022 15:18, Tomi Valkeinen wrote:
>From: Jacopo Mondi <jacopo+renesas@jmondi.org>
>
>Add a for_each_active_route() macro to replace the repeated pattern
>of iterating on the active routes of a routing table.
>
>Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
>Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>---
> .clang-format                         |  1 +
> drivers/media/v4l2-core/v4l2-subdev.c | 20 ++++++++++++++++++++
> include/media/v4l2-subdev.h           | 13 +++++++++++++
> 3 files changed, 34 insertions(+)
>
>diff --git a/.clang-format b/.clang-format
>index 1247d54f9e49..31f39ae78f7b 100644
>--- a/.clang-format
>+++ b/.clang-format
>@@ -190,6 +190,7 @@ ForEachMacros:
>   - 'for_each_active_dev_scope'
>   - 'for_each_active_drhd_unit'
>   - 'for_each_active_iommu'
>+  - 'for_each_active_route'
>   - 'for_each_aggr_pgid'
>   - 'for_each_available_child_of_node'
>   - 'for_each_bench'
>diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>index 3ae4f39a50e4..1049c07d2e49 100644
>--- a/drivers/media/v4l2-core/v4l2-subdev.c
>+++ b/drivers/media/v4l2-core/v4l2-subdev.c
>@@ -1212,6 +1212,26 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
> }
> EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing);
>
>+struct v4l2_subdev_route *
>+__v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting *routing,
>+				struct v4l2_subdev_route *route)
>+{
>+	if (route)
>+		++route;
>+	else
>+		route = &routing->routes[0];
>+
>+	for (; route < routing->routes + routing->num_routes; ++route) {
>+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
>+			continue;
>+
>+		return route;
>+	}
>+
>+	return NULL;
>+}
>+EXPORT_SYMBOL_GPL(__v4l2_subdev_next_active_route);
>+
> #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
>
> #endif /* CONFIG_MEDIA_CONTROLLER */
>diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
>index 7962e6572bda..89e58208e330 100644
>--- a/include/media/v4l2-subdev.h
>+++ b/include/media/v4l2-subdev.h
>@@ -1435,6 +1435,19 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
> 			    struct v4l2_subdev_state *state,
> 			    const struct v4l2_subdev_krouting *routing);
>
>+struct v4l2_subdev_route *
>+__v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting *routing,
>+				struct v4l2_subdev_route *route);
>+
>+/**
>+ * for_each_active_route - iterate on all active routes of a routing table
>+ * @routing: The routing table
>+ * @route: The route iterator
>+ */
>+#define for_each_active_route(routing, route) \
>+	for ((route) = NULL;                  \
>+	     ((route) = __v4l2_subdev_next_active_route((routing), (route)));)
Hi, shouldn't it be something like:
	for ((route) = NULL; (route) ; (route) = __v4l2_subdev_next_active_route((routing), (route)))

Thanks,
Dafna

>+
> #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
>
> #endif /* CONFIG_MEDIA_CONTROLLER */
>-- 
>2.34.1
>

Re: [PATCH v15 10/19] media: subdev: add stream based configuration

[Dafna Hirschfeld]

On 03.10.2022 15:18, Tomi Valkeinen wrote:
>Add support to manage configurations (format, crop, compose) per stream,
>instead of per pad. This is accomplished with data structures that hold
>an array of all subdev's stream configurations.
>
>The number of streams can vary at runtime based on routing. Every time
>the routing is changed, the stream configurations need to be
>re-initialized.
>
>Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>---
> .../v4l/vidioc-subdev-enum-frame-interval.rst |   5 +-
> .../v4l/vidioc-subdev-enum-frame-size.rst     |   5 +-
> .../v4l/vidioc-subdev-enum-mbus-code.rst      |   5 +-
> .../media/v4l/vidioc-subdev-g-crop.rst        |   5 +-
> .../media/v4l/vidioc-subdev-g-fmt.rst         |   5 +-
> .../v4l/vidioc-subdev-g-frame-interval.rst    |   5 +-
> .../media/v4l/vidioc-subdev-g-selection.rst   |   5 +-
> drivers/media/v4l2-core/v4l2-subdev.c         | 154 +++++++++++++++++-
> include/media/v4l2-subdev.h                   |  79 +++++++++
> include/uapi/linux/v4l2-subdev.h              |  28 +++-
> 10 files changed, 275 insertions(+), 21 deletions(-)
>
>diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
>index 3703943b412f..8def4c05d3da 100644
>--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
>+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
>@@ -92,7 +92,10 @@ multiple pads of the same sub-device is not defined.
>       - Frame intervals to be enumerated, from enum
> 	:ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
>     * - __u32
>-      - ``reserved``\ [8]
>+      - ``stream``
>+      - Stream identifier.
>+    * - __u32
>+      - ``reserved``\ [7]
>       - Reserved for future extensions. Applications and drivers must set
> 	the array to zero.
>
>diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
>index c25a9896df0e..3ef361c0dca7 100644
>--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
>+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
>@@ -97,7 +97,10 @@ information about try formats.
>       - Frame sizes to be enumerated, from enum
> 	:ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
>     * - __u32
>-      - ``reserved``\ [8]
>+      - ``stream``
>+      - Stream identifier.
>+    * - __u32
>+      - ``reserved``\ [7]
>       - Reserved for future extensions. Applications and drivers must set
> 	the array to zero.
>
>diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
>index 417f1a19bcc4..248f6f9ee7c5 100644
>--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
>+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
>@@ -73,7 +73,10 @@ information about the try formats.
>       - ``flags``
>       - See :ref:`v4l2-subdev-mbus-code-flags`
>     * - __u32
>-      - ``reserved``\ [7]
>+      - ``stream``
>+      - Stream identifier.
>+    * - __u32
>+      - ``reserved``\ [6]
>       - Reserved for future extensions. Applications and drivers must set
> 	the array to zero.
>
>diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
>index bd15c0a5a66b..1d267f7e7991 100644
>--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
>+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
>@@ -96,7 +96,10 @@ modified format should be as close as possible to the original request.
>       - ``rect``
>       - Crop rectangle boundaries, in pixels.
>     * - __u32
>-      - ``reserved``\ [8]
>+      - ``stream``
>+      - Stream identifier.
>+    * - __u32
>+      - ``reserved``\ [7]
>       - Reserved for future extensions. Applications and drivers must set
> 	the array to zero.
>
>diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
>index 7acdbb939d89..ed253a1e44b7 100644
>--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
>+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
>@@ -102,7 +102,10 @@ should be as close as possible to the original request.
>       - Definition of an image format, see :c:type:`v4l2_mbus_framefmt` for
> 	details.
>     * - __u32
>-      - ``reserved``\ [8]
>+      - ``stream``
>+      - Stream identifier.
>+    * - __u32
>+      - ``reserved``\ [7]
>       - Reserved for future extensions. Applications and drivers must set
> 	the array to zero.
>
>diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
>index d7fe7543c506..842f962d2aea 100644
>--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
>+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
>@@ -90,7 +90,10 @@ the same sub-device is not defined.
>       - ``interval``
>       - Period, in seconds, between consecutive video frames.
>     * - __u32
>-      - ``reserved``\ [9]
>+      - ``stream``
>+      - Stream identifier.
>+    * - __u32
>+      - ``reserved``\ [8]
>       - Reserved for future extensions. Applications and drivers must set
> 	the array to zero.
>
>diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
>index f9172a42f036..6b629c19168c 100644
>--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
>+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
>@@ -94,7 +94,10 @@ Selection targets and flags are documented in
>       - ``r``
>       - Selection rectangle, in pixels.
>     * - __u32
>-      - ``reserved``\ [8]
>+      - ``stream``
>+      - Stream identifier.
>+    * - __u32
>+      - ``reserved``\ [7]
>       - Reserved for future extensions. Applications and drivers must set
> 	the array to zero.
>
>diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>index 1049c07d2e49..be778e619694 100644
>--- a/drivers/media/v4l2-core/v4l2-subdev.c
>+++ b/drivers/media/v4l2-core/v4l2-subdev.c
>@@ -159,8 +159,22 @@ static inline int check_pad(struct v4l2_subdev *sd, u32 pad)
> 	return 0;
> }
>
>-static int check_state_pads(u32 which, struct v4l2_subdev_state *state)
>+static int check_state(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
>+		       u32 which, u32 pad, u32 stream)
> {
>+	if (sd->flags & V4L2_SUBDEV_FL_STREAMS) {
>+#if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
>+		if (!v4l2_subdev_state_get_stream_format(state, pad, stream))
>+			return -EINVAL;
>+		return 0;
>+#else
>+		return -EINVAL;
>+#endif
>+	}
>+
>+	if (stream != 0)
>+		return -EINVAL;
>+
> 	if (which == V4L2_SUBDEV_FORMAT_TRY && (!state || !state->pads))
> 		return -EINVAL;
>
>@@ -175,7 +189,7 @@ static inline int check_format(struct v4l2_subdev *sd,
> 		return -EINVAL;
>
> 	return check_which(format->which) ? : check_pad(sd, format->pad) ? :
>-	       check_state_pads(format->which, state);
>+	       check_state(sd, state, format->which, format->pad, format->stream);
> }
>
> static int call_get_fmt(struct v4l2_subdev *sd,
>@@ -202,7 +216,7 @@ static int call_enum_mbus_code(struct v4l2_subdev *sd,
> 		return -EINVAL;
>
> 	return check_which(code->which) ? : check_pad(sd, code->pad) ? :
>-	       check_state_pads(code->which, state) ? :
>+	       check_state(sd, state, code->which, code->pad, code->stream) ? :
> 	       sd->ops->pad->enum_mbus_code(sd, state, code);
> }
>
>@@ -214,7 +228,7 @@ static int call_enum_frame_size(struct v4l2_subdev *sd,
> 		return -EINVAL;
>
> 	return check_which(fse->which) ? : check_pad(sd, fse->pad) ? :
>-	       check_state_pads(fse->which, state) ? :
>+	       check_state(sd, state, fse->which, fse->pad, fse->stream) ? :
> 	       sd->ops->pad->enum_frame_size(sd, state, fse);
> }
>
>@@ -249,7 +263,7 @@ static int call_enum_frame_interval(struct v4l2_subdev *sd,
> 		return -EINVAL;
>
> 	return check_which(fie->which) ? : check_pad(sd, fie->pad) ? :
>-	       check_state_pads(fie->which, state) ? :
>+	       check_state(sd, state, fie->which, fie->pad, fie->stream) ? :
> 	       sd->ops->pad->enum_frame_interval(sd, state, fie);
> }
>
>@@ -261,7 +275,7 @@ static inline int check_selection(struct v4l2_subdev *sd,
> 		return -EINVAL;
>
> 	return check_which(sel->which) ? : check_pad(sd, sel->pad) ? :
>-	       check_state_pads(sel->which, state);
>+	       check_state(sd, state, sel->which, sel->pad, sel->stream);
> }
>
> static int call_get_selection(struct v4l2_subdev *sd,
>@@ -1095,7 +1109,8 @@ __v4l2_subdev_state_alloc(struct v4l2_subdev *sd, const char *lock_name,
> 	else
> 		state->lock = &state->_lock;
>
>-	if (sd->entity.num_pads) {
>+	/* Drivers that support streams do not need the legacy pad config */
>+	if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS) && sd->entity.num_pads) {
> 		state->pads = kvcalloc(sd->entity.num_pads,
> 				       sizeof(*state->pads), GFP_KERNEL);
> 		if (!state->pads) {
>@@ -1135,6 +1150,7 @@ void __v4l2_subdev_state_free(struct v4l2_subdev_state *state)
> 	mutex_destroy(&state->_lock);
>
> 	kfree(state->routing.routes);
>+	kvfree(state->stream_configs.configs);
> 	kvfree(state->pads);
> 	kfree(state);
> }
>@@ -1164,6 +1180,59 @@ EXPORT_SYMBOL_GPL(v4l2_subdev_cleanup);
>
> #if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
>
>+static int
>+v4l2_subdev_init_stream_configs(struct v4l2_subdev_stream_configs *stream_configs,
>+				const struct v4l2_subdev_krouting *routing)
>+{
>+	struct v4l2_subdev_stream_configs new_configs = { 0 };
>+	struct v4l2_subdev_route *route;
>+	u32 format_idx = 0;
I think you don't need both format_idx and idx,
also, it can be 'int'

Thanks,
Dafna

>+
>+	/* Count number of formats needed */
>+	for_each_active_route(routing, route) {
>+		/*
>+		 * Each route needs a format on both ends of the route, except
>+		 * for source streams which only need one format.
>+		 */
>+		new_configs.num_configs +=
>+			(route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE) ? 1 : 2;
>+	}
>+
>+	if (new_configs.num_configs) {
>+		new_configs.configs = kvcalloc(new_configs.num_configs,
>+					       sizeof(*new_configs.configs),
>+					       GFP_KERNEL);
>+
>+		if (!new_configs.configs)
>+			return -ENOMEM;
>+	}
>+
>+	/*
>+	 * Fill in the 'pad' and stream' value for each item in the array from
>+	 * the routing table
>+	 */
>+	for_each_active_route(routing, route) {
>+		u32 idx;
>+
>+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE)) {
>+			idx = format_idx++;
>+
>+			new_configs.configs[idx].pad = route->sink_pad;
>+			new_configs.configs[idx].stream = route->sink_stream;
>+		}
>+
>+		idx = format_idx++;
>+
>+		new_configs.configs[idx].pad = route->source_pad;
>+		new_configs.configs[idx].stream = route->source_stream;
>+	}
>+
>+	kvfree(stream_configs->configs);
>+	*stream_configs = new_configs;
>+
>+	return 0;
>+}
>+
> int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
> 			struct v4l2_subdev_format *format)
> {
>@@ -1190,6 +1259,7 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
> 	const struct v4l2_subdev_krouting *src = routing;
> 	struct v4l2_subdev_krouting new_routing = { 0 };
> 	size_t bytes;
>+	int r;
>
> 	if (unlikely(check_mul_overflow(src->num_routes, sizeof(*src->routes),
> 					&bytes)))
>@@ -1205,6 +1275,13 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
>
> 	new_routing.num_routes = src->num_routes;
>
>+	r = v4l2_subdev_init_stream_configs(&state->stream_configs,
>+					    &new_routing);
>+	if (r) {
>+		kfree(new_routing.routes);
>+		return r;
>+	}
>+
> 	kfree(dst->routes);
> 	*dst = new_routing;
>
>@@ -1232,6 +1309,69 @@ __v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting *routing,
> }
> EXPORT_SYMBOL_GPL(__v4l2_subdev_next_active_route);
>
>+struct v4l2_mbus_framefmt *
>+v4l2_subdev_state_get_stream_format(struct v4l2_subdev_state *state,
>+				    unsigned int pad, u32 stream)
>+{
>+	struct v4l2_subdev_stream_configs *stream_configs;
>+	unsigned int i;
>+
>+	lockdep_assert_held(state->lock);
>+
>+	stream_configs = &state->stream_configs;
>+
>+	for (i = 0; i < stream_configs->num_configs; ++i) {
>+		if (stream_configs->configs[i].pad == pad &&
>+		    stream_configs->configs[i].stream == stream)
>+			return &stream_configs->configs[i].fmt;
>+	}
>+
>+	return NULL;
>+}
>+EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_stream_format);
>+
>+struct v4l2_rect *
>+v4l2_subdev_state_get_stream_crop(struct v4l2_subdev_state *state,
>+				  unsigned int pad, u32 stream)
>+{
>+	struct v4l2_subdev_stream_configs *stream_configs;
>+	unsigned int i;
>+
>+	lockdep_assert_held(state->lock);
>+
>+	stream_configs = &state->stream_configs;
>+
>+	for (i = 0; i < stream_configs->num_configs; ++i) {
>+		if (stream_configs->configs[i].pad == pad &&
>+		    stream_configs->configs[i].stream == stream)
>+			return &stream_configs->configs[i].crop;
>+	}
>+
>+	return NULL;
>+}
>+EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_stream_crop);
>+
>+struct v4l2_rect *
>+v4l2_subdev_state_get_stream_compose(struct v4l2_subdev_state *state,
>+				     unsigned int pad, u32 stream)
>+{
>+	struct v4l2_subdev_stream_configs *stream_configs;
>+	unsigned int i;
>+
>+	lockdep_assert_held(state->lock);
>+
>+	stream_configs = &state->stream_configs;
>+
>+	for (i = 0; i < stream_configs->num_configs; ++i) {
>+		if (stream_configs->configs[i].pad == pad &&
>+		    stream_configs->configs[i].stream == stream)
>+			return &stream_configs->configs[i].compose;
>+	}
>+
>+	return NULL;
>+}
>+EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_stream_compose);
>+
> #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
>
> #endif /* CONFIG_MEDIA_CONTROLLER */
>diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
>index 89e58208e330..d6273ad2eea8 100644
>--- a/include/media/v4l2-subdev.h
>+++ b/include/media/v4l2-subdev.h
>@@ -695,6 +695,37 @@ struct v4l2_subdev_pad_config {
> 	struct v4l2_rect try_compose;
> };
>
>+/**
>+ * struct v4l2_subdev_stream_config - Used for storing stream configuration.
>+ *
>+ * @pad: pad number
>+ * @stream: stream number
>+ * @fmt: &struct v4l2_mbus_framefmt
>+ * @crop: &struct v4l2_rect to be used for crop
>+ * @compose: &struct v4l2_rect to be used for compose
>+ *
>+ * This structure stores configuration for a stream.
>+ */
>+struct v4l2_subdev_stream_config {
>+	u32 pad;
>+	u32 stream;
>+
>+	struct v4l2_mbus_framefmt fmt;
>+	struct v4l2_rect crop;
>+	struct v4l2_rect compose;
>+};
>+
>+/**
>+ * struct v4l2_subdev_stream_configs - A collection of stream configs.
>+ *
>+ * @num_configs: number of entries in @config.
>+ * @configs: an array of &struct v4l2_subdev_stream_configs.
>+ */
>+struct v4l2_subdev_stream_configs {
>+	u32 num_configs;
>+	struct v4l2_subdev_stream_config *configs;
>+};
>+
> /**
>  * struct v4l2_subdev_krouting - subdev routing table
>  *
>@@ -715,6 +746,7 @@ struct v4l2_subdev_krouting {
>  * @lock: mutex for the state. May be replaced by the user.
>  * @pads: &struct v4l2_subdev_pad_config array
>  * @routing: routing table for the subdev
>+ * @stream_configs: stream configurations (only for V4L2_SUBDEV_FL_STREAMS)
>  *
>  * This structure only needs to be passed to the pad op if the 'which' field
>  * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
>@@ -726,6 +758,7 @@ struct v4l2_subdev_state {
> 	struct mutex *lock;
> 	struct v4l2_subdev_pad_config *pads;
> 	struct v4l2_subdev_krouting routing;
>+	struct v4l2_subdev_stream_configs stream_configs;
> };
>
> /**
>@@ -1448,6 +1481,52 @@ __v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting *routing,
> 	for ((route) = NULL;                  \
> 	     ((route) = __v4l2_subdev_next_active_route((routing), (route)));)
>
>+/**
>+ * v4l2_subdev_state_get_stream_format() - Get pointer to a stream format
>+ * @state: subdevice state
>+ * @pad: pad id
>+ * @stream: stream id
>+ *
>+ * This returns a pointer to &struct v4l2_mbus_framefmt for the given pad +
>+ * stream in the subdev state.
>+ *
>+ * If the state does not contain the given pad + stream, NULL is returned.
>+ */
>+struct v4l2_mbus_framefmt *
>+v4l2_subdev_state_get_stream_format(struct v4l2_subdev_state *state,
>+				    unsigned int pad, u32 stream);
>+
>+/**
>+ * v4l2_subdev_state_get_stream_crop() - Get pointer to a stream crop rectangle
>+ * @state: subdevice state
>+ * @pad: pad id
>+ * @stream: stream id
>+ *
>+ * This returns a pointer to crop rectangle for the given pad + stream in the
>+ * subdev state.
>+ *
>+ * If the state does not contain the given pad + stream, NULL is returned.
>+ */
>+struct v4l2_rect *
>+v4l2_subdev_state_get_stream_crop(struct v4l2_subdev_state *state,
>+				  unsigned int pad, u32 stream);
>+
>+/**
>+ * v4l2_subdev_state_get_stream_compose() - Get pointer to a stream compose
>+ *					    rectangle
>+ * @state: subdevice state
>+ * @pad: pad id
>+ * @stream: stream id
>+ *
>+ * This returns a pointer to compose rectangle for the given pad + stream in the
>+ * subdev state.
>+ *
>+ * If the state does not contain the given pad + stream, NULL is returned.
>+ */
>+struct v4l2_rect *
>+v4l2_subdev_state_get_stream_compose(struct v4l2_subdev_state *state,
>+				     unsigned int pad, u32 stream);
>+
> #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
>
> #endif /* CONFIG_MEDIA_CONTROLLER */
>diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
>index db4af236d7ed..3ba76d0c7a50 100644
>--- a/include/uapi/linux/v4l2-subdev.h
>+++ b/include/uapi/linux/v4l2-subdev.h
>@@ -45,13 +45,15 @@ enum v4l2_subdev_format_whence {
>  * @which: format type (from enum v4l2_subdev_format_whence)
>  * @pad: pad number, as reported by the media API
>  * @format: media bus format (format code and frame size)
>+ * @stream: stream number, defined in subdev routing
>  * @reserved: drivers and applications must zero this array
>  */
> struct v4l2_subdev_format {
> 	__u32 which;
> 	__u32 pad;
> 	struct v4l2_mbus_framefmt format;
>-	__u32 reserved[8];
>+	__u32 stream;
>+	__u32 reserved[7];
> };
>
> /**
>@@ -59,13 +61,15 @@ struct v4l2_subdev_format {
>  * @which: format type (from enum v4l2_subdev_format_whence)
>  * @pad: pad number, as reported by the media API
>  * @rect: pad crop rectangle boundaries
>+ * @stream: stream number, defined in subdev routing
>  * @reserved: drivers and applications must zero this array
>  */
> struct v4l2_subdev_crop {
> 	__u32 which;
> 	__u32 pad;
> 	struct v4l2_rect rect;
>-	__u32 reserved[8];
>+	__u32 stream;
>+	__u32 reserved[7];
> };
>
> #define V4L2_SUBDEV_MBUS_CODE_CSC_COLORSPACE	0x00000001
>@@ -81,6 +85,7 @@ struct v4l2_subdev_crop {
>  * @code: format code (MEDIA_BUS_FMT_ definitions)
>  * @which: format type (from enum v4l2_subdev_format_whence)
>  * @flags: flags set by the driver, (V4L2_SUBDEV_MBUS_CODE_*)
>+ * @stream: stream number, defined in subdev routing
>  * @reserved: drivers and applications must zero this array
>  */
> struct v4l2_subdev_mbus_code_enum {
>@@ -89,7 +94,8 @@ struct v4l2_subdev_mbus_code_enum {
> 	__u32 code;
> 	__u32 which;
> 	__u32 flags;
>-	__u32 reserved[7];
>+	__u32 stream;
>+	__u32 reserved[6];
> };
>
> /**
>@@ -102,6 +108,7 @@ struct v4l2_subdev_mbus_code_enum {
>  * @min_height: minimum frame height, in pixels
>  * @max_height: maximum frame height, in pixels
>  * @which: format type (from enum v4l2_subdev_format_whence)
>+ * @stream: stream number, defined in subdev routing
>  * @reserved: drivers and applications must zero this array
>  */
> struct v4l2_subdev_frame_size_enum {
>@@ -113,19 +120,22 @@ struct v4l2_subdev_frame_size_enum {
> 	__u32 min_height;
> 	__u32 max_height;
> 	__u32 which;
>-	__u32 reserved[8];
>+	__u32 stream;
>+	__u32 reserved[7];
> };
>
> /**
>  * struct v4l2_subdev_frame_interval - Pad-level frame rate
>  * @pad: pad number, as reported by the media API
>  * @interval: frame interval in seconds
>+ * @stream: stream number, defined in subdev routing
>  * @reserved: drivers and applications must zero this array
>  */
> struct v4l2_subdev_frame_interval {
> 	__u32 pad;
> 	struct v4l2_fract interval;
>-	__u32 reserved[9];
>+	__u32 stream;
>+	__u32 reserved[8];
> };
>
> /**
>@@ -137,6 +147,7 @@ struct v4l2_subdev_frame_interval {
>  * @height: frame height in pixels
>  * @interval: frame interval in seconds
>  * @which: format type (from enum v4l2_subdev_format_whence)
>+ * @stream: stream number, defined in subdev routing
>  * @reserved: drivers and applications must zero this array
>  */
> struct v4l2_subdev_frame_interval_enum {
>@@ -147,7 +158,8 @@ struct v4l2_subdev_frame_interval_enum {
> 	__u32 height;
> 	struct v4l2_fract interval;
> 	__u32 which;
>-	__u32 reserved[8];
>+	__u32 stream;
>+	__u32 reserved[7];
> };
>
> /**
>@@ -159,6 +171,7 @@ struct v4l2_subdev_frame_interval_enum {
>  *	    defined in v4l2-common.h; V4L2_SEL_TGT_* .
>  * @flags: constraint flags, defined in v4l2-common.h; V4L2_SEL_FLAG_*.
>  * @r: coordinates of the selection window
>+ * @stream: stream number, defined in subdev routing
>  * @reserved: for future use, set to zero for now
>  *
>  * Hardware may use multiple helper windows to process a video stream.
>@@ -171,7 +184,8 @@ struct v4l2_subdev_selection {
> 	__u32 target;
> 	__u32 flags;
> 	struct v4l2_rect r;
>-	__u32 reserved[8];
>+	__u32 stream;
>+	__u32 reserved[7];
> };
>
> /**
>-- 
>2.34.1
>

Re: [PATCH v15 12/19] media: subdev: add "opposite" stream helper funcs

[Dafna Hirschfeld]

On 03.10.2022 15:18, Tomi Valkeinen wrote:
>Add two helper functions to make dealing with streams easier:
>
>v4l2_subdev_routing_find_opposite_end - given a routing table and a pad
>+ stream, return the pad + stream on the opposite side of the subdev.
>
>v4l2_subdev_state_get_opposite_stream_format - return a pointer to the
>format on the pad + stream on the opposite side from the given pad +
>stream.
>
>Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
>Reviewed-by: Jacopo Mondi <jacopo@jmondi.org>
>---
> drivers/media/v4l2-core/v4l2-subdev.c | 49 +++++++++++++++++++++++++++
> include/media/v4l2-subdev.h           | 36 ++++++++++++++++++++
> 2 files changed, 85 insertions(+)
>
>diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>index 1cea6ec750c0..7d2d8e8d3aea 100644
>--- a/drivers/media/v4l2-core/v4l2-subdev.c
>+++ b/drivers/media/v4l2-core/v4l2-subdev.c
>@@ -1514,6 +1514,55 @@ v4l2_subdev_state_get_stream_compose(struct v4l2_subdev_state *state,
> }
> EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_stream_compose);
>
>+int v4l2_subdev_routing_find_opposite_end(const struct v4l2_subdev_krouting *routing,
>+					  u32 pad, u32 stream, u32 *other_pad,
>+					  u32 *other_stream)
>+{
>+	unsigned int i;
>+
>+	for (i = 0; i < routing->num_routes; ++i) {
>+		struct v4l2_subdev_route *route = &routing->routes[i];
>+
>+		if (route->source_pad == pad &&
>+		    route->source_stream == stream) {
>+			if (other_pad)
>+				*other_pad = route->sink_pad;
>+			if (other_stream)
>+				*other_stream = route->sink_stream;
>+			return 0;
>+		}
>+
>+		if (route->sink_pad == pad && route->sink_stream == stream) {

Hi, I think here you should also check that the FL_SOURCE flag is not set to make sure
that sink_pad/stream are used

Thanks,
Dafna

>+			if (other_pad)
>+				*other_pad = route->source_pad;
>+			if (other_stream)
>+				*other_stream = route->source_stream;
>+			return 0;
>+		}
>+	}
>+
>+	return -EINVAL;
>+}
>+EXPORT_SYMBOL_GPL(v4l2_subdev_routing_find_opposite_end);
>+
>+struct v4l2_mbus_framefmt *
>+v4l2_subdev_state_get_opposite_stream_format(struct v4l2_subdev_state *state,
>+					     u32 pad, u32 stream)
>+{
>+	u32 other_pad, other_stream;
>+	int ret;
>+
>+	ret = v4l2_subdev_routing_find_opposite_end(&state->routing,
>+						    pad, stream,
>+						    &other_pad, &other_stream);
>+	if (ret)
>+		return NULL;
>+
>+	return v4l2_subdev_state_get_stream_format(state, other_pad,
>+						   other_stream);
>+}
>+EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_opposite_stream_format);
>+
> #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
>
> #endif /* CONFIG_MEDIA_CONTROLLER */
>diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
>index d6273ad2eea8..6f4719e28ad1 100644
>--- a/include/media/v4l2-subdev.h
>+++ b/include/media/v4l2-subdev.h
>@@ -1527,6 +1527,42 @@ struct v4l2_rect *
> v4l2_subdev_state_get_stream_compose(struct v4l2_subdev_state *state,
> 				     unsigned int pad, u32 stream);
>
>+/**
>+ * v4l2_subdev_routing_find_opposite_end() - Find the opposite stream
>+ * @routing: routing used to find the opposite side
>+ * @pad: pad id
>+ * @stream: stream id
>+ * @other_pad: pointer used to return the opposite pad
>+ * @other_stream: pointer used to return the opposite stream
>+ *
>+ * This function uses the routing table to find the pad + stream which is
>+ * opposite the given pad + stream.
>+ *
>+ * @other_pad and/or @other_stream can be NULL if the caller does not need the
>+ * value.
>+ *
>+ * Returns 0 on success, or -EINVAL if no matching route is found.
>+ */
>+int v4l2_subdev_routing_find_opposite_end(const struct v4l2_subdev_krouting *routing,
>+					  u32 pad, u32 stream, u32 *other_pad,
>+					  u32 *other_stream);
>+
>+/**
>+ * v4l2_subdev_state_get_opposite_stream_format() - Get pointer to opposite
>+ *                                                  stream format
>+ * @state: subdevice state
>+ * @pad: pad id
>+ * @stream: stream id
>+ *
>+ * This returns a pointer to &struct v4l2_mbus_framefmt for the pad + stream
>+ * that is opposite the given pad + stream in the subdev state.
>+ *
>+ * If the state does not contain the given pad + stream, NULL is returned.
>+ */
>+struct v4l2_mbus_framefmt *
>+v4l2_subdev_state_get_opposite_stream_format(struct v4l2_subdev_state *state,
>+					     u32 pad, u32 stream);
>+
> #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
>
> #endif /* CONFIG_MEDIA_CONTROLLER */
>-- 
>2.34.1
>

Re: [PATCH v15 17/19] media: v4l2-subdev: Add subdev .(enable|disable)_streams() operations

[Dafna Hirschfeld]

On 03.10.2022 15:18, Tomi Valkeinen wrote:
>From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
>
>Add two new subdev pad operations, .enable_streams() and
>.disable_streams(), to allow control of individual streams per pad. This
>is a superset of what the video .s_stream() operation implements.
>
>To help with handling of backward compatibility, add two wrapper
>functions around those operations, and require their usage in drivers.

Hi, what does it mean 'require their usage in drivers'?
Drivers suppose to call these helpers in the s_stream implementation?

Thanks,
Dafna

>
>Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
>Reviewed-by: Jacopo Mondi <jacopo@jmondi.org>
>---
> drivers/media/v4l2-core/v4l2-subdev.c | 224 ++++++++++++++++++++++++++
> include/media/v4l2-subdev.h           |  85 ++++++++++
> 2 files changed, 309 insertions(+)
>
>diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>index cc8666c5069b..101c1cfc0123 100644
>--- a/drivers/media/v4l2-core/v4l2-subdev.c
>+++ b/drivers/media/v4l2-core/v4l2-subdev.c
>@@ -1717,6 +1717,230 @@ int v4l2_subdev_routing_validate(struct v4l2_subdev *sd,
> }
> EXPORT_SYMBOL_GPL(v4l2_subdev_routing_validate);
>
>+static int v4l2_subdev_enable_streams_fallback(struct v4l2_subdev *sd, u32 pad,
>+					       u64 streams_mask)
>+{
>+	struct device *dev = sd->entity.graph_obj.mdev->dev;
>+	unsigned int i;
>+	int ret;
>+
>+	/*
>+	 * The subdev doesn't implement pad-based stream enable, fall back
>+	 * on the .s_stream() operation. This can only be done for subdevs that
>+	 * have a single source pad, as sd->enabled_streams is global to the
>+	 * subdev.
>+	 */
>+	if (!(sd->entity.pads[pad].flags & MEDIA_PAD_FL_SOURCE))
>+		return -EOPNOTSUPP;
>+
>+	for (i = 0; i < sd->entity.num_pads; ++i) {
>+		if (i != pad && sd->entity.pads[i].flags & MEDIA_PAD_FL_SOURCE)
>+			return -EOPNOTSUPP;
>+	}
>+
>+	if (sd->enabled_streams & streams_mask) {
>+		dev_dbg(dev, "set of streams %#llx already enabled on %s:%u\n",
>+			streams_mask, sd->entity.name, pad);
>+		return -EALREADY;
>+	}
>+
>+	/* Start streaming when the first streams are enabled. */
>+	if (!sd->enabled_streams) {
>+		ret = v4l2_subdev_call(sd, video, s_stream, 1);
>+		if (ret)
>+			return ret;
>+	}
>+
>+	sd->enabled_streams |= streams_mask;
>+
>+	return 0;
>+}
>+
>+int v4l2_subdev_enable_streams(struct v4l2_subdev *sd, u32 pad,
>+			       u64 streams_mask)
>+{
>+	struct device *dev = sd->entity.graph_obj.mdev->dev;
>+	struct v4l2_subdev_state *state;
>+	u64 found_streams = 0;
>+	unsigned int i;
>+	int ret;
>+
>+	/* A few basic sanity checks first. */
>+	if (pad >= sd->entity.num_pads)
>+		return -EINVAL;
>+
>+	if (!streams_mask)
>+		return 0;
>+
>+	/* Fallback on .s_stream() if .enable_streams() isn't available. */
>+	if (!sd->ops->pad || !sd->ops->pad->enable_streams)
>+		return v4l2_subdev_enable_streams_fallback(sd, pad,
>+							   streams_mask);
>+
>+	state = v4l2_subdev_lock_and_get_active_state(sd);
>+
>+	/*
>+	 * Verify that the requested streams exist and that they are not
>+	 * already enabled.
>+	 */
>+	for (i = 0; i < state->stream_configs.num_configs; ++i) {
>+		struct v4l2_subdev_stream_config *cfg =
>+			&state->stream_configs.configs[i];
>+
>+		if (cfg->pad != pad || !(streams_mask & BIT_ULL(cfg->stream)))
>+			continue;
>+
>+		found_streams |= BIT_ULL(cfg->stream);
>+
>+		if (cfg->enabled) {
>+			dev_dbg(dev, "stream %u already enabled on %s:%u\n",
>+				cfg->stream, sd->entity.name, pad);
>+			ret = -EALREADY;
>+			goto done;
>+		}
>+	}
>+
>+	if (found_streams != streams_mask) {
>+		dev_dbg(dev, "streams 0x%llx not found on %s:%u\n",
>+			streams_mask & ~found_streams, sd->entity.name, pad);
>+		ret = -EINVAL;
>+		goto done;
>+	}
>+
>+	/* Call the .enable_streams() operation. */
>+	ret = v4l2_subdev_call(sd, pad, enable_streams, state, pad,
>+			       streams_mask);
>+	if (ret)
>+		goto done;
>+
>+	/* Mark the streams as enabled. */
>+	for (i = 0; i < state->stream_configs.num_configs; ++i) {
>+		struct v4l2_subdev_stream_config *cfg =
>+			&state->stream_configs.configs[i];
>+
>+		if (cfg->pad == pad && (streams_mask & BIT_ULL(cfg->stream)))
>+			cfg->enabled = true;
>+	}
>+
>+done:
>+	v4l2_subdev_unlock_state(state);
>+
>+	return ret;
>+}
>+EXPORT_SYMBOL_GPL(v4l2_subdev_enable_streams);
>+
>+static int v4l2_subdev_disable_streams_fallback(struct v4l2_subdev *sd, u32 pad,
>+						u64 streams_mask)
>+{
>+	struct device *dev = sd->entity.graph_obj.mdev->dev;
>+	unsigned int i;
>+	int ret;
>+
>+	/*
>+	 * If the subdev doesn't implement pad-based stream enable, fall  back
>+	 * on the .s_stream() operation. This can only be done for subdevs that
>+	 * have a single source pad, as sd->enabled_streams is global to the
>+	 * subdev.
>+	 */
>+	if (!(sd->entity.pads[pad].flags & MEDIA_PAD_FL_SOURCE))
>+		return -EOPNOTSUPP;
>+
>+	for (i = 0; i < sd->entity.num_pads; ++i) {
>+		if (i != pad && sd->entity.pads[i].flags & MEDIA_PAD_FL_SOURCE)
>+			return -EOPNOTSUPP;
>+	}
>+
>+	if ((sd->enabled_streams & streams_mask) != streams_mask) {
>+		dev_dbg(dev, "set of streams %#llx already disabled on %s:%u\n",
>+			streams_mask, sd->entity.name, pad);
>+		return -EALREADY;
>+	}
>+
>+	/* Stop streaming when the last streams are disabled. */
>+	if (!(sd->enabled_streams & ~streams_mask)) {
>+		ret = v4l2_subdev_call(sd, video, s_stream, 0);
>+		if (ret)
>+			return ret;
>+	}
>+
>+	sd->enabled_streams &= ~streams_mask;
>+
>+	return 0;
>+}
>+
>+int v4l2_subdev_disable_streams(struct v4l2_subdev *sd, u32 pad,
>+				u64 streams_mask)
>+{
>+	struct device *dev = sd->entity.graph_obj.mdev->dev;
>+	struct v4l2_subdev_state *state;
>+	u64 found_streams = 0;
>+	unsigned int i;
>+	int ret;
>+
>+	/* A few basic sanity checks first. */
>+	if (pad >= sd->entity.num_pads)
>+		return -EINVAL;
>+
>+	if (!streams_mask)
>+		return 0;
>+
>+	/* Fallback on .s_stream() if .disable_streams() isn't available. */
>+	if (!sd->ops->pad || !sd->ops->pad->disable_streams)
>+		return v4l2_subdev_disable_streams_fallback(sd, pad,
>+							    streams_mask);
>+
>+	state = v4l2_subdev_lock_and_get_active_state(sd);
>+
>+	/*
>+	 * Verify that the requested streams exist and that they are not
>+	 * already disabled.
>+	 */
>+	for (i = 0; i < state->stream_configs.num_configs; ++i) {
>+		struct v4l2_subdev_stream_config *cfg =
>+			&state->stream_configs.configs[i];
>+
>+		if (cfg->pad != pad || !(streams_mask & BIT_ULL(cfg->stream)))
>+			continue;
>+
>+		found_streams |= BIT_ULL(cfg->stream);
>+
>+		if (!cfg->enabled) {
>+			dev_dbg(dev, "stream %u already disabled on %s:%u\n",
>+				cfg->stream, sd->entity.name, pad);
>+			ret = -EALREADY;
>+			goto done;
>+		}
>+	}
>+
>+	if (found_streams != streams_mask) {
>+		dev_dbg(dev, "streams 0x%llx not found on %s:%u\n",
>+			streams_mask & ~found_streams, sd->entity.name, pad);
>+		ret = -EINVAL;
>+		goto done;
>+	}
>+
>+	/* Call the .disable_streams() operation. */
>+	ret = v4l2_subdev_call(sd, pad, disable_streams, state, pad,
>+			       streams_mask);
>+	if (ret)
>+		goto done;
>+
>+	/* Mark the streams as disabled. */
>+	for (i = 0; i < state->stream_configs.num_configs; ++i) {
>+		struct v4l2_subdev_stream_config *cfg =
>+			&state->stream_configs.configs[i];
>+
>+		if (cfg->pad == pad && (streams_mask & BIT_ULL(cfg->stream)))
>+			cfg->enabled = false;
>+	}
>+
>+done:
>+	v4l2_subdev_unlock_state(state);
>+
>+	return ret;
>+}
>+EXPORT_SYMBOL_GPL(v4l2_subdev_disable_streams);
>+
> #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
>
> #endif /* CONFIG_MEDIA_CONTROLLER */
>diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
>index 8c002d65e08e..bff824367e0b 100644
>--- a/include/media/v4l2-subdev.h
>+++ b/include/media/v4l2-subdev.h
>@@ -700,6 +700,7 @@ struct v4l2_subdev_pad_config {
>  *
>  * @pad: pad number
>  * @stream: stream number
>+ * @enabled: has the stream been enabled with v4l2_subdev_enable_stream()
>  * @fmt: &struct v4l2_mbus_framefmt
>  * @crop: &struct v4l2_rect to be used for crop
>  * @compose: &struct v4l2_rect to be used for compose
>@@ -709,6 +710,7 @@ struct v4l2_subdev_pad_config {
> struct v4l2_subdev_stream_config {
> 	u32 pad;
> 	u32 stream;
>+	bool enabled;
>
> 	struct v4l2_mbus_framefmt fmt;
> 	struct v4l2_rect crop;
>@@ -814,6 +816,18 @@ struct v4l2_subdev_state {
>  *
>  * @set_routing: enable or disable data connection routes described in the
>  *		 subdevice routing table.
>+ *
>+ * @enable_streams: Enable the streams defined in streams_mask on the given
>+ *	source pad. Subdevs that implement this operation must use the active
>+ *	state management provided by the subdev core (enabled through a call to
>+ *	v4l2_subdev_init_finalize() at initialization time). Do not call
>+ *	directly, use v4l2_subdev_enable_streams() instead.
>+ *
>+ * @disable_streams: Disable the streams defined in streams_mask on the given
>+ *	source pad. Subdevs that implement this operation must use the active
>+ *	state management provided by the subdev core (enabled through a call to
>+ *	v4l2_subdev_init_finalize() at initialization time). Do not call
>+ *	directly, use v4l2_subdev_disable_streams() instead.
>  */
> struct v4l2_subdev_pad_ops {
> 	int (*init_cfg)(struct v4l2_subdev *sd,
>@@ -860,6 +874,12 @@ struct v4l2_subdev_pad_ops {
> 			   struct v4l2_subdev_state *state,
> 			   enum v4l2_subdev_format_whence which,
> 			   struct v4l2_subdev_krouting *route);
>+	int (*enable_streams)(struct v4l2_subdev *sd,
>+			      struct v4l2_subdev_state *state, u32 pad,
>+			      u64 streams_mask);
>+	int (*disable_streams)(struct v4l2_subdev *sd,
>+			       struct v4l2_subdev_state *state, u32 pad,
>+			       u64 streams_mask);
> };
>
> /**
>@@ -1005,6 +1025,10 @@ struct v4l2_subdev_platform_data {
>  * @active_state: Active state for the subdev (NULL for subdevs tracking the
>  *		  state internally). Initialized by calling
>  *		  v4l2_subdev_init_finalize().
>+ * @enabled_streams: Bitmask of enabled streams used by
>+ *		     v4l2_subdev_enable_streams() and
>+ *		     v4l2_subdev_disable_streams() helper functions for fallback
>+ *		     cases.
>  *
>  * Each instance of a subdev driver should create this struct, either
>  * stand-alone or embedded in a larger struct.
>@@ -1050,6 +1074,7 @@ struct v4l2_subdev {
> 	 * doesn't support it.
> 	 */
> 	struct v4l2_subdev_state *active_state;
>+	u64 enabled_streams;
> };
>
>
>@@ -1641,6 +1666,66 @@ int v4l2_subdev_routing_validate(struct v4l2_subdev *sd,
> 				 const struct v4l2_subdev_krouting *routing,
> 				 enum v4l2_subdev_routing_restriction disallow);
>
>+/**
>+ * v4l2_subdev_enable_streams() - Enable streams on a pad
>+ * @sd: The subdevice
>+ * @pad: The pad
>+ * @streams_mask: Bitmask of streams to enable
>+ *
>+ * This function enables streams on a source @pad of a subdevice. The pad is
>+ * identified by its index, while the streams are identified by the
>+ * @streams_mask bitmask. This allows enabling multiple streams on a pad at
>+ * once.
>+ *
>+ * Enabling a stream that is already enabled isn't allowed. If @streams_mask
>+ * contains an already enabled stream, this function returns -EALREADY without
>+ * performing any operation.
>+ *
>+ * Per-stream enable is only available for subdevs that implement the
>+ * .enable_streams() and .disable_streams() operations. For other subdevs, this
>+ * function implements a best-effort compatibility by calling the .s_stream()
>+ * operation, limited to subdevs that have a single source pad.
>+ *
>+ * Return:
>+ * * 0: Success
>+ * * -EALREADY: One of the streams in streams_mask is already enabled
>+ * * -EINVAL: The pad index is invalid, or doesn't correspond to a source pad
>+ * * -EOPNOTSUPP: Falling back to the legacy .s_stream() operation is
>+ *   impossible because the subdev has multiple source pads
>+ */
>+int v4l2_subdev_enable_streams(struct v4l2_subdev *sd, u32 pad,
>+			       u64 streams_mask);
>+
>+/**
>+ * v4l2_subdev_disable_streams() - Disable streams on a pad
>+ * @sd: The subdevice
>+ * @pad: The pad
>+ * @streams_mask: Bitmask of streams to disable
>+ *
>+ * This function disables streams on a source @pad of a subdevice. The pad is
>+ * identified by its index, while the streams are identified by the
>+ * @streams_mask bitmask. This allows disabling multiple streams on a pad at
>+ * once.
>+ *
>+ * Disabling a streams that is not enabled isn't allowed. If @streams_mask
>+ * contains a disabled stream, this function returns -EALREADY without
>+ * performing any operation.
>+ *
>+ * Per-stream disable is only available for subdevs that implement the
>+ * .enable_streams() and .disable_streams() operations. For other subdevs, this
>+ * function implements a best-effort compatibility by calling the .s_stream()
>+ * operation, limited to subdevs that have a single source pad.
>+ *
>+ * Return:
>+ * * 0: Success
>+ * * -EALREADY: One of the streams in streams_mask is not enabled
>+ * * -EINVAL: The pad index is invalid, or doesn't correspond to a source pad
>+ * * -EOPNOTSUPP: Falling back to the legacy .s_stream() operation is
>+ *   impossible because the subdev has multiple source pads
>+ */
>+int v4l2_subdev_disable_streams(struct v4l2_subdev *sd, u32 pad,
>+				u64 streams_mask);
>+
> #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
>
> #endif /* CONFIG_MEDIA_CONTROLLER */
>-- 
>2.34.1
>

[PATCH 1/1] media: Documentation: Interaction between routes, formats and selections

[Sakari Ailus]

Document how setting up routes interacts with formats and selections.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
Moi,

This could be added on top of the set.

Comments are welcome.

 .../userspace-api/media/v4l/dev-subdev.rst    | 48 ++++++++++++++-----
 1 file changed, 37 insertions(+), 11 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index 5075b1828b32d..830235eb01598 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -406,6 +406,8 @@ pixel array is not rectangular but cross-shaped or round. The maximum
 size may also be smaller than the BOUNDS rectangle.
 
 
+.. _format-propagation:
+
 Order of configuration and format propagation
 ---------------------------------------------
 
@@ -507,12 +509,12 @@ source pads.
 Streams, multiplexed media pads and internal routing
 ----------------------------------------------------
 
-Commonly V4L2 subdevices support only separate video streams, that is, only a
-single stream can pass through a media link and a media pad. Thus each pad
-contains a format configuration for that single stream. In some cases a subdev
-can do stream processing and split a stream into two or compose two streams
-into one, but the inputs and outputs for the subdev are still a single stream
-per pad.
+Commonly V4L2 subdevices do not support multiple, unrelated video streams.
+Only a single stream can pass through a media link and a media
+pad. Thus each pad contains a format and selection configuration for that single stream.
+A subdev can do stream processing and split a stream into
+two or compose two streams into one, but the inputs and outputs for the
+subdev are still a single stream per pad.
 
 Some hardware, e.g. MIPI CSI-2, support multiplexed streams, that is, multiple
 data streams are transmitted on the same bus, which is represented by a media
@@ -539,15 +541,37 @@ streams from one end of the link to the other, and subdevices have routing
 tables which describe how the incoming streams from sink pads are routed to the
 source pads.
 
-A stream ID (often just "stream") is a media link-local identifier for a stream.
+A stream ID is a media link-local identifier for a stream.
 In other words, a particular stream ID must exist on both sides of a media
 link, but another stream ID can be used for the same stream at the other side
-of the subdevice.
+of the subdevice. The same stream ID is used to refer to the stream on
+both pads of the link on all ioctls operating on pads.
 
 A stream at a specific point in the media pipeline is identified with the
-subdev and a (pad, stream) pair. For subdevices that do not support
+subdev and a pad--stream pair. For subdevices that do not support
 multiplexed streams the 'stream' is always 0.
 
+Interaction between routes, formats and selections
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The addition of routes to the V4L2 sub-device interface moves the
+sub-device formats and selections from pads to pad--stream pairs. Instead
+of the sub-device wide merging of streams from all source pads towards all
+sink pads, this takes place separately for each route. The stream ID on
+the sink pad for each configured route is used to configure format and
+selections on the sink pad. Similarly, the stream ID on the source pad of
+each configured route is used to configure format and selections on the
+source pad.
+
+Any number of routes from streams on sink pads towards streams on source
+pads is allowed, to the extent supported by drivers. For every stream on a
+sink pad, however, only a single route is allowed.
+
+Pad--stream pairs are not static but are replaced by calls to the
+:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. This also
+applies to stream format and selection configurations which that are
+reverted to driver defaults.
+
 Configuring streams
 ^^^^^^^^^^^^^^^^^^^
 
@@ -565,8 +589,10 @@ Controller API <media_controller>`
 setting the routing table will reset all the stream configurations in a media
 entity.
 
-3) Configure streams. Each route endpoint must be configured
-with :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>`.
+3) Configure formats and selections on routes. Each route is configured
+separately as documented plain subdevices in :ref:`<format-propagation>`.
+The stream ID is set to the same stream ID used on sink and source pads of
+the :ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl.
 
 Multiplexed streams setup example
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-- 
2.30.2

Re: [PATCH v15 17/19] media: v4l2-subdev: Add subdev .(enable|disable)_streams() operations

[Tomi Valkeinen]

Hi,

On 10/10/2022 19:53, Dafna Hirschfeld wrote:
> On 03.10.2022 15:18, Tomi Valkeinen wrote:
>> From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
>>
>> Add two new subdev pad operations, .enable_streams() and
>> .disable_streams(), to allow control of individual streams per pad. This
>> is a superset of what the video .s_stream() operation implements.
>>
>> To help with handling of backward compatibility, add two wrapper
>> functions around those operations, and require their usage in drivers.
> 
> Hi, what does it mean 'require their usage in drivers'?
> Drivers suppose to call these helpers in the s_stream implementation?

The drivers should use v4l2_subdev_enable_streams() and 
v4l2_subdev_disable_streams() when they want to enable/disable the 
source subdevice (which is most likely in .s_stream or 
.enable_streams/.disable_streams).

The "require" there means that drivers should never call 
.enable_streams/.disable_streams directly, as mentioned in their help 
texts, but instead they should always call the helpers.

  Tomi

Re: [PATCH v15 08/19] media: subdev: Add for_each_active_route() macro

[Tomi Valkeinen]

On 09/10/2022 08:38, Dafna Hirschfeld wrote:
> On 03.10.2022 15:18, Tomi Valkeinen wrote:
>> From: Jacopo Mondi <jacopo+renesas@jmondi.org>
>>
>> Add a for_each_active_route() macro to replace the repeated pattern
>> of iterating on the active routes of a routing table.
>>
>> Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>> .clang-format                         |  1 +
>> drivers/media/v4l2-core/v4l2-subdev.c | 20 ++++++++++++++++++++
>> include/media/v4l2-subdev.h           | 13 +++++++++++++
>> 3 files changed, 34 insertions(+)
>>
>> diff --git a/.clang-format b/.clang-format
>> index 1247d54f9e49..31f39ae78f7b 100644
>> --- a/.clang-format
>> +++ b/.clang-format
>> @@ -190,6 +190,7 @@ ForEachMacros:
>>   - 'for_each_active_dev_scope'
>>   - 'for_each_active_drhd_unit'
>>   - 'for_each_active_iommu'
>> +  - 'for_each_active_route'
>>   - 'for_each_aggr_pgid'
>>   - 'for_each_available_child_of_node'
>>   - 'for_each_bench'
>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c 
>> b/drivers/media/v4l2-core/v4l2-subdev.c
>> index 3ae4f39a50e4..1049c07d2e49 100644
>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>> @@ -1212,6 +1212,26 @@ int v4l2_subdev_set_routing(struct v4l2_subdev 
>> *sd,
>> }
>> EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing);
>>
>> +struct v4l2_subdev_route *
>> +__v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting 
>> *routing,
>> +                struct v4l2_subdev_route *route)
>> +{
>> +    if (route)
>> +        ++route;
>> +    else
>> +        route = &routing->routes[0];
>> +
>> +    for (; route < routing->routes + routing->num_routes; ++route) {
>> +        if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
>> +            continue;
>> +
>> +        return route;
>> +    }
>> +
>> +    return NULL;
>> +}
>> +EXPORT_SYMBOL_GPL(__v4l2_subdev_next_active_route);
>> +
>> #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
>>
>> #endif /* CONFIG_MEDIA_CONTROLLER */
>> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
>> index 7962e6572bda..89e58208e330 100644
>> --- a/include/media/v4l2-subdev.h
>> +++ b/include/media/v4l2-subdev.h
>> @@ -1435,6 +1435,19 @@ int v4l2_subdev_set_routing(struct v4l2_subdev 
>> *sd,
>>                 struct v4l2_subdev_state *state,
>>                 const struct v4l2_subdev_krouting *routing);
>>
>> +struct v4l2_subdev_route *
>> +__v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting 
>> *routing,
>> +                struct v4l2_subdev_route *route);
>> +
>> +/**
>> + * for_each_active_route - iterate on all active routes of a routing 
>> table
>> + * @routing: The routing table
>> + * @route: The route iterator
>> + */
>> +#define for_each_active_route(routing, route) \
>> +    for ((route) = NULL;                  \
>> +         ((route) = __v4l2_subdev_next_active_route((routing), 
>> (route)));)
> Hi, shouldn't it be something like:
>      for ((route) = NULL; (route) ; (route) = 
> __v4l2_subdev_next_active_route((routing), (route)))

What you suggest would never do anything: you initialize route to NULL, 
and then check if the route is !NULL.

I also find the current version a bit "interesting", but afaics, it 
works correctly.

  Tomi

Re: [PATCH v15 07/19] media: subdev: add v4l2_subdev_set_routing helper()

[Yunke Cao]

Hi Tomi,

On Wed, Oct 12, 2022 at 2:03 PM Tomi Valkeinen
<tomi.valkeinen@ideasonboard.com> wrote:
>
> Add a helper function to set the subdev routing. The helper can be used
> from subdev driver's set_routing op to store the routing table.
>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 31 +++++++++++++++++++++++++++
>  include/media/v4l2-subdev.h           | 16 ++++++++++++++
>  2 files changed, 47 insertions(+)
>
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index fff17b8536fc..3ae4f39a50e4 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -12,6 +12,7 @@
>  #include <linux/ioctl.h>
>  #include <linux/mm.h>
>  #include <linux/module.h>
> +#include <linux/overflow.h>
>  #include <linux/slab.h>
>  #include <linux/types.h>
>  #include <linux/version.h>
> @@ -1181,6 +1182,36 @@ int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_get_fmt);
>
> +int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
> +                           struct v4l2_subdev_state *state,
> +                           const struct v4l2_subdev_krouting *routing)
> +{
> +       struct v4l2_subdev_krouting *dst = &state->routing;
> +       const struct v4l2_subdev_krouting *src = routing;
> +       struct v4l2_subdev_krouting new_routing = { 0 };
> +       size_t bytes;
> +
> +       if (unlikely(check_mul_overflow(src->num_routes, sizeof(*src->routes),

Do we need to cast (size_t)src->num_routes here?
My compiler is complaining:
./include/linux/overflow.h:85:22: error: comparison of distinct
pointer types lacks a cast [-Werror]
   85 |         (void) (&__a == &__b);                  \

Thanks,
Yunke


> +                                       &bytes)))
> +               return -EOVERFLOW;
> +
> +       lockdep_assert_held(state->lock);
> +
> +       if (src->num_routes > 0) {
> +               new_routing.routes = kmemdup(src->routes, bytes, GFP_KERNEL);
> +               if (!new_routing.routes)
> +                       return -ENOMEM;
> +       }
> +
> +       new_routing.num_routes = src->num_routes;
> +
> +       kfree(dst->routes);
> +       *dst = new_routing;
> +
> +       return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing);
> +
>  #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
>
>  #endif /* CONFIG_MEDIA_CONTROLLER */
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 45c41f4d6a2b..7962e6572bda 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -1419,6 +1419,22 @@ v4l2_subdev_lock_and_get_active_state(struct v4l2_subdev *sd)
>  int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
>                         struct v4l2_subdev_format *format);
>
> +/**
> + * v4l2_subdev_set_routing() - Set given routing to subdev state
> + * @sd: The subdevice
> + * @state: The subdevice state
> + * @routing: Routing that will be copied to subdev state
> + *
> + * This will release old routing table (if any) from the state, allocate
> + * enough space for the given routing, and copy the routing.
> + *
> + * This can be used from the subdev driver's set_routing op, after validating
> + * the routing.
> + */
> +int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
> +                           struct v4l2_subdev_state *state,
> +                           const struct v4l2_subdev_krouting *routing);
> +
>  #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
>
>  #endif /* CONFIG_MEDIA_CONTROLLER */
> --
> 2.34.1
>
>

Re: [PATCH v15 10/19] media: subdev: add stream based configuration

[Tomi Valkeinen]

On 09/10/2022 09:24, Dafna Hirschfeld wrote:
> On 03.10.2022 15:18, Tomi Valkeinen wrote:
>> Add support to manage configurations (format, crop, compose) per stream,
>> instead of per pad. This is accomplished with data structures that hold
>> an array of all subdev's stream configurations.
>>
>> The number of streams can vary at runtime based on routing. Every time
>> the routing is changed, the stream configurations need to be
>> re-initialized.
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>> .../v4l/vidioc-subdev-enum-frame-interval.rst |   5 +-
>> .../v4l/vidioc-subdev-enum-frame-size.rst     |   5 +-
>> .../v4l/vidioc-subdev-enum-mbus-code.rst      |   5 +-
>> .../media/v4l/vidioc-subdev-g-crop.rst        |   5 +-
>> .../media/v4l/vidioc-subdev-g-fmt.rst         |   5 +-
>> .../v4l/vidioc-subdev-g-frame-interval.rst    |   5 +-
>> .../media/v4l/vidioc-subdev-g-selection.rst   |   5 +-
>> drivers/media/v4l2-core/v4l2-subdev.c         | 154 +++++++++++++++++-
>> include/media/v4l2-subdev.h                   |  79 +++++++++
>> include/uapi/linux/v4l2-subdev.h              |  28 +++-
>> 10 files changed, 275 insertions(+), 21 deletions(-)
>>
>> diff --git 
>> a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
>> index 3703943b412f..8def4c05d3da 100644
>> --- 
>> a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
>> +++ 
>> b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
>> @@ -92,7 +92,10 @@ multiple pads of the same sub-device is not defined.
>>       - Frame intervals to be enumerated, from enum
>>     :ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
>>     * - __u32
>> -      - ``reserved``\ [8]
>> +      - ``stream``
>> +      - Stream identifier.
>> +    * - __u32
>> +      - ``reserved``\ [7]
>>       - Reserved for future extensions. Applications and drivers must set
>>     the array to zero.
>>
>> diff --git 
>> a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
>> index c25a9896df0e..3ef361c0dca7 100644
>> --- 
>> a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
>> +++ 
>> b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
>> @@ -97,7 +97,10 @@ information about try formats.
>>       - Frame sizes to be enumerated, from enum
>>     :ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
>>     * - __u32
>> -      - ``reserved``\ [8]
>> +      - ``stream``
>> +      - Stream identifier.
>> +    * - __u32
>> +      - ``reserved``\ [7]
>>       - Reserved for future extensions. Applications and drivers must set
>>     the array to zero.
>>
>> diff --git 
>> a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
>> index 417f1a19bcc4..248f6f9ee7c5 100644
>> --- 
>> a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
>> +++ 
>> b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
>> @@ -73,7 +73,10 @@ information about the try formats.
>>       - ``flags``
>>       - See :ref:`v4l2-subdev-mbus-code-flags`
>>     * - __u32
>> -      - ``reserved``\ [7]
>> +      - ``stream``
>> +      - Stream identifier.
>> +    * - __u32
>> +      - ``reserved``\ [6]
>>       - Reserved for future extensions. Applications and drivers must set
>>     the array to zero.
>>
>> diff --git 
>> a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst 
>> b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
>> index bd15c0a5a66b..1d267f7e7991 100644
>> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
>> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
>> @@ -96,7 +96,10 @@ modified format should be as close as possible to 
>> the original request.
>>       - ``rect``
>>       - Crop rectangle boundaries, in pixels.
>>     * - __u32
>> -      - ``reserved``\ [8]
>> +      - ``stream``
>> +      - Stream identifier.
>> +    * - __u32
>> +      - ``reserved``\ [7]
>>       - Reserved for future extensions. Applications and drivers must set
>>     the array to zero.
>>
>> diff --git 
>> a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst 
>> b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
>> index 7acdbb939d89..ed253a1e44b7 100644
>> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
>> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
>> @@ -102,7 +102,10 @@ should be as close as possible to the original 
>> request.
>>       - Definition of an image format, see 
>> :c:type:`v4l2_mbus_framefmt` for
>>     details.
>>     * - __u32
>> -      - ``reserved``\ [8]
>> +      - ``stream``
>> +      - Stream identifier.
>> +    * - __u32
>> +      - ``reserved``\ [7]
>>       - Reserved for future extensions. Applications and drivers must set
>>     the array to zero.
>>
>> diff --git 
>> a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
>> index d7fe7543c506..842f962d2aea 100644
>> --- 
>> a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
>> +++ 
>> b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
>> @@ -90,7 +90,10 @@ the same sub-device is not defined.
>>       - ``interval``
>>       - Period, in seconds, between consecutive video frames.
>>     * - __u32
>> -      - ``reserved``\ [9]
>> +      - ``stream``
>> +      - Stream identifier.
>> +    * - __u32
>> +      - ``reserved``\ [8]
>>       - Reserved for future extensions. Applications and drivers must set
>>     the array to zero.
>>
>> diff --git 
>> a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst 
>> b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
>> index f9172a42f036..6b629c19168c 100644
>> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
>> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
>> @@ -94,7 +94,10 @@ Selection targets and flags are documented in
>>       - ``r``
>>       - Selection rectangle, in pixels.
>>     * - __u32
>> -      - ``reserved``\ [8]
>> +      - ``stream``
>> +      - Stream identifier.
>> +    * - __u32
>> +      - ``reserved``\ [7]
>>       - Reserved for future extensions. Applications and drivers must set
>>     the array to zero.
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c 
>> b/drivers/media/v4l2-core/v4l2-subdev.c
>> index 1049c07d2e49..be778e619694 100644
>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>> @@ -159,8 +159,22 @@ static inline int check_pad(struct v4l2_subdev 
>> *sd, u32 pad)
>>     return 0;
>> }
>>
>> -static int check_state_pads(u32 which, struct v4l2_subdev_state *state)
>> +static int check_state(struct v4l2_subdev *sd, struct 
>> v4l2_subdev_state *state,
>> +               u32 which, u32 pad, u32 stream)
>> {
>> +    if (sd->flags & V4L2_SUBDEV_FL_STREAMS) {
>> +#if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
>> +        if (!v4l2_subdev_state_get_stream_format(state, pad, stream))
>> +            return -EINVAL;
>> +        return 0;
>> +#else
>> +        return -EINVAL;
>> +#endif
>> +    }
>> +
>> +    if (stream != 0)
>> +        return -EINVAL;
>> +
>>     if (which == V4L2_SUBDEV_FORMAT_TRY && (!state || !state->pads))
>>         return -EINVAL;
>>
>> @@ -175,7 +189,7 @@ static inline int check_format(struct v4l2_subdev 
>> *sd,
>>         return -EINVAL;
>>
>>     return check_which(format->which) ? : check_pad(sd, format->pad) ? :
>> -           check_state_pads(format->which, state);
>> +           check_state(sd, state, format->which, format->pad, 
>> format->stream);
>> }
>>
>> static int call_get_fmt(struct v4l2_subdev *sd,
>> @@ -202,7 +216,7 @@ static int call_enum_mbus_code(struct v4l2_subdev 
>> *sd,
>>         return -EINVAL;
>>
>>     return check_which(code->which) ? : check_pad(sd, code->pad) ? :
>> -           check_state_pads(code->which, state) ? :
>> +           check_state(sd, state, code->which, code->pad, 
>> code->stream) ? :
>>            sd->ops->pad->enum_mbus_code(sd, state, code);
>> }
>>
>> @@ -214,7 +228,7 @@ static int call_enum_frame_size(struct v4l2_subdev 
>> *sd,
>>         return -EINVAL;
>>
>>     return check_which(fse->which) ? : check_pad(sd, fse->pad) ? :
>> -           check_state_pads(fse->which, state) ? :
>> +           check_state(sd, state, fse->which, fse->pad, fse->stream) ? :
>>            sd->ops->pad->enum_frame_size(sd, state, fse);
>> }
>>
>> @@ -249,7 +263,7 @@ static int call_enum_frame_interval(struct 
>> v4l2_subdev *sd,
>>         return -EINVAL;
>>
>>     return check_which(fie->which) ? : check_pad(sd, fie->pad) ? :
>> -           check_state_pads(fie->which, state) ? :
>> +           check_state(sd, state, fie->which, fie->pad, fie->stream) ? :
>>            sd->ops->pad->enum_frame_interval(sd, state, fie);
>> }
>>
>> @@ -261,7 +275,7 @@ static inline int check_selection(struct 
>> v4l2_subdev *sd,
>>         return -EINVAL;
>>
>>     return check_which(sel->which) ? : check_pad(sd, sel->pad) ? :
>> -           check_state_pads(sel->which, state);
>> +           check_state(sd, state, sel->which, sel->pad, sel->stream);
>> }
>>
>> static int call_get_selection(struct v4l2_subdev *sd,
>> @@ -1095,7 +1109,8 @@ __v4l2_subdev_state_alloc(struct v4l2_subdev 
>> *sd, const char *lock_name,
>>     else
>>         state->lock = &state->_lock;
>>
>> -    if (sd->entity.num_pads) {
>> +    /* Drivers that support streams do not need the legacy pad config */
>> +    if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS) && sd->entity.num_pads) {
>>         state->pads = kvcalloc(sd->entity.num_pads,
>>                        sizeof(*state->pads), GFP_KERNEL);
>>         if (!state->pads) {
>> @@ -1135,6 +1150,7 @@ void __v4l2_subdev_state_free(struct 
>> v4l2_subdev_state *state)
>>     mutex_destroy(&state->_lock);
>>
>>     kfree(state->routing.routes);
>> +    kvfree(state->stream_configs.configs);
>>     kvfree(state->pads);
>>     kfree(state);
>> }
>> @@ -1164,6 +1180,59 @@ EXPORT_SYMBOL_GPL(v4l2_subdev_cleanup);
>>
>> #if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
>>
>> +static int
>> +v4l2_subdev_init_stream_configs(struct v4l2_subdev_stream_configs 
>> *stream_configs,
>> +                const struct v4l2_subdev_krouting *routing)
>> +{
>> +    struct v4l2_subdev_stream_configs new_configs = { 0 };
>> +    struct v4l2_subdev_route *route;
>> +    u32 format_idx = 0;
> I think you don't need both format_idx and idx,
> also, it can be 'int'

Yes, we can do with just a single variable.

I'll keep it as unsigned, as it's an index to an array. Signed doesn't 
make sense there.

  Tomi

Re: [PATCH v15 12/19] media: subdev: add "opposite" stream helper funcs

[Tomi Valkeinen]

On 09/10/2022 10:06, Dafna Hirschfeld wrote:
> On 03.10.2022 15:18, Tomi Valkeinen wrote:
>> Add two helper functions to make dealing with streams easier:
>>
>> v4l2_subdev_routing_find_opposite_end - given a routing table and a pad
>> + stream, return the pad + stream on the opposite side of the subdev.
>>
>> v4l2_subdev_state_get_opposite_stream_format - return a pointer to the
>> format on the pad + stream on the opposite side from the given pad +
>> stream.
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
>> Reviewed-by: Jacopo Mondi <jacopo@jmondi.org>
>> ---
>> drivers/media/v4l2-core/v4l2-subdev.c | 49 +++++++++++++++++++++++++++
>> include/media/v4l2-subdev.h           | 36 ++++++++++++++++++++
>> 2 files changed, 85 insertions(+)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c 
>> b/drivers/media/v4l2-core/v4l2-subdev.c
>> index 1cea6ec750c0..7d2d8e8d3aea 100644
>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>> @@ -1514,6 +1514,55 @@ v4l2_subdev_state_get_stream_compose(struct 
>> v4l2_subdev_state *state,
>> }
>> EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_stream_compose);
>>
>> +int v4l2_subdev_routing_find_opposite_end(const struct 
>> v4l2_subdev_krouting *routing,
>> +                      u32 pad, u32 stream, u32 *other_pad,
>> +                      u32 *other_stream)
>> +{
>> +    unsigned int i;
>> +
>> +    for (i = 0; i < routing->num_routes; ++i) {
>> +        struct v4l2_subdev_route *route = &routing->routes[i];
>> +
>> +        if (route->source_pad == pad &&
>> +            route->source_stream == stream) {
>> +            if (other_pad)
>> +                *other_pad = route->sink_pad;
>> +            if (other_stream)
>> +                *other_stream = route->sink_stream;
>> +            return 0;
>> +        }
>> +
>> +        if (route->sink_pad == pad && route->sink_stream == stream) {
> 
> Hi, I think here you should also check that the FL_SOURCE flag is not 
> set to make sure
> that sink_pad/stream are used

Yes, you are correct. I'll add the check.

  Tomi

Re: [PATCH v15 07/19] media: subdev: add v4l2_subdev_set_routing helper()

[Tomi Valkeinen]

On 12/10/2022 09:22, Yunke Cao wrote:
> Hi Tomi,
> 
> On Wed, Oct 12, 2022 at 2:03 PM Tomi Valkeinen
> <tomi.valkeinen@ideasonboard.com> wrote:
>>
>> Add a helper function to set the subdev routing. The helper can be used
>> from subdev driver's set_routing op to store the routing table.
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
>> ---
>>   drivers/media/v4l2-core/v4l2-subdev.c | 31 +++++++++++++++++++++++++++
>>   include/media/v4l2-subdev.h           | 16 ++++++++++++++
>>   2 files changed, 47 insertions(+)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>> index fff17b8536fc..3ae4f39a50e4 100644
>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>> @@ -12,6 +12,7 @@
>>   #include <linux/ioctl.h>
>>   #include <linux/mm.h>
>>   #include <linux/module.h>
>> +#include <linux/overflow.h>
>>   #include <linux/slab.h>
>>   #include <linux/types.h>
>>   #include <linux/version.h>
>> @@ -1181,6 +1182,36 @@ int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
>>   }
>>   EXPORT_SYMBOL_GPL(v4l2_subdev_get_fmt);
>>
>> +int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
>> +                           struct v4l2_subdev_state *state,
>> +                           const struct v4l2_subdev_krouting *routing)
>> +{
>> +       struct v4l2_subdev_krouting *dst = &state->routing;
>> +       const struct v4l2_subdev_krouting *src = routing;
>> +       struct v4l2_subdev_krouting new_routing = { 0 };
>> +       size_t bytes;
>> +
>> +       if (unlikely(check_mul_overflow(src->num_routes, sizeof(*src->routes),
> 
> Do we need to cast (size_t)src->num_routes here?
> My compiler is complaining:
> ./include/linux/overflow.h:85:22: error: comparison of distinct
> pointer types lacks a cast [-Werror]
>     85 |         (void) (&__a == &__b);                  \
> 

Yes, I think we should do that. Thanks! I need to remember to compile 
with other compilers than arm32 too =).

  Tomi

Re: [PATCH v15 05/19] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Tomi Valkeinen]

Hi,

On 04/10/2022 13:05, Sakari Ailus wrote:
> Hi Hans,
> 
> On Tue, Oct 04, 2022 at 10:43:55AM +0200, Hans Verkuil wrote:
>> Hi Sakari,
>>
>> On 10/4/22 00:01, Sakari Ailus wrote:
>>> Hi Hans,
>>>
>>> On Mon, Oct 03, 2022 at 04:26:32PM +0200, Hans Verkuil wrote:
>>>>> +#define V4L2_SUBDEV_ROUTE_FL_SOURCE		(1U << 1)
>>>>
>>>> Can we rename this to _FL_INTERNAL_SOURCE? Just 'SOURCE' is very confusing
>>>> IMHO. The name 'INTERNAL_SOURCE' makes it clear that it is generated internally,
>>>> and so does not come from an external sink-side endpoint.
>>>>
>>>> I also think that the documentation for this flag in patch 04/19 is very vague,
>>>> I'll comment on that patch as well.
>>>
>>> Having descriptive names is important but I think "SOURCE" as such is fine
>>> for the purpose. Adding "INTERNAL_" adds 9 characters to what is already a
>>> very long name, making the flag very clumsy to use in code. That's why I
>>> would prefer to keep it as-is.
>>>
>>
>> _FL_SOURCE is meaningless (at least to me): there are so many 'source' and 'sink'
>> references, that just plain 'SOURCE' doesn't help me understand what the flag
>> means. I did consider INT_SOURCE, but I thought 'INT' is too close to 'interrupt'.
>> I'm OK with that, though.
>>
>> Another alternative would be _FL_NO_SINK: that clearly conveys that 1) there is
>> no sink, and implies that 2) the source is internally generated.
>>
>> Or perhaps: _FL_SOURCE_ONLY?
> 
> This appears as the best compromise IMO. NO_SINK is shorter but conveying
> the meaning through negation is what I don't like too much.
> 

SOURCE_ONLY sounds fine to me.

  Tomi

Re: [PATCH v15 04/19] media: Documentation: Add GS_ROUTING documentation

[Tomi Valkeinen]

Hi,

On 03/10/2022 17:31, Hans Verkuil wrote:
> Hi Tomi,
> 
> On 10/3/22 14:18, Tomi Valkeinen wrote:
>> From: Jacopo Mondi <jacopo+renesas@jmondi.org>
>>
>> Add documentation for VIDIOC_SUBDEV_G/S_ROUTING ioctl and add
>> description of multiplexed media pads and internal routing to the
>> V4L2-subdev documentation section.
>>
>> Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
>> ---
>>   .../userspace-api/media/v4l/dev-subdev.rst    |   2 +
>>   .../userspace-api/media/v4l/user-func.rst     |   1 +
>>   .../media/v4l/vidioc-subdev-g-routing.rst     | 155 ++++++++++++++++++
>>   3 files changed, 158 insertions(+)
>>   create mode 100644 Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
>>
>> diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
>> index fd1de0a73a9f..a67c2749089a 100644
>> --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
>> +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
>> @@ -29,6 +29,8 @@ will feature a character device node on which ioctls can be called to
>>   
>>   -  negotiate image formats on individual pads
>>   
>> +-  inspect and modify internal data routing between pads of the same entity
>> +
>>   Sub-device character device nodes, conventionally named
>>   ``/dev/v4l-subdev*``, use major number 81.
>>   
>> diff --git a/Documentation/userspace-api/media/v4l/user-func.rst b/Documentation/userspace-api/media/v4l/user-func.rst
>> index 53e604bd7d60..228c1521f190 100644
>> --- a/Documentation/userspace-api/media/v4l/user-func.rst
>> +++ b/Documentation/userspace-api/media/v4l/user-func.rst
>> @@ -70,6 +70,7 @@ Function Reference
>>       vidioc-subdev-g-crop
>>       vidioc-subdev-g-fmt
>>       vidioc-subdev-g-frame-interval
>> +    vidioc-subdev-g-routing
>>       vidioc-subdev-g-selection
>>       vidioc-subdev-querycap
>>       vidioc-subscribe-event
>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
>> new file mode 100644
>> index 000000000000..6d8fc3b11352
>> --- /dev/null
>> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
>> @@ -0,0 +1,155 @@
>> +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
>> +.. c:namespace:: V4L
>> +
>> +.. _VIDIOC_SUBDEV_G_ROUTING:
>> +
>> +******************************************************
>> +ioctl VIDIOC_SUBDEV_G_ROUTING, VIDIOC_SUBDEV_S_ROUTING
>> +******************************************************
>> +
>> +Name
>> +====
>> +
>> +VIDIOC_SUBDEV_G_ROUTING - VIDIOC_SUBDEV_S_ROUTING - Get or set routing between streams of media pads in a media entity.
>> +
>> +
>> +Synopsis
>> +========
>> +
>> +.. c:function:: int ioctl( int fd, VIDIOC_SUBDEV_G_ROUTING, struct v4l2_subdev_routing *argp )
>> +    :name: VIDIOC_SUBDEV_G_ROUTING
>> +
>> +.. c:function:: int ioctl( int fd, VIDIOC_SUBDEV_S_ROUTING, struct v4l2_subdev_routing *argp )
>> +    :name: VIDIOC_SUBDEV_S_ROUTING
>> +
>> +
>> +Arguments
>> +=========
>> +
>> +``fd``
>> +    File descriptor returned by :ref:`open() <func-open>`.
>> +
>> +``argp``
>> +    Pointer to struct :c:type:`v4l2_subdev_routing`.
>> +
>> +
>> +Description
>> +===========
>> +
>> +These ioctls are used to get and set the routing in a media entity.
>> +The routing configuration determines the flows of data inside an entity.
>> +
>> +Drivers report their current routing tables using the
>> +``VIDIOC_SUBDEV_G_ROUTING`` ioctl and application may enable or disable routes
>> +with the ``VIDIOC_SUBDEV_S_ROUTING`` ioctl, by adding or removing routes and
>> +setting or clearing flags of the  ``flags`` field of a
>> +struct :c:type:`v4l2_subdev_route`.
>> +
>> +All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called. This
>> +means that the userspace mut reconfigure all streams after calling the ioctl
> 
> typo: mut -> must
> 
>> +with e.g. ``VIDIOC_SUBDEV_S_FMT``.
>> +
>> +A special case for routing are routes marked with
>> +``V4L2_SUBDEV_ROUTE_FL_SOURCE`` flag. These routes are used to describe
>> +source endpoints on sensors and the sink fields are unused.
> 
> This is very vague. As mentioned in my review for 05/19 I think this flag should
> be renamed to INTERNAL_SOURCE. Then the last sentence can change to:
> 
> "These routes are used to describe source endpoints where the stream is internally
> created (such as a sensor) and so the sink fields are unused."
> 
>> +
>> +When inspecting routes through ``VIDIOC_SUBDEV_G_ROUTING`` and the application
>> +provided ``num_routes`` is not big enough to contain all the available routes
>> +the subdevice exposes, drivers return the ENOSPC error code and adjust the
>> +value of the ``num_routes`` field. Application should then reserve enough memory
>> +for all the route entries and call ``VIDIOC_SUBDEV_G_ROUTING`` again.
>> +
>> +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
>> +
>> +.. c:type:: v4l2_subdev_routing
>> +
>> +.. flat-table:: struct v4l2_subdev_routing
>> +    :header-rows:  0
>> +    :stub-columns: 0
>> +    :widths:       1 1 2
>> +
>> +    * - __u32
>> +      - ``which``
>> +      - Format to modified, from enum
>> +        :ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
>> +    * - struct :c:type:`v4l2_subdev_route`
>> +      - ``routes[]``
>> +      - Array of struct :c:type:`v4l2_subdev_route` entries
>> +    * - __u32
>> +      - ``num_routes``
>> +      - Number of entries of the routes array
>> +    * - __u32
>> +      - ``reserved``\ [5]
>> +      - Reserved for future extensions. Applications and drivers must set
>> +	the array to zero.
>> +
>> +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
>> +
>> +.. c:type:: v4l2_subdev_route
>> +
>> +.. flat-table:: struct v4l2_subdev_route
>> +    :header-rows:  0
>> +    :stub-columns: 0
>> +    :widths:       1 1 2
>> +
>> +    * - __u32
>> +      - ``sink_pad``
>> +      - Sink pad number.
>> +    * - __u32
>> +      - ``sink_stream``
>> +      - Sink pad stream number.
>> +    * - __u32
>> +      - ``source_pad``
>> +      - Source pad number.
>> +    * - __u32
>> +      - ``source_stream``
>> +      - Source pad stream number.
>> +    * - __u32
>> +      - ``flags``
>> +      - Route enable/disable flags
>> +	:ref:`v4l2_subdev_routing_flags <v4l2-subdev-routing-flags>`.
>> +    * - __u32
>> +      - ``reserved``\ [5]
>> +      - Reserved for future extensions. Applications and drivers must set
>> +	the array to zero.
>> +
>> +.. tabularcolumns:: |p{6.6cm}|p{2.2cm}|p{8.7cm}|
>> +
>> +.. _v4l2-subdev-routing-flags:
>> +
>> +.. flat-table:: enum v4l2_subdev_routing_flags
>> +    :header-rows:  0
>> +    :stub-columns: 0
>> +    :widths:       3 1 4
>> +
>> +    * - V4L2_SUBDEV_ROUTE_FL_ACTIVE
>> +      - 0
>> +      - The route is enabled. Set by applications.
>> +    * - V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
>> +      - 1
>> +      - The route is immutable. Set by the driver.
>> +    * - V4L2_SUBDEV_ROUTE_FL_SOURCE
>> +      - 2
>> +      - The route is a source route, and the ``sink_pad`` and ``sink_stream``
>> +        fields are unused. Set by the driver.
> 
> Same issue as above, it's very vague.
> 
> "Used to describe a route source endpoint where the stream is internally
> created (such as a sensor) and so the sink fields are unused."

I think this and the above suggestion are good, I'll make the change.

  Tomi

Re: [PATCH 1/1] media: Documentation: Interaction between routes, formats and selections

[Tomi Valkeinen]

Hoi,

On 11/10/2022 13:47, Sakari Ailus wrote:
> Document how setting up routes interacts with formats and selections.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
> Moi,
> 
> This could be added on top of the set.
> 
> Comments are welcome.
> 
>   .../userspace-api/media/v4l/dev-subdev.rst    | 48 ++++++++++++++-----
>   1 file changed, 37 insertions(+), 11 deletions(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> index 5075b1828b32d..830235eb01598 100644
> --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> @@ -406,6 +406,8 @@ pixel array is not rectangular but cross-shaped or round. The maximum
>   size may also be smaller than the BOUNDS rectangle.
>   
>   
> +.. _format-propagation:
> +
>   Order of configuration and format propagation
>   ---------------------------------------------
>   
> @@ -507,12 +509,12 @@ source pads.
>   Streams, multiplexed media pads and internal routing
>   ----------------------------------------------------
>   
> -Commonly V4L2 subdevices support only separate video streams, that is, only a
> -single stream can pass through a media link and a media pad. Thus each pad
> -contains a format configuration for that single stream. In some cases a subdev
> -can do stream processing and split a stream into two or compose two streams
> -into one, but the inputs and outputs for the subdev are still a single stream
> -per pad.
> +Commonly V4L2 subdevices do not support multiple, unrelated video streams.

I wonder if we should use some other word than "commonly", as it seems 
pretty common nowadays that there is at least a embedded data. Should we 
instead say, e.g., "Simple V4L2 subdevices..."

> +Only a single stream can pass through a media link and a media

This could be a continuation of the previous sentence: "... video 
streams, and only a single stream..."

> +pad. Thus each pad contains a format and selection configuration for that single stream.
> +A subdev can do stream processing and split a stream into
> +two or compose two streams into one, but the inputs and outputs for the
> +subdev are still a single stream per pad.
>   
>   Some hardware, e.g. MIPI CSI-2, support multiplexed streams, that is, multiple
>   data streams are transmitted on the same bus, which is represented by a media
> @@ -539,15 +541,37 @@ streams from one end of the link to the other, and subdevices have routing
>   tables which describe how the incoming streams from sink pads are routed to the
>   source pads.
>   
> -A stream ID (often just "stream") is a media link-local identifier for a stream.
> +A stream ID is a media link-local identifier for a stream.
>   In other words, a particular stream ID must exist on both sides of a media
>   link, but another stream ID can be used for the same stream at the other side
> -of the subdevice.
> +of the subdevice. The same stream ID is used to refer to the stream on
> +both pads of the link on all ioctls operating on pads.
>   
>   A stream at a specific point in the media pipeline is identified with the
> -subdev and a (pad, stream) pair. For subdevices that do not support
> +subdev and a pad--stream pair. For subdevices that do not support

Is there a double-dash on purpose?

Btw, above you removed the "(often just "stream")", but here you use 
"stream" instead of "stream ID".

>   multiplexed streams the 'stream' is always 0.
>   
> +Interaction between routes, formats and selections
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +The addition of routes to the V4L2 sub-device interface moves the
> +sub-device formats and selections from pads to pad--stream pairs. Instead

I guess it is, as you use it here also...

> +of the sub-device wide merging of streams from all source pads towards all
> +sink pads, this takes place separately for each route. The stream ID on
> +the sink pad for each configured route is used to configure format and
> +selections on the sink pad. Similarly, the stream ID on the source pad of
> +each configured route is used to configure format and selections on the
> +source pad.

Hmm, "stream ID is used to configure format" sounds odd to me. The ioctl 
is used to configure the format, but rather than using only pad ID to 
identify the configuration target, you now use pad ID - stream ID pair.

And is that just extra duplication above to talk separately about sink 
and source sides? They don't matter here, the point is just the 
pad-stream vs only pad.

> +
> +Any number of routes from streams on sink pads towards streams on source

This also sounds a bit odd to me: "a route from a stream on a sink pad". 
I think we're missing a word here which means the specific point of a 
stream in the pipeline.

I have suggested "stream endpoint", but it's not really an "end" point. 
"Stream waypoint"? So you would configure a format to a subdev's stream 
waypoint. And "Any number of routes from stream waypoints on sink pads 
towards stream waypoints on source pads".

So a "stream waypoint" would be a triplet of subdev-padID-streamID, or 
just padID-streamID if the subdev is obvious.

Even if we don't find a perfect or even a very good term for this, I 
think we should just use something. Using just "stream" makes things 
quite confusing, in my opinion.

> +pads is allowed, to the extent supported by drivers. For every stream on a
> +sink pad, however, only a single route is allowed.

Hmm, why is that?

> +
> +Pad--stream pairs are not static but are replaced by calls to the
> +:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. This also
> +applies to stream format and selection configurations which that are

Should that "applies to" be "affects the"?

> +reverted to driver defaults.
> +
>   Configuring streams
>   ^^^^^^^^^^^^^^^^^^^
>   
> @@ -565,8 +589,10 @@ Controller API <media_controller>`
>   setting the routing table will reset all the stream configurations in a media
>   entity.
>   
> -3) Configure streams. Each route endpoint must be configured

Oh, here I seem to use "route endpoint".

> -with :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>`.
> +3) Configure formats and selections on routes. Each route is configured

I'm not sure if we "configure formats on routes". Earlier I think we've 
talked about configuring the streams (stream waypoints).

> +separately as documented plain subdevices in :ref:`<format-propagation>`.

Is there something missing from above? Or drop "plain subdevices"?

> +The stream ID is set to the same stream ID used on sink and source pads of
> +the :ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl.

Yes. But I think this could be said more clearly if we have a good word 
for the padID-streamID pair.

  Tomi

Re: [PATCH v15 08/19] media: subdev: Add for_each_active_route() macro

[Sakari Ailus]

On Wed, Oct 12, 2022 at 09:15:31AM +0300, Tomi Valkeinen wrote:
> On 09/10/2022 08:38, Dafna Hirschfeld wrote:
> > On 03.10.2022 15:18, Tomi Valkeinen wrote:
> > > From: Jacopo Mondi <jacopo+renesas@jmondi.org>
> > > 
> > > Add a for_each_active_route() macro to replace the repeated pattern
> > > of iterating on the active routes of a routing table.
> > > 
> > > Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
> > > Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > > ---
> > > .clang-format                         |  1 +
> > > drivers/media/v4l2-core/v4l2-subdev.c | 20 ++++++++++++++++++++
> > > include/media/v4l2-subdev.h           | 13 +++++++++++++
> > > 3 files changed, 34 insertions(+)
> > > 
> > > diff --git a/.clang-format b/.clang-format
> > > index 1247d54f9e49..31f39ae78f7b 100644
> > > --- a/.clang-format
> > > +++ b/.clang-format
> > > @@ -190,6 +190,7 @@ ForEachMacros:
> > >   - 'for_each_active_dev_scope'
> > >   - 'for_each_active_drhd_unit'
> > >   - 'for_each_active_iommu'
> > > +  - 'for_each_active_route'
> > >   - 'for_each_aggr_pgid'
> > >   - 'for_each_available_child_of_node'
> > >   - 'for_each_bench'
> > > diff --git a/drivers/media/v4l2-core/v4l2-subdev.c
> > > b/drivers/media/v4l2-core/v4l2-subdev.c
> > > index 3ae4f39a50e4..1049c07d2e49 100644
> > > --- a/drivers/media/v4l2-core/v4l2-subdev.c
> > > +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> > > @@ -1212,6 +1212,26 @@ int v4l2_subdev_set_routing(struct
> > > v4l2_subdev *sd,
> > > }
> > > EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing);
> > > 
> > > +struct v4l2_subdev_route *
> > > +__v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting
> > > *routing,
> > > +                struct v4l2_subdev_route *route)
> > > +{
> > > +    if (route)
> > > +        ++route;
> > > +    else
> > > +        route = &routing->routes[0];
> > > +
> > > +    for (; route < routing->routes + routing->num_routes; ++route) {
> > > +        if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
> > > +            continue;
> > > +
> > > +        return route;
> > > +    }
> > > +
> > > +    return NULL;
> > > +}
> > > +EXPORT_SYMBOL_GPL(__v4l2_subdev_next_active_route);
> > > +
> > > #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
> > > 
> > > #endif /* CONFIG_MEDIA_CONTROLLER */
> > > diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> > > index 7962e6572bda..89e58208e330 100644
> > > --- a/include/media/v4l2-subdev.h
> > > +++ b/include/media/v4l2-subdev.h
> > > @@ -1435,6 +1435,19 @@ int v4l2_subdev_set_routing(struct
> > > v4l2_subdev *sd,
> > >                 struct v4l2_subdev_state *state,
> > >                 const struct v4l2_subdev_krouting *routing);
> > > 
> > > +struct v4l2_subdev_route *
> > > +__v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting
> > > *routing,
> > > +                struct v4l2_subdev_route *route);
> > > +
> > > +/**
> > > + * for_each_active_route - iterate on all active routes of a
> > > routing table
> > > + * @routing: The routing table
> > > + * @route: The route iterator
> > > + */
> > > +#define for_each_active_route(routing, route) \
> > > +    for ((route) = NULL;                  \
> > > +         ((route) = __v4l2_subdev_next_active_route((routing),
> > > (route)));)
> > Hi, shouldn't it be something like:
> >      for ((route) = NULL; (route) ; (route) =
> > __v4l2_subdev_next_active_route((routing), (route)))
> 
> What you suggest would never do anything: you initialize route to NULL, and
> then check if the route is !NULL.
> 
> I also find the current version a bit "interesting", but afaics, it works
> correctly.

It would look better IMO written differently, yes. But this likely results
in fewer instructions so I'm fine with it. ;)

-- 
Sakari Ailus

Re: [PATCH v15 11/19] media: subdev: use streams in v4l2_subdev_link_validate()

[Sakari Ailus]

Moi,

On Mon, Oct 03, 2022 at 03:18:44PM +0300, Tomi Valkeinen wrote:
> Update v4l2_subdev_link_validate() to use routing and streams for
> validation.
> 
> Instead of just looking at the format on the pad on both ends of the
> link, the routing tables are used to collect all the streams going from
> the source to the sink over the link, and the streams' formats on both
> ends of the link are verified.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 182 +++++++++++++++++++++++---
>  1 file changed, 162 insertions(+), 20 deletions(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index be778e619694..1cea6ec750c0 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -1014,7 +1014,7 @@ int v4l2_subdev_link_validate_default(struct v4l2_subdev *sd,
>  EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate_default);
>  
>  static int
> -v4l2_subdev_link_validate_get_format(struct media_pad *pad,
> +v4l2_subdev_link_validate_get_format(struct media_pad *pad, u32 stream,
>  				     struct v4l2_subdev_format *fmt)
>  {
>  	if (is_media_entity_v4l2_subdev(pad->entity)) {
> @@ -1023,7 +1023,11 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
>  
>  		fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
>  		fmt->pad = pad->index;
> -		return v4l2_subdev_call_state_active(sd, pad, get_fmt, fmt);
> +		fmt->stream = stream;
> +
> +		return v4l2_subdev_call(sd, pad, get_fmt,
> +					v4l2_subdev_get_locked_active_state(sd),
> +					fmt);
>  	}
>  
>  	WARN(pad->entity->function != MEDIA_ENT_F_IO_V4L,
> @@ -1033,31 +1037,169 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
>  	return -EINVAL;
>  }
>  
> +#if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
> +
> +static void __v4l2_link_validate_get_streams(struct media_pad *pad,
> +					     u64 *streams_mask)
> +{
> +	struct v4l2_subdev_route *route;
> +	struct v4l2_subdev_state *state;
> +	struct v4l2_subdev *subdev;
> +
> +	subdev = media_entity_to_v4l2_subdev(pad->entity);
> +
> +	*streams_mask = 0;
> +
> +	state = v4l2_subdev_get_locked_active_state(subdev);
> +	if (WARN_ON(!state))
> +		return;
> +
> +	for_each_active_route(&state->routing, route) {
> +		u32 route_pad;
> +		u32 route_stream;
> +
> +		if (pad->flags & MEDIA_PAD_FL_SOURCE) {
> +			route_pad = route->source_pad;
> +			route_stream = route->source_stream;
> +		} else {
> +			route_pad = route->sink_pad;
> +			route_stream = route->sink_stream;
> +		}
> +
> +		if (route_pad != pad->index)
> +			continue;
> +
> +		*streams_mask |= BIT_ULL(route_stream);
> +	}
> +}
> +
> +#endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
> +
> +static void v4l2_link_validate_get_streams(struct media_pad *pad,
> +					   u64 *streams_mask)
> +{
> +	struct v4l2_subdev *subdev = media_entity_to_v4l2_subdev(pad->entity);
> +
> +	if (!(subdev->flags & V4L2_SUBDEV_FL_STREAMS)) {
> +		/* Non-streams subdevs have an implicit stream 0 */
> +		*streams_mask = BIT_ULL(0);
> +		return;
> +	}
> +
> +#if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
> +	__v4l2_link_validate_get_streams(pad, streams_mask);
> +#else
> +	/* This shouldn't happen */
> +	*streams_mask = 0;
> +#endif
> +}
> +
> +static int v4l2_subdev_link_validate_locked(struct media_link *link)
> +{
> +	struct v4l2_subdev *sink_subdev =
> +		media_entity_to_v4l2_subdev(link->sink->entity);
> +	struct device *dev = sink_subdev->entity.graph_obj.mdev->dev;
> +	u64 source_streams_mask;
> +	u64 sink_streams_mask;
> +	u64 dangling_sink_streams;
> +	u32 stream;
> +	int ret;
> +
> +	dev_dbg(dev, "validating link \"%s\":%u -> \"%s\":%u\n",
> +		link->source->entity->name, link->source->index,
> +		link->sink->entity->name, link->sink->index);
> +
> +	v4l2_link_validate_get_streams(link->source, &source_streams_mask);
> +	v4l2_link_validate_get_streams(link->sink, &sink_streams_mask);
> +
> +	/*
> +	 * It is ok to have more source streams than sink streams as extra
> +	 * source streams can just be ignored by the receiver, but having extra
> +	 * sink streams is an error as streams must have a source.
> +	 */
> +	dangling_sink_streams = (source_streams_mask ^ sink_streams_mask) &
> +				sink_streams_mask;
> +	if (dangling_sink_streams) {
> +		dev_err(dev, "Dangling sink streams: mask %#llx\n",
> +			dangling_sink_streams);
> +		return -EINVAL;
> +	}
> +
> +	/* Validate source and sink stream formats */
> +
> +	for_each_set_bit(stream, (void *)&sink_streams_mask, 64) {

Does this work as expected? The second argument is expected to be unsigned
long (or an array of two of them) whereas you have a u64.

> +		struct v4l2_subdev_format sink_fmt, source_fmt;
> +
> +		dev_dbg(dev, "validating stream \"%s\":%u:%u -> \"%s\":%u:%u\n",
> +			link->source->entity->name, link->source->index, stream,
> +			link->sink->entity->name, link->sink->index, stream);
> +
> +		ret = v4l2_subdev_link_validate_get_format(link->source, stream,
> +							   &source_fmt);
> +		if (ret < 0) {
> +			dev_dbg(dev,
> +				"Failed to get format for \"%s\":%u:%u (but that's ok)\n",
> +				link->source->entity->name, link->source->index,
> +				stream);
> +			continue;
> +		}
> +
> +		ret = v4l2_subdev_link_validate_get_format(link->sink, stream,
> +							   &sink_fmt);
> +		if (ret < 0) {
> +			dev_dbg(dev,
> +				"Failed to get format for \"%s\":%u:%u (but that's ok)\n",
> +				link->sink->entity->name, link->sink->index,
> +				stream);
> +			continue;
> +		}
> +
> +		/* TODO: add stream number to link_validate() */
> +		ret = v4l2_subdev_call(sink_subdev, pad, link_validate, link,
> +				       &source_fmt, &sink_fmt);
> +		if (!ret)
> +			continue;
> +
> +		if (ret != -ENOIOCTLCMD)
> +			return ret;
> +
> +		ret = v4l2_subdev_link_validate_default(sink_subdev, link,
> +							&source_fmt, &sink_fmt);
> +
> +		if (ret)
> +			return ret;
> +	}
> +
> +	return 0;
> +}
> +
>  int v4l2_subdev_link_validate(struct media_link *link)
>  {
> -	struct v4l2_subdev *sink;
> -	struct v4l2_subdev_format sink_fmt, source_fmt;
> -	int rval;
> +	struct v4l2_subdev *source_sd, *sink_sd;
> +	struct v4l2_subdev_state *source_state, *sink_state;
> +	int ret;
>  
> -	rval = v4l2_subdev_link_validate_get_format(
> -		link->source, &source_fmt);
> -	if (rval < 0)
> -		return 0;
> +	sink_sd = media_entity_to_v4l2_subdev(link->sink->entity);
> +	source_sd = media_entity_to_v4l2_subdev(link->source->entity);
>  
> -	rval = v4l2_subdev_link_validate_get_format(
> -		link->sink, &sink_fmt);
> -	if (rval < 0)
> -		return 0;
> +	sink_state = v4l2_subdev_get_unlocked_active_state(sink_sd);
> +	source_state = v4l2_subdev_get_unlocked_active_state(source_sd);
>  
> -	sink = media_entity_to_v4l2_subdev(link->sink->entity);
> +	if (sink_state)
> +		v4l2_subdev_lock_state(sink_state);
>  
> -	rval = v4l2_subdev_call(sink, pad, link_validate, link,
> -				&source_fmt, &sink_fmt);
> -	if (rval != -ENOIOCTLCMD)
> -		return rval;
> +	if (source_state)
> +		v4l2_subdev_lock_state(source_state);
>  
> -	return v4l2_subdev_link_validate_default(
> -		sink, link, &source_fmt, &sink_fmt);
> +	ret = v4l2_subdev_link_validate_locked(link);
> +
> +	if (sink_state)
> +		v4l2_subdev_unlock_state(sink_state);
> +
> +	if (source_state)
> +		v4l2_subdev_unlock_state(source_state);
> +
> +	return ret;
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
>  

-- 
Terveisin,

Sakari Ailus

Re: [PATCH v15 11/19] media: subdev: use streams in v4l2_subdev_link_validate()

[Tomi Valkeinen]

On 14/10/2022 13:54, Sakari Ailus wrote:
> Moi,
> 
> On Mon, Oct 03, 2022 at 03:18:44PM +0300, Tomi Valkeinen wrote:
>> Update v4l2_subdev_link_validate() to use routing and streams for
>> validation.
>>
>> Instead of just looking at the format on the pad on both ends of the
>> link, the routing tables are used to collect all the streams going from
>> the source to the sink over the link, and the streams' formats on both
>> ends of the link are verified.
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>>   drivers/media/v4l2-core/v4l2-subdev.c | 182 +++++++++++++++++++++++---
>>   1 file changed, 162 insertions(+), 20 deletions(-)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>> index be778e619694..1cea6ec750c0 100644
>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>> @@ -1014,7 +1014,7 @@ int v4l2_subdev_link_validate_default(struct v4l2_subdev *sd,
>>   EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate_default);
>>   
>>   static int
>> -v4l2_subdev_link_validate_get_format(struct media_pad *pad,
>> +v4l2_subdev_link_validate_get_format(struct media_pad *pad, u32 stream,
>>   				     struct v4l2_subdev_format *fmt)
>>   {
>>   	if (is_media_entity_v4l2_subdev(pad->entity)) {
>> @@ -1023,7 +1023,11 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
>>   
>>   		fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
>>   		fmt->pad = pad->index;
>> -		return v4l2_subdev_call_state_active(sd, pad, get_fmt, fmt);
>> +		fmt->stream = stream;
>> +
>> +		return v4l2_subdev_call(sd, pad, get_fmt,
>> +					v4l2_subdev_get_locked_active_state(sd),
>> +					fmt);
>>   	}
>>   
>>   	WARN(pad->entity->function != MEDIA_ENT_F_IO_V4L,
>> @@ -1033,31 +1037,169 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
>>   	return -EINVAL;
>>   }
>>   
>> +#if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
>> +
>> +static void __v4l2_link_validate_get_streams(struct media_pad *pad,
>> +					     u64 *streams_mask)
>> +{
>> +	struct v4l2_subdev_route *route;
>> +	struct v4l2_subdev_state *state;
>> +	struct v4l2_subdev *subdev;
>> +
>> +	subdev = media_entity_to_v4l2_subdev(pad->entity);
>> +
>> +	*streams_mask = 0;
>> +
>> +	state = v4l2_subdev_get_locked_active_state(subdev);
>> +	if (WARN_ON(!state))
>> +		return;
>> +
>> +	for_each_active_route(&state->routing, route) {
>> +		u32 route_pad;
>> +		u32 route_stream;
>> +
>> +		if (pad->flags & MEDIA_PAD_FL_SOURCE) {
>> +			route_pad = route->source_pad;
>> +			route_stream = route->source_stream;
>> +		} else {
>> +			route_pad = route->sink_pad;
>> +			route_stream = route->sink_stream;
>> +		}
>> +
>> +		if (route_pad != pad->index)
>> +			continue;
>> +
>> +		*streams_mask |= BIT_ULL(route_stream);
>> +	}
>> +}
>> +
>> +#endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
>> +
>> +static void v4l2_link_validate_get_streams(struct media_pad *pad,
>> +					   u64 *streams_mask)
>> +{
>> +	struct v4l2_subdev *subdev = media_entity_to_v4l2_subdev(pad->entity);
>> +
>> +	if (!(subdev->flags & V4L2_SUBDEV_FL_STREAMS)) {
>> +		/* Non-streams subdevs have an implicit stream 0 */
>> +		*streams_mask = BIT_ULL(0);
>> +		return;
>> +	}
>> +
>> +#if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
>> +	__v4l2_link_validate_get_streams(pad, streams_mask);
>> +#else
>> +	/* This shouldn't happen */
>> +	*streams_mask = 0;
>> +#endif
>> +}
>> +
>> +static int v4l2_subdev_link_validate_locked(struct media_link *link)
>> +{
>> +	struct v4l2_subdev *sink_subdev =
>> +		media_entity_to_v4l2_subdev(link->sink->entity);
>> +	struct device *dev = sink_subdev->entity.graph_obj.mdev->dev;
>> +	u64 source_streams_mask;
>> +	u64 sink_streams_mask;
>> +	u64 dangling_sink_streams;
>> +	u32 stream;
>> +	int ret;
>> +
>> +	dev_dbg(dev, "validating link \"%s\":%u -> \"%s\":%u\n",
>> +		link->source->entity->name, link->source->index,
>> +		link->sink->entity->name, link->sink->index);
>> +
>> +	v4l2_link_validate_get_streams(link->source, &source_streams_mask);
>> +	v4l2_link_validate_get_streams(link->sink, &sink_streams_mask);
>> +
>> +	/*
>> +	 * It is ok to have more source streams than sink streams as extra
>> +	 * source streams can just be ignored by the receiver, but having extra
>> +	 * sink streams is an error as streams must have a source.
>> +	 */
>> +	dangling_sink_streams = (source_streams_mask ^ sink_streams_mask) &
>> +				sink_streams_mask;
>> +	if (dangling_sink_streams) {
>> +		dev_err(dev, "Dangling sink streams: mask %#llx\n",
>> +			dangling_sink_streams);
>> +		return -EINVAL;
>> +	}
>> +
>> +	/* Validate source and sink stream formats */
>> +
>> +	for_each_set_bit(stream, (void *)&sink_streams_mask, 64) {
> 
> Does this work as expected? The second argument is expected to be unsigned
> long (or an array of two of them) whereas you have a u64.

Where do you see that? I thought find_next_bit (used by 
for_each_set_bit) is given a start address and arbitrarily large 
bit-size number.

  Tomi

Re: [PATCH v15 11/19] media: subdev: use streams in v4l2_subdev_link_validate()

[Sakari Ailus]

Moi,

On Fri, Oct 14, 2022 at 02:10:35PM +0300, Tomi Valkeinen wrote:
> On 14/10/2022 13:54, Sakari Ailus wrote:
> > Moi,
> > 
> > On Mon, Oct 03, 2022 at 03:18:44PM +0300, Tomi Valkeinen wrote:
> > > Update v4l2_subdev_link_validate() to use routing and streams for
> > > validation.
> > > 
> > > Instead of just looking at the format on the pad on both ends of the
> > > link, the routing tables are used to collect all the streams going from
> > > the source to the sink over the link, and the streams' formats on both
> > > ends of the link are verified.
> > > 
> > > Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > > ---
> > >   drivers/media/v4l2-core/v4l2-subdev.c | 182 +++++++++++++++++++++++---
> > >   1 file changed, 162 insertions(+), 20 deletions(-)
> > > 
> > > diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> > > index be778e619694..1cea6ec750c0 100644
> > > --- a/drivers/media/v4l2-core/v4l2-subdev.c
> > > +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> > > @@ -1014,7 +1014,7 @@ int v4l2_subdev_link_validate_default(struct v4l2_subdev *sd,
> > >   EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate_default);
> > >   static int
> > > -v4l2_subdev_link_validate_get_format(struct media_pad *pad,
> > > +v4l2_subdev_link_validate_get_format(struct media_pad *pad, u32 stream,
> > >   				     struct v4l2_subdev_format *fmt)
> > >   {
> > >   	if (is_media_entity_v4l2_subdev(pad->entity)) {
> > > @@ -1023,7 +1023,11 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
> > >   		fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
> > >   		fmt->pad = pad->index;
> > > -		return v4l2_subdev_call_state_active(sd, pad, get_fmt, fmt);
> > > +		fmt->stream = stream;
> > > +
> > > +		return v4l2_subdev_call(sd, pad, get_fmt,
> > > +					v4l2_subdev_get_locked_active_state(sd),
> > > +					fmt);
> > >   	}
> > >   	WARN(pad->entity->function != MEDIA_ENT_F_IO_V4L,
> > > @@ -1033,31 +1037,169 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
> > >   	return -EINVAL;
> > >   }
> > > +#if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
> > > +
> > > +static void __v4l2_link_validate_get_streams(struct media_pad *pad,
> > > +					     u64 *streams_mask)
> > > +{
> > > +	struct v4l2_subdev_route *route;
> > > +	struct v4l2_subdev_state *state;
> > > +	struct v4l2_subdev *subdev;
> > > +
> > > +	subdev = media_entity_to_v4l2_subdev(pad->entity);
> > > +
> > > +	*streams_mask = 0;
> > > +
> > > +	state = v4l2_subdev_get_locked_active_state(subdev);
> > > +	if (WARN_ON(!state))
> > > +		return;
> > > +
> > > +	for_each_active_route(&state->routing, route) {
> > > +		u32 route_pad;
> > > +		u32 route_stream;
> > > +
> > > +		if (pad->flags & MEDIA_PAD_FL_SOURCE) {
> > > +			route_pad = route->source_pad;
> > > +			route_stream = route->source_stream;
> > > +		} else {
> > > +			route_pad = route->sink_pad;
> > > +			route_stream = route->sink_stream;
> > > +		}
> > > +
> > > +		if (route_pad != pad->index)
> > > +			continue;
> > > +
> > > +		*streams_mask |= BIT_ULL(route_stream);
> > > +	}
> > > +}
> > > +
> > > +#endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
> > > +
> > > +static void v4l2_link_validate_get_streams(struct media_pad *pad,
> > > +					   u64 *streams_mask)
> > > +{
> > > +	struct v4l2_subdev *subdev = media_entity_to_v4l2_subdev(pad->entity);
> > > +
> > > +	if (!(subdev->flags & V4L2_SUBDEV_FL_STREAMS)) {
> > > +		/* Non-streams subdevs have an implicit stream 0 */
> > > +		*streams_mask = BIT_ULL(0);
> > > +		return;
> > > +	}
> > > +
> > > +#if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
> > > +	__v4l2_link_validate_get_streams(pad, streams_mask);
> > > +#else
> > > +	/* This shouldn't happen */
> > > +	*streams_mask = 0;
> > > +#endif
> > > +}
> > > +
> > > +static int v4l2_subdev_link_validate_locked(struct media_link *link)
> > > +{
> > > +	struct v4l2_subdev *sink_subdev =
> > > +		media_entity_to_v4l2_subdev(link->sink->entity);
> > > +	struct device *dev = sink_subdev->entity.graph_obj.mdev->dev;
> > > +	u64 source_streams_mask;
> > > +	u64 sink_streams_mask;
> > > +	u64 dangling_sink_streams;
> > > +	u32 stream;
> > > +	int ret;
> > > +
> > > +	dev_dbg(dev, "validating link \"%s\":%u -> \"%s\":%u\n",
> > > +		link->source->entity->name, link->source->index,
> > > +		link->sink->entity->name, link->sink->index);
> > > +
> > > +	v4l2_link_validate_get_streams(link->source, &source_streams_mask);
> > > +	v4l2_link_validate_get_streams(link->sink, &sink_streams_mask);
> > > +
> > > +	/*
> > > +	 * It is ok to have more source streams than sink streams as extra
> > > +	 * source streams can just be ignored by the receiver, but having extra
> > > +	 * sink streams is an error as streams must have a source.
> > > +	 */
> > > +	dangling_sink_streams = (source_streams_mask ^ sink_streams_mask) &
> > > +				sink_streams_mask;
> > > +	if (dangling_sink_streams) {
> > > +		dev_err(dev, "Dangling sink streams: mask %#llx\n",
> > > +			dangling_sink_streams);
> > > +		return -EINVAL;
> > > +	}
> > > +
> > > +	/* Validate source and sink stream formats */
> > > +
> > > +	for_each_set_bit(stream, (void *)&sink_streams_mask, 64) {
> > 
> > Does this work as expected? The second argument is expected to be unsigned
> > long (or an array of two of them) whereas you have a u64.
> 
> Where do you see that? I thought find_next_bit (used by for_each_set_bit) is
> given a start address and arbitrarily large bit-size number.

sink_streams_mask is a u64 while for_each_set_bit() expects an array of
unsigned longs. Endianness matters.

-- 
Terveisin,

Sakari Ailus

Re: [PATCH v15 11/19] media: subdev: use streams in v4l2_subdev_link_validate()

[Tomi Valkeinen]

On 17/10/2022 01:37, Sakari Ailus wrote:
> Moi,
> 
> On Fri, Oct 14, 2022 at 02:10:35PM +0300, Tomi Valkeinen wrote:
>> On 14/10/2022 13:54, Sakari Ailus wrote:
>>> Moi,
>>>
>>> On Mon, Oct 03, 2022 at 03:18:44PM +0300, Tomi Valkeinen wrote:
>>>> Update v4l2_subdev_link_validate() to use routing and streams for
>>>> validation.
>>>>
>>>> Instead of just looking at the format on the pad on both ends of the
>>>> link, the routing tables are used to collect all the streams going from
>>>> the source to the sink over the link, and the streams' formats on both
>>>> ends of the link are verified.
>>>>
>>>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>>>> ---
>>>>    drivers/media/v4l2-core/v4l2-subdev.c | 182 +++++++++++++++++++++++---
>>>>    1 file changed, 162 insertions(+), 20 deletions(-)
>>>>
>>>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>>>> index be778e619694..1cea6ec750c0 100644
>>>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>>>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>>>> @@ -1014,7 +1014,7 @@ int v4l2_subdev_link_validate_default(struct v4l2_subdev *sd,
>>>>    EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate_default);
>>>>    static int
>>>> -v4l2_subdev_link_validate_get_format(struct media_pad *pad,
>>>> +v4l2_subdev_link_validate_get_format(struct media_pad *pad, u32 stream,
>>>>    				     struct v4l2_subdev_format *fmt)
>>>>    {
>>>>    	if (is_media_entity_v4l2_subdev(pad->entity)) {
>>>> @@ -1023,7 +1023,11 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
>>>>    		fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
>>>>    		fmt->pad = pad->index;
>>>> -		return v4l2_subdev_call_state_active(sd, pad, get_fmt, fmt);
>>>> +		fmt->stream = stream;
>>>> +
>>>> +		return v4l2_subdev_call(sd, pad, get_fmt,
>>>> +					v4l2_subdev_get_locked_active_state(sd),
>>>> +					fmt);
>>>>    	}
>>>>    	WARN(pad->entity->function != MEDIA_ENT_F_IO_V4L,
>>>> @@ -1033,31 +1037,169 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
>>>>    	return -EINVAL;
>>>>    }
>>>> +#if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
>>>> +
>>>> +static void __v4l2_link_validate_get_streams(struct media_pad *pad,
>>>> +					     u64 *streams_mask)
>>>> +{
>>>> +	struct v4l2_subdev_route *route;
>>>> +	struct v4l2_subdev_state *state;
>>>> +	struct v4l2_subdev *subdev;
>>>> +
>>>> +	subdev = media_entity_to_v4l2_subdev(pad->entity);
>>>> +
>>>> +	*streams_mask = 0;
>>>> +
>>>> +	state = v4l2_subdev_get_locked_active_state(subdev);
>>>> +	if (WARN_ON(!state))
>>>> +		return;
>>>> +
>>>> +	for_each_active_route(&state->routing, route) {
>>>> +		u32 route_pad;
>>>> +		u32 route_stream;
>>>> +
>>>> +		if (pad->flags & MEDIA_PAD_FL_SOURCE) {
>>>> +			route_pad = route->source_pad;
>>>> +			route_stream = route->source_stream;
>>>> +		} else {
>>>> +			route_pad = route->sink_pad;
>>>> +			route_stream = route->sink_stream;
>>>> +		}
>>>> +
>>>> +		if (route_pad != pad->index)
>>>> +			continue;
>>>> +
>>>> +		*streams_mask |= BIT_ULL(route_stream);
>>>> +	}
>>>> +}
>>>> +
>>>> +#endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
>>>> +
>>>> +static void v4l2_link_validate_get_streams(struct media_pad *pad,
>>>> +					   u64 *streams_mask)
>>>> +{
>>>> +	struct v4l2_subdev *subdev = media_entity_to_v4l2_subdev(pad->entity);
>>>> +
>>>> +	if (!(subdev->flags & V4L2_SUBDEV_FL_STREAMS)) {
>>>> +		/* Non-streams subdevs have an implicit stream 0 */
>>>> +		*streams_mask = BIT_ULL(0);
>>>> +		return;
>>>> +	}
>>>> +
>>>> +#if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
>>>> +	__v4l2_link_validate_get_streams(pad, streams_mask);
>>>> +#else
>>>> +	/* This shouldn't happen */
>>>> +	*streams_mask = 0;
>>>> +#endif
>>>> +}
>>>> +
>>>> +static int v4l2_subdev_link_validate_locked(struct media_link *link)
>>>> +{
>>>> +	struct v4l2_subdev *sink_subdev =
>>>> +		media_entity_to_v4l2_subdev(link->sink->entity);
>>>> +	struct device *dev = sink_subdev->entity.graph_obj.mdev->dev;
>>>> +	u64 source_streams_mask;
>>>> +	u64 sink_streams_mask;
>>>> +	u64 dangling_sink_streams;
>>>> +	u32 stream;
>>>> +	int ret;
>>>> +
>>>> +	dev_dbg(dev, "validating link \"%s\":%u -> \"%s\":%u\n",
>>>> +		link->source->entity->name, link->source->index,
>>>> +		link->sink->entity->name, link->sink->index);
>>>> +
>>>> +	v4l2_link_validate_get_streams(link->source, &source_streams_mask);
>>>> +	v4l2_link_validate_get_streams(link->sink, &sink_streams_mask);
>>>> +
>>>> +	/*
>>>> +	 * It is ok to have more source streams than sink streams as extra
>>>> +	 * source streams can just be ignored by the receiver, but having extra
>>>> +	 * sink streams is an error as streams must have a source.
>>>> +	 */
>>>> +	dangling_sink_streams = (source_streams_mask ^ sink_streams_mask) &
>>>> +				sink_streams_mask;
>>>> +	if (dangling_sink_streams) {
>>>> +		dev_err(dev, "Dangling sink streams: mask %#llx\n",
>>>> +			dangling_sink_streams);
>>>> +		return -EINVAL;
>>>> +	}
>>>> +
>>>> +	/* Validate source and sink stream formats */
>>>> +
>>>> +	for_each_set_bit(stream, (void *)&sink_streams_mask, 64) {
>>>
>>> Does this work as expected? The second argument is expected to be unsigned
>>> long (or an array of two of them) whereas you have a u64.
>>
>> Where do you see that? I thought find_next_bit (used by for_each_set_bit) is
>> given a start address and arbitrarily large bit-size number.
> 
> sink_streams_mask is a u64 while for_each_set_bit() expects an array of
> unsigned longs. Endianness matters.

Yes, you're right. I can't find any ready helper for this, so I'll just 
do it manually:

for (stream = 0; stream < sizeof(sink_streams_mask) * 8; ++stream) {
	if (!(sink_streams_mask & BIT_ULL(stream)))
         	continue;
	...
}

  Tomi

Re: [PATCH 1/1] media: Documentation: Interaction between routes, formats and selections

[Sakari Ailus]

Moi,

On Wed, Oct 12, 2022 at 01:30:53PM +0300, Tomi Valkeinen wrote:
> Hoi,
> 
> On 11/10/2022 13:47, Sakari Ailus wrote:
> > Document how setting up routes interacts with formats and selections.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> > Moi,
> > 
> > This could be added on top of the set.
> > 
> > Comments are welcome.
> > 
> >   .../userspace-api/media/v4l/dev-subdev.rst    | 48 ++++++++++++++-----
> >   1 file changed, 37 insertions(+), 11 deletions(-)
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > index 5075b1828b32d..830235eb01598 100644
> > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > @@ -406,6 +406,8 @@ pixel array is not rectangular but cross-shaped or round. The maximum
> >   size may also be smaller than the BOUNDS rectangle.
> > +.. _format-propagation:
> > +
> >   Order of configuration and format propagation
> >   ---------------------------------------------
> > @@ -507,12 +509,12 @@ source pads.
> >   Streams, multiplexed media pads and internal routing
> >   ----------------------------------------------------
> > -Commonly V4L2 subdevices support only separate video streams, that is, only a
> > -single stream can pass through a media link and a media pad. Thus each pad
> > -contains a format configuration for that single stream. In some cases a subdev
> > -can do stream processing and split a stream into two or compose two streams
> > -into one, but the inputs and outputs for the subdev are still a single stream
> > -per pad.
> > +Commonly V4L2 subdevices do not support multiple, unrelated video streams.
> 
> I wonder if we should use some other word than "commonly", as it seems
> pretty common nowadays that there is at least a embedded data. Should we
> instead say, e.g., "Simple V4L2 subdevices..."
> 
> > +Only a single stream can pass through a media link and a media
> 
> This could be a continuation of the previous sentence: "... video streams,
> and only a single stream..."

Sounds good (for both).

> 
> > +pad. Thus each pad contains a format and selection configuration for that single stream.
> > +A subdev can do stream processing and split a stream into
> > +two or compose two streams into one, but the inputs and outputs for the
> > +subdev are still a single stream per pad.
> >   Some hardware, e.g. MIPI CSI-2, support multiplexed streams, that is, multiple
> >   data streams are transmitted on the same bus, which is represented by a media
> > @@ -539,15 +541,37 @@ streams from one end of the link to the other, and subdevices have routing
> >   tables which describe how the incoming streams from sink pads are routed to the
> >   source pads.
> > -A stream ID (often just "stream") is a media link-local identifier for a stream.
> > +A stream ID is a media link-local identifier for a stream.
> >   In other words, a particular stream ID must exist on both sides of a media
> >   link, but another stream ID can be used for the same stream at the other side
> > -of the subdevice.
> > +of the subdevice. The same stream ID is used to refer to the stream on
> > +both pads of the link on all ioctls operating on pads.
> >   A stream at a specific point in the media pipeline is identified with the
> > -subdev and a (pad, stream) pair. For subdevices that do not support
> > +subdev and a pad--stream pair. For subdevices that do not support
> 
> Is there a double-dash on purpose?

Indeed a single one is correct here. I'l replace it, also below.

> 
> Btw, above you removed the "(often just "stream")", but here you use
> "stream" instead of "stream ID".

We shouldn't mix the two, indeed, as they are different. Indeed, I'll read
this through once more with that in mind.

> 
> >   multiplexed streams the 'stream' is always 0.
> > +Interaction between routes, formats and selections
> > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > +
> > +The addition of routes to the V4L2 sub-device interface moves the
> > +sub-device formats and selections from pads to pad--stream pairs. Instead
> 
> I guess it is, as you use it here also...
> 
> > +of the sub-device wide merging of streams from all source pads towards all
> > +sink pads, this takes place separately for each route. The stream ID on
> > +the sink pad for each configured route is used to configure format and
> > +selections on the sink pad. Similarly, the stream ID on the source pad of
> > +each configured route is used to configure format and selections on the
> > +source pad.
> 
> Hmm, "stream ID is used to configure format" sounds odd to me. The ioctl is
> used to configure the format, but rather than using only pad ID to identify
> the configuration target, you now use pad ID - stream ID pair.

The text continues to say "on the sink pad". I think it's fine as-is.

> 
> And is that just extra duplication above to talk separately about sink and
> source sides? They don't matter here, the point is just the pad-stream vs
> only pad.

Yes, I'll rework the text for v2.

> 
> > +
> > +Any number of routes from streams on sink pads towards streams on source
> 
> This also sounds a bit odd to me: "a route from a stream on a sink pad". I
> think we're missing a word here which means the specific point of a stream
> in the pipeline.
> 
> I have suggested "stream endpoint", but it's not really an "end" point.
> "Stream waypoint"? So you would configure a format to a subdev's stream
> waypoint. And "Any number of routes from stream waypoints on sink pads
> towards stream waypoints on source pads".
> 
> So a "stream waypoint" would be a triplet of subdev-padID-streamID, or just
> padID-streamID if the subdev is obvious.
> 
> Even if we don't find a perfect or even a very good term for this, I think
> we should just use something. Using just "stream" makes things quite
> confusing, in my opinion.
> 
> > +pads is allowed, to the extent supported by drivers. For every stream on a
> > +sink pad, however, only a single route is allowed.
> 
> Hmm, why is that?

We could try to define and document what it would semantically mean but
there likely wouldn't be any devices that support it. The wording makes
future extensions possible in case we'll need it.

> 
> > +
> > +Pad--stream pairs are not static but are replaced by calls to the
> > +:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. This also
> > +applies to stream format and selection configurations which that are
> 
> Should that "applies to" be "affects the"?

I'll reword the paragraph.

> 
> > +reverted to driver defaults.
> > +
> >   Configuring streams
> >   ^^^^^^^^^^^^^^^^^^^
> > @@ -565,8 +589,10 @@ Controller API <media_controller>`
> >   setting the routing table will reset all the stream configurations in a media
> >   entity.
> > -3) Configure streams. Each route endpoint must be configured
> 
> Oh, here I seem to use "route endpoint".
> 
> > -with :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>`.
> > +3) Configure formats and selections on routes. Each route is configured
> 
> I'm not sure if we "configure formats on routes". Earlier I think we've
> talked about configuring the streams (stream waypoints).

Each route has two pad id-stream id pairs (unless sink or source). But yes,
it would be more precise to also refer to pad / stream id pairs instead of
just routes. The propagation takes place in a route still.

I'll rework this for v2.

> 
> > +separately as documented plain subdevices in :ref:`<format-propagation>`.
> 
> Is there something missing from above? Or drop "plain subdevices"?

Yes.

> 
> > +The stream ID is set to the same stream ID used on sink and source pads of
> > +the :ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl.
> 
> Yes. But I think this could be said more clearly if we have a good word for
> the padID-streamID pair.

Good point. Unless it is clear enough that it refers to just that, it
probably will create more confusion than prevent any. If we didn't have
stream that means a different thing it'd be easy. I'm fine with the current
terminology though.

-- 
Terveisin,

Sakari Ailus