[PATCH v6 00/28] Generic line based metadata support, internal pads

[PATCH v6 00/28] Generic line based metadata support, internal pads

[Sakari Ailus]

Hi folks,

Here are a few patches to add support generic, line based metadata as well
as internal pads. While the amount of code is not very large, to the
contrary it is quite small actually IMO, I presume what this is about and
why it is being proposed requires some explaining.

Metadata mbus codes and formats have existed for some time in V4L2. They
however have been only used by drivers that produce the data itself and
effectively this metadata has always been statistics of some sort (at
least when it comes to ISPs). What is different here is that we intend to
add support for metadata originating from camera sensors.

Camera sensors produce different kinds of metadata, embedded data (usually
register address--value pairs used to capture the frame, in a more or less
sensor specific format), histograms (in a very sensor specific format),
dark pixels etc. The number of these formats is probably going to be about
as large as image data formats if not larger, as the image data formats
are much better standardised but a smaller subset of them will be
supported by V4L2, at least initially but possibly much more in the long
run.

Having this many device specific formats would be a major problem for all
the other drivers along that pipeline (not to mention the users of those
drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2
receiver drivers that have DMA: the poor driver developer would not only
need to know camera sensor specific formats but to choose the specific
packing of that format suitable for the DMA used by the hardware. It is
unlikely many of these would ever get tested while being present on the
driver API. Also adding new sensors with new embedded data formats would
involve updating all bridge and CSI-2 receiver drivers. I don't expect
this to be a workable approach.

Instead what I'm proposing is to use specific metadata formats on the
sensor devices only, on internal pads (more about those soon) of the
sensors, only visible in the UAPI, and then generic mbus formats along the
pipeline and finally generic V4L2 metadata formats on the DMAs (specific
to bit depth and packing). This would unsnarl the two, defining what data
there is (specific mbus code) and how that is transported and packed
(generic mbus codes and V4L2 formats).

The user space would be required to "know" the path of that data from the
sensor's internal pad to the V4L2 video node. I do not see this as these
devices require at least some knowledge of the pipeline, i.e. hardware at
hand. Separating what the data means and how it is packed may even be
beneficial: it allows separating code that interprets the data (sensor
internal mbus code) from the code that accesses it (packing).

These formats are in practice line based, meaning that there may be
padding at the end of the line, depending on the bus as well as the DMA.
If non-line based formats are needed, it is always possible to set the
"height" field to 1.

The internal (source) pads are an alternative to source routes [1]. The
source routes were not universally liked and I do have to say I like
re-using existing interface concepts (pads and everything you can do with
pads, including access format, selections etc.) wherever it makes sense,
instead of duplicating functionality.

Effectively internal source pads behave mostly just like sink pads, but
they describe a flow of data that originates from a sub-device instead of
arriving to a sub-device. The SUBDEV_S_ROUTING IOCTLs are used to enable
and disable routes from internal source pads to sub-device's source pads.
The subdev format IOCTLs are usable, too, so one can find which subdev
format is available on given internal source pad.

This set depends on these patches:

<URL:https://lore.kernel.org/linux-media/20231002105557.28972-1-sakari.ailus@linux.intel.com/T/#t>

I've also pushed these here and I'll keep updating the branch, I've also
included untested OV2740 patches:

<URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=metadata>

Questions and comments are most welcome.

Preliminary media-ctl and yavta patches can be found here:

<URL:https://git.retiisi.eu/?p=~sailus/yavta.git;a=shortlog;h=refs/heads/metadata>
<URL:https://git.retiisi.eu/?p=~sailus/v4l-utils.git;a=shortlog;h=refs/heads/metadata>

I have used IMX219 as an example on routing in a sensor driver in this
version. I also hope I can add OV2740 support in the next version after
testing the patches.

[1] <URL:https://lore.kernel.org/linux-media/20220831141357.1396081-20-tomi.valkeinen@ideasonboard.com/>

since v5:

- Rebase on new set of preparation patches.

- Switch CCS driver from s_stream to enable_streams/disable_streams. Keep
  streaming state information --- the sensor remains in streaming state if
  any of the streams is enabled.

- Fix setting mbus code on embedded data in get_frame_desc() op in the CCS
  driver.

since v4:

- Add a patch to acquire two sub-device states that may use the same lock.

- Add a patch for CCS driver to remove ccs_get_crop_compose() helper.

- Add a patch for CCS driver moving acquiring and releasing the mutex to
  the s_stream callback.

- Add a patch for CCS driver to rely on sub-device state locking using a
  single driver-provided lock.

- Fixed calculating minimum number of routes in copying the routes
  (thanks, Laurent).

- Moved a label in S_ROUTING handling to make Clang happy (hopefully).

- Fixed setting emb_data_ctrl register for CCS embedded data support.

- Rebase on Laurent's cleanup patches.

- Wrap a few long lines.

- Write in embedded data documentation sensor drivers generally don't
  allow configuring it.

since v3:

- Separate preparation patches from this set.

- Add a definition for "Data unit", a pixel that is not image data and use
  it instead in format documentation.

- Fix more numbered lists in dev-subdev.rst.

- Remove a redundant definition for V4L2_META_FMT_GENERIC_CSI2_2_24 ---
  V4L2_META_FMT_GENERIC_CSI2_12 can be used instead.

- Use "X" instead of "p" to denote padding in format documentation.

- Use IMX219 in examples instead of CCS.

- Document that the generic V4L2 CSI-2 metadata formats use padding
  defined in CSI-2 spec and packing defined in CCS spec.

- Add patches to align [GS]_ROUTING behaviour with V4L2. This means mainly
  returning configured routes as part of S_ROUTING as well. "len_routes"
  field is added to denote the length of the array and having more routes
  than fits in the array is no longer an error. Also added more reserved
  fields.

- Added trivial support for S_ROUTING (via G_ROUTING implementation) for
  use in drivers with static-only routes.

- Added helper functions to obtain mbus format as well as crop and compose
  rectangles that are streams-independent.

- Added a patch to define generic CSI-2 long packet types.

- Removed MEDIA_BUS_FMT_IS_META() macro. It didn't seem useful in the end.

- Use a single CCS embedded data format. The bit depth can be selected
  using the meta stream on the source pad.

- Fix mbus code numbers (there were holes due to removed redundant
  formats).

- Fix generic mbus code documentation (byte was being used instead of
  bit).

- Fix spelling of "length".

- Added a patch to remove v4l2_subdev_enable_streams_api that disables
  streams API. This should be merged once libcamera support for streams
  works nicely.

- Don't use strings in printing frame descriptor flags.

- Warn on string truncation in printing frame descriptor.

since v2:

- Add a better example, with formats.

- Add CCS static data media bus codes.

- Added an example demonstrating the use of internal pads. --- Is the
  level of detail enough for the purpose?

- Improved documentation.

- Added a macro to tell whether a format is a metadata format.
  (Documentation could be added.)

- A small ReST syntax fix in the same section.

- Drop leftovers of a patch checking for the INTERNAL_SOURCE flag.

since v1:

- Make the new pad flag just "INTERNAL", requiring either SINK or SOURCE
  pad flag to accompany it. Removed the union in struct v4l2_subdev_route.

- Add the term "stream" to MC glossary.

- Improved and fixed documentation (according to comments).

- Note these formats are little endian.

- Remove 1X8 from the names of the mbus codes. These formats have generally
  8 bits per pixel.

- Fix mbus code numbering (had holes in RFC).

- Add new metadata fields to debug prints.

- Fix a minor documentation build issue.

Sakari Ailus (28):
  media: mc: Add INTERNAL pad flag
  media: uapi: Add generic serial metadata mbus formats
  media: uapi: Document which mbus format fields are valid for metadata
  media: uapi: Add generic 8-bit metadata format definitions
  media: v4l: Support line-based metadata capture
  media: uapi: ccs: Add media bus code for MIPI CCS embedded data
  media: Documentation: ccs: Document routing
  media: Documentation: Additional streams generally don't harm capture
  media: Documentation: Document embedded data guidelines for camera
    sensors
  media: Documentation: v4l: Document source routes
  media: Documentation: Document S_ROUTING behaviour
  media: v4l: subdev: Add helpers for format, crop and compose pointers
  media: v4l: subdev: Add a function to lock two sub-device states, use
    it
  media: v4l: subdev: Move G_ROUTING handling below S_ROUTING
  media: v4l: subdev: Copy argument back to user also for S_ROUTING
  media: v4l: subdev: Add len_routes field to struct v4l2_subdev_routing
  media: v4l: subdev: Return routes set using S_ROUTING
  media: uapi: Allow a larger number of routes than there's room for
  media: v4l: subdev: Add trivial set_routing support
  media: uapi: v4l: subdev: Enable streams API
  media: ccs: No need to set streaming to false in power off
  media: ccs: Use {enable,disable}_streams operations
  media: ccs: Track streaming state
  media: ccs: Move ccs_validate_csi_data_format up
  media: ccs: Support frame descriptors
  media: ccs: Add support for embedded data stream
  media: ccs: Remove ccs_get_crop_compose helper
  media: ccs: Rely on sub-device state locking

 .../media/drivers/camera-sensor.rst           |  28 +
 .../userspace-api/media/drivers/ccs.rst       |  34 +-
 .../userspace-api/media/glossary.rst          |  14 +
 .../media/mediactl/media-types.rst            |   6 +
 .../userspace-api/media/v4l/dev-meta.rst      |  15 +
 .../userspace-api/media/v4l/dev-subdev.rst    | 208 ++++-
 .../userspace-api/media/v4l/meta-formats.rst  |   1 +
 .../media/v4l/metafmt-generic.rst             | 304 +++++++
 .../media/v4l/subdev-formats.rst              | 288 ++++++
 .../media/v4l/vidioc-enum-fmt.rst             |   7 +
 .../media/v4l/vidioc-subdev-g-routing.rst     |  40 +-
 .../media/videodev2.h.rst.exceptions          |   1 +
 drivers/media/i2c/ccs/ccs-core.c              | 859 ++++++++++++------
 drivers/media/i2c/ccs/ccs-quirk.h             |   7 +
 drivers/media/i2c/ccs/ccs.h                   |  23 +-
 drivers/media/mc/mc-entity.c                  |  10 +-
 drivers/media/v4l2-core/v4l2-ioctl.c          |  19 +-
 drivers/media/v4l2-core/v4l2-subdev.c         | 145 +--
 include/media/v4l2-subdev.h                   |  98 ++
 include/uapi/linux/media-bus-format.h         |  12 +
 include/uapi/linux/media.h                    |   1 +
 include/uapi/linux/v4l2-mediabus.h            |  18 +-
 include/uapi/linux/v4l2-subdev.h              |   8 +-
 include/uapi/linux/videodev2.h                |  18 +
 24 files changed, 1799 insertions(+), 365 deletions(-)
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst


base-commit: 0fa78064f6e3a354616fb24462864900c0db3191
-- 
2.39.2

[PATCH v6 01/28] media: mc: Add INTERNAL pad flag

[Sakari Ailus]

Internal source pads will be used as routing endpoints in V4L2
[GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.

Also prevent creating links to pads that have been flagged as internal and
initialising source pads with internal flag set.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 Documentation/userspace-api/media/glossary.rst         |  6 ++++++
 .../userspace-api/media/mediactl/media-types.rst       |  6 ++++++
 drivers/media/mc/mc-entity.c                           | 10 ++++++++--
 include/uapi/linux/media.h                             |  1 +
 4 files changed, 21 insertions(+), 2 deletions(-)

diff --git a/Documentation/userspace-api/media/glossary.rst b/Documentation/userspace-api/media/glossary.rst
index 96a360edbf3b..f7b99a4527c7 100644
--- a/Documentation/userspace-api/media/glossary.rst
+++ b/Documentation/userspace-api/media/glossary.rst
@@ -173,6 +173,12 @@ Glossary
 	An integrated circuit that integrates all components of a computer
 	or other electronic systems.
 
+_media-glossary-stream:
+    Stream
+	A distinct flow of data (image data or metadata) over a media pipeline
+	from source to sink. A source may be e.g. an image sensor and a sink
+	e.g. a memory buffer.
+
     V4L2 API
 	**V4L2 userspace API**
 
diff --git a/Documentation/userspace-api/media/mediactl/media-types.rst b/Documentation/userspace-api/media/mediactl/media-types.rst
index 0ffeece1e0c8..28941da27790 100644
--- a/Documentation/userspace-api/media/mediactl/media-types.rst
+++ b/Documentation/userspace-api/media/mediactl/media-types.rst
@@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
 .. _MEDIA-PAD-FL-SINK:
 .. _MEDIA-PAD-FL-SOURCE:
 .. _MEDIA-PAD-FL-MUST-CONNECT:
+.. _MEDIA-PAD-FL-INTERNAL:
 
 .. flat-table:: Media pad flags
     :header-rows:  0
@@ -382,6 +383,11 @@ Types and flags used to represent the media graph elements
 	  when this flag isn't set; the absence of the flag doesn't imply
 	  there is none.
 
+    *  -  ``MEDIA_PAD_FL_INTERNAL``
+       -  The internal flag indicates an internal pad that has no external
+	  connections. Such a pad shall not be connected with a link. The
+	  internal flag indicates that the :ref:``stream
+	  <media-glossary-stream>`` either starts or ends in the entity.
 
 One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
 must be set for every pad.
diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 543a392f8635..f5f290781021 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -213,7 +213,9 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 		iter->index = i++;
 
 		if (hweight32(iter->flags & (MEDIA_PAD_FL_SINK |
-					     MEDIA_PAD_FL_SOURCE)) != 1) {
+					     MEDIA_PAD_FL_SOURCE)) != 1 ||
+		    (iter->flags & MEDIA_PAD_FL_INTERNAL &&
+		     !(iter->flags & MEDIA_PAD_FL_SINK))) {
 			ret = -EINVAL;
 			break;
 		}
@@ -1075,7 +1077,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
 
 	for (i = 0; i < entity->num_pads; i++) {
 		if ((entity->pads[i].flags &
-		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
+		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
+		      MEDIA_PAD_FL_INTERNAL)) != pad_type)
 			continue;
 
 		if (entity->pads[i].sig_type == sig_type)
@@ -1100,6 +1103,9 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
 		return -EINVAL;
 	if (WARN_ON(!(sink->pads[sink_pad].flags & MEDIA_PAD_FL_SINK)))
 		return -EINVAL;
+	if (WARN_ON(source->pads[source_pad].flags & MEDIA_PAD_FL_INTERNAL) ||
+	    WARN_ON(source->pads[sink_pad].flags & MEDIA_PAD_FL_INTERNAL))
+		return -EINVAL;
 
 	link = media_add_link(&source->links);
 	if (link == NULL)
diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
index 1c80b1d6bbaf..80cfd12a43fc 100644
--- a/include/uapi/linux/media.h
+++ b/include/uapi/linux/media.h
@@ -208,6 +208,7 @@ struct media_entity_desc {
 #define MEDIA_PAD_FL_SINK			(1U << 0)
 #define MEDIA_PAD_FL_SOURCE			(1U << 1)
 #define MEDIA_PAD_FL_MUST_CONNECT		(1U << 2)
+#define MEDIA_PAD_FL_INTERNAL			(1U << 3)
 
 struct media_pad_desc {
 	__u32 entity;		/* entity ID */
-- 
2.39.2

[PATCH v6 02/28] media: uapi: Add generic serial metadata mbus formats

[Sakari Ailus]

Add generic serial metadata mbus formats. These formats describe data
width and packing but not the content itself. The reason for specifying
such formats is that the formats as such are fairly device specific but
they are still handled by CSI-2 receiver drivers that should not be aware
of device specific formats. What makes generic metadata formats possible
is that these formats are parsed by software only, after capturing the
data to system memory.

Also add a definition for "Data unit" to cover what is essentially a pixel
but is not image data.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../userspace-api/media/glossary.rst          |   8 +
 .../media/v4l/subdev-formats.rst              | 258 ++++++++++++++++++
 include/uapi/linux/media-bus-format.h         |   9 +
 3 files changed, 275 insertions(+)

diff --git a/Documentation/userspace-api/media/glossary.rst b/Documentation/userspace-api/media/glossary.rst
index f7b99a4527c7..65217b8a44cc 100644
--- a/Documentation/userspace-api/media/glossary.rst
+++ b/Documentation/userspace-api/media/glossary.rst
@@ -25,6 +25,14 @@ Glossary
 
 	See :ref:`cec`.
 
+.. _media-glossary-data-unit:
+
+    Data unit
+
+	Unit of data transported by a bus. On parallel buses, this is called a
+	sample while on serial buses the data unit is logical. If the data unit
+	is image data, it may also be called a pixel.
+
     Device Driver
 	Part of the Linux Kernel that implements support for a hardware
 	component.
diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
index a3a35eeed708..c54bf834d839 100644
--- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
+++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
@@ -8234,3 +8234,261 @@ The following table lists the existing metadata formats.
 	both sides of the link and the bus format is a fixed
 	metadata format that is not configurable from userspace.
 	Width and height will be set to 0 for this format.
+
+Generic Serial Metadata Formats
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Generic serial metadata formats are used on serial buses where the actual data
+content is more or less device specific but the data is transmitted and received
+by multiple devices that do not process the data in any way, simply writing
+it to system memory for processing in software at the end of the pipeline.
+
+The more specific variant describing the actual data is used on the internal
+source pad of the originating sub-device.
+
+"b" in an array cell signifies a byte of data, followed by the number of the bit
+and finally the bit number in subscript. "X" indicates a padding bit.
+
+.. _media-bus-format-generic-meta:
+
+.. cssclass: longtable
+
+.. flat-table:: Generic Serial Metadata Formats
+    :header-rows:  2
+    :stub-columns: 0
+
+    * - Identifier
+      - Code
+      -
+      - :cspan:`23` Data organization within bus ``Data unit
+	<media-glossary-data-unit>``
+    * -
+      -
+      - Bit
+      - 23
+      - 22
+      - 21
+      - 20
+      - 19
+      - 18
+      - 17
+      - 16
+      - 15
+      - 14
+      - 13
+      - 12
+      - 11
+      - 10
+      - 9
+      - 8
+      - 7
+      - 6
+      - 5
+      - 4
+      - 3
+      - 2
+      - 1
+      - 0
+    * .. _MEDIA-BUS-FMT-META-8:
+
+      - MEDIA_BUS_FMT_META_8
+      - 0x8001
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+    * .. _MEDIA-BUS-FMT-META-10:
+
+      - MEDIA_BUS_FMT_META_10
+      - 0x8002
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - X
+      - X
+    * .. _MEDIA-BUS-FMT-META-12:
+
+      - MEDIA_BUS_FMT_META_12
+      - 0x8003
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - X
+      - X
+      - X
+      - X
+    * .. _MEDIA-BUS-FMT-META-14:
+
+      - MEDIA_BUS_FMT_META_14
+      - 0x8004
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+    * .. _MEDIA-BUS-FMT-META-16:
+
+      - MEDIA_BUS_FMT_META_16
+      - 0x8005
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+    * .. _MEDIA-BUS-FMT-META-20:
+
+      - MEDIA_BUS_FMT_META_20
+      - 0x8006
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+    * .. _MEDIA-BUS-FMT-META-24:
+
+      - MEDIA_BUS_FMT_META_24
+      - 0x8007
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
index a03c543cb072..9ee031397372 100644
--- a/include/uapi/linux/media-bus-format.h
+++ b/include/uapi/linux/media-bus-format.h
@@ -173,4 +173,13 @@
  */
 #define MEDIA_BUS_FMT_METADATA_FIXED		0x7001
 
+/* Generic line based metadata formats for serial buses. Next is 0x8008. */
+#define MEDIA_BUS_FMT_META_8			0x8001
+#define MEDIA_BUS_FMT_META_10			0x8002
+#define MEDIA_BUS_FMT_META_12			0x8003
+#define MEDIA_BUS_FMT_META_14			0x8004
+#define MEDIA_BUS_FMT_META_16			0x8005
+#define MEDIA_BUS_FMT_META_20			0x8006
+#define MEDIA_BUS_FMT_META_24			0x8007
+
 #endif /* __LINUX_MEDIA_BUS_FORMAT_H */
-- 
2.39.2

[PATCH v6 03/28] media: uapi: Document which mbus format fields are valid for metadata

[Sakari Ailus]

Now that metadata mbus formats have been added, it is necessary to define
which fields in struct v4l2_mbus_format are applicable to them (not many).

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 include/uapi/linux/v4l2-mediabus.h | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)

diff --git a/include/uapi/linux/v4l2-mediabus.h b/include/uapi/linux/v4l2-mediabus.h
index 6b07b73473b5..3cadb3b58b85 100644
--- a/include/uapi/linux/v4l2-mediabus.h
+++ b/include/uapi/linux/v4l2-mediabus.h
@@ -19,12 +19,18 @@
  * @width:	image width
  * @height:	image height
  * @code:	data format code (from enum v4l2_mbus_pixelcode)
- * @field:	used interlacing type (from enum v4l2_field)
- * @colorspace:	colorspace of the data (from enum v4l2_colorspace)
- * @ycbcr_enc:	YCbCr encoding of the data (from enum v4l2_ycbcr_encoding)
- * @hsv_enc:	HSV encoding of the data (from enum v4l2_hsv_encoding)
- * @quantization: quantization of the data (from enum v4l2_quantization)
- * @xfer_func:  transfer function of the data (from enum v4l2_xfer_func)
+ * @field:	used interlacing type (from enum v4l2_field), not applicable
+ *		to metadata mbus codes
+ * @colorspace:	colorspace of the data (from enum v4l2_colorspace), zero on
+ *		metadata mbus codes
+ * @ycbcr_enc:	YCbCr encoding of the data (from enum v4l2_ycbcr_encoding), zero
+ *		on metadata mbus codes
+ * @hsv_enc:	HSV encoding of the data (from enum v4l2_hsv_encoding), zero on
+ *		metadata mbus codes
+ * @quantization: quantization of the data (from enum v4l2_quantization), zero
+ *		on metadata mbus codes
+ * @xfer_func:  transfer function of the data (from enum v4l2_xfer_func), zero
+ *		on metadata mbus codes
  * @flags:	flags (V4L2_MBUS_FRAMEFMT_*)
  * @reserved:  reserved bytes that can be later used
  */
-- 
2.39.2

[PATCH v6 04/28] media: uapi: Add generic 8-bit metadata format definitions

[Sakari Ailus]

Generic 8-bit metadata formats define the in-memory data layout but not
the format of the data itself. The reasoning for having such formats is to
allow CSI-2 receiver drivers to receive and DMA drivers to write the data
to memory without knowing a large number of device specific formats.

These formats may be used only in conjunction of a Media controller
pipeline where the internal pad of the source sub-device defines the
specific format of the data (using an mbus code).

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../userspace-api/media/v4l/dev-subdev.rst    |   2 +
 .../userspace-api/media/v4l/meta-formats.rst  |   1 +
 .../media/v4l/metafmt-generic.rst             | 304 ++++++++++++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c          |   7 +
 include/uapi/linux/videodev2.h                |   8 +
 5 files changed, 322 insertions(+)
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst

diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index 43988516acdd..f375b820ab68 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -506,6 +506,8 @@ source pads.
 
     subdev-formats
 
+.. _subdev-routing:
+
 Streams, multiplexed media pads and internal routing
 ----------------------------------------------------
 
diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
index 0bb61fc5bc00..919f595576b9 100644
--- a/Documentation/userspace-api/media/v4l/meta-formats.rst
+++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
@@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only.
     metafmt-vsp1-hgo
     metafmt-vsp1-hgt
     metafmt-vivid
+    metafmt-generic
diff --git a/Documentation/userspace-api/media/v4l/metafmt-generic.rst b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
new file mode 100644
index 000000000000..2ebab1f895e0
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
@@ -0,0 +1,304 @@
+.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
+
+**************************************************************************************************************************************************************************************************************************************************************************************************************************
+V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O')
+**************************************************************************************************************************************************************************************************************************************************************************************************************************
+
+
+Generic line-based metadata formats
+
+
+Description
+===========
+
+These generic line-based metadata formats define the memory layout of the data
+without defining the format or meaning of the metadata itself. These formats may
+only be used with a Media controller pipeline where the more specific format is
+defined in an :ref:`internal source pad <MEDIA-PAD-FL-INTERNAL>` of the source
+sub-device. See also :ref:`source routes <subdev-routing>`.
+
+.. _v4l2-meta-fmt-generic-8:
+
+V4L2_META_FMT_GENERIC_8
+-----------------------
+
+The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format.
+
+This format is also used on CSI-2 for both 8 bits per ``Data unit
+<media-glossary-data-unit>`` as well as for 16 bits per Data unit when two bytes
+of metadata are packed into one 16-bit Data unit.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_8.**
+Each cell is one byte. "M" denotes a byte of metadata.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+    * - start + 4:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+
+.. _v4l2-meta-fmt-generic-csi2-10:
+
+V4L2_META_FMT_GENERIC_CSI2_10
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
+for each 8 bits of data. Every four bytes of metadata is followed by a single
+byte of padding. The way the data is packed follows the MIPI CSI-2 specification
+and the padding is defined in the MIPI CCS specification.
+
+This format is also used in conjunction with 20 bits per ``Data unit
+<media-glossary-data-unit>`` formats that pack two bytes of metadata into one
+Data unit.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
+Each cell is one byte. "M" denotes a byte of metadata and "X" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - X
+    * - start + 5:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - X
+
+.. _v4l2-meta-fmt-generic-csi2-12:
+
+V4L2_META_FMT_GENERIC_CSI2_12
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_12 contains packed 8-bit generic metadata, 12 bits
+for each 8 bits of data. Every four bytes of metadata is followed by a single
+byte of padding. The way the data is packed follows the MIPI CSI-2 specification
+and the padding is defined in the MIPI CCS specification.
+
+This format is also used in conjunction with 24 bits per ``Data unit
+<media-glossary-data-unit>`` formats that pack two bytes of metadata into one
+Data unit.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_12.**
+Each cell is one byte. "M" denotes a byte of metadata and "X" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - X
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - X
+    * - start + 6:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - X
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - X
+
+.. _v4l2-meta-fmt-generic-csi2-14:
+
+V4L2_META_FMT_GENERIC_CSI2_14
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_14 contains packed 8-bit generic metadata, 14 bits
+for each 8 bits of data. Every four bytes of metadata is followed by three bytes
+of padding. The way the data is packed follows the MIPI CSI-2 specification and
+the padding is defined in the MIPI CCS specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_14.**
+Each cell is one byte. "M" denotes a byte of metadata and "X" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - X
+      - X
+      - X
+    * - start + 7:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - X
+      - X
+      - X
+
+.. _v4l2-meta-fmt-generic-csi2-16:
+
+V4L2_META_FMT_GENERIC_CSI2_16
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_16 contains packed 8-bit generic metadata, 16 bits
+for each 8 bits of data. Every byte of metadata is followed by one byte of
+padding. The way the data is packed follows the MIPI CSI-2 specification and the
+padding is defined in the MIPI CCS specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_16.**
+Each cell is one byte. "M" denotes a byte of metadata and "X" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - X
+      - M\ :sub:`10`
+      - X
+      - M\ :sub:`20`
+      - X
+      - M\ :sub:`30`
+      - X
+    * - start + 8:
+      - M\ :sub:`01`
+      - X
+      - M\ :sub:`11`
+      - X
+      - M\ :sub:`21`
+      - X
+      - M\ :sub:`31`
+      - X
+
+.. _v4l2-meta-fmt-generic-csi2-20:
+
+V4L2_META_FMT_GENERIC_CSI2_20
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_20 contains packed 8-bit generic metadata, 20 bits
+for each 8 bits of data. Every byte of metadata is followed by alternating one
+and two bytes of padding. The way the data is packed follows the MIPI CSI-2
+specification and the padding is defined in the MIPI CCS specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_20.**
+Each cell is one byte. "M" denotes a byte of metadata and "X" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - X
+      - M\ :sub:`10`
+      - X
+      - X
+      - M\ :sub:`20`
+      - X
+      - M\ :sub:`30`
+      - X
+      - X
+    * - start + 10:
+      - M\ :sub:`01`
+      - X
+      - M\ :sub:`11`
+      - X
+      - X
+      - M\ :sub:`21`
+      - X
+      - M\ :sub:`31`
+      - X
+      - X
+
+.. _v4l2-meta-fmt-generic-csi2-24:
+
+V4L2_META_FMT_GENERIC_CSI2_24
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_24 contains packed 8-bit generic metadata, 24 bits
+for each 8 bits of data. Every byte of metadata is followed by two bytes of
+padding. The way the data is packed follows the MIPI CSI-2 specification and the
+padding is defined in the MIPI CCS specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_24.**
+Each cell is one byte. "M" denotes a byte of metadata and "X" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - X
+      - X
+      - M\ :sub:`10`
+      - X
+      - X
+      - M\ :sub:`20`
+      - X
+      - X
+      - M\ :sub:`30`
+      - X
+      - X
+    * - start + 12:
+      - M\ :sub:`01`
+      - X
+      - X
+      - M\ :sub:`11`
+      - X
+      - X
+      - M\ :sub:`21`
+      - X
+      - X
+      - M\ :sub:`31`
+      - X
+      - X
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index 9b1de54ce379..ce4b3929ff5f 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -1452,6 +1452,13 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
 	case V4L2_PIX_FMT_Y210:		descr = "10-bit YUYV Packed"; break;
 	case V4L2_PIX_FMT_Y212:		descr = "12-bit YUYV Packed"; break;
 	case V4L2_PIX_FMT_Y216:		descr = "16-bit YUYV Packed"; break;
+	case V4L2_META_FMT_GENERIC_8:	descr = "8-bit Generic Metadata"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_10:	descr = "8b Generic Meta, 10b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_12:	descr = "8b Generic Meta, 12b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_14:	descr = "8b Generic Meta, 14b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_16:	descr = "8b Generic Meta, 16b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_20:	descr = "8b Generic Meta, 20b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_24:	descr = "8b Generic Meta, 24b CSI-2"; break;
 
 	default:
 		/* Compressed formats */
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index c3d4e490ce7c..2b16b06ad278 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -839,6 +839,14 @@ struct v4l2_pix_format {
 #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
 #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
 
+#define V4L2_META_FMT_GENERIC_8		v4l2_fourcc('M', 'E', 'T', '8') /* Generic 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_10	v4l2_fourcc('M', 'C', '1', 'A') /* 10-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_12	v4l2_fourcc('M', 'C', '1', 'C') /* 12-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_14	v4l2_fourcc('M', 'C', '1', 'E') /* 14-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_16	v4l2_fourcc('M', 'C', '1', 'G') /* 16-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_20	v4l2_fourcc('M', 'C', '1', 'K') /* 20-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_24	v4l2_fourcc('M', 'C', '1', 'O') /* 24-bit CSI-2 packed 8-bit metadata */
+
 /* priv field value to indicates that subsequent fields are valid. */
 #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
 
-- 
2.39.2

[PATCH v6 05/28] media: v4l: Support line-based metadata capture

[Sakari Ailus]

many camera sensors, among other devices, transmit embedded data and image
data for each CSI-2 frame. This embedded data typically contains register
configuration of the sensor that has been used to capture the image data
of the same frame.

The embedded data is received by the CSI-2 receiver and has the same
properties as the image data, including that it is line based: it has
width, height and bytesperline (stride).

Add these fields to struct v4l2_meta_format and document them.

Also add V4L2_FMT_FLAG_META_LINE_BASED to tell a given format is
line-based i.e. these fields of struct v4l2_meta_format are valid for it.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../userspace-api/media/v4l/dev-meta.rst          | 15 +++++++++++++++
 .../userspace-api/media/v4l/vidioc-enum-fmt.rst   |  7 +++++++
 .../media/videodev2.h.rst.exceptions              |  1 +
 drivers/media/v4l2-core/v4l2-ioctl.c              |  5 +++--
 include/uapi/linux/videodev2.h                    | 10 ++++++++++
 5 files changed, 36 insertions(+), 2 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/dev-meta.rst b/Documentation/userspace-api/media/v4l/dev-meta.rst
index 0e7e1ee1471a..4dfd79e0a705 100644
--- a/Documentation/userspace-api/media/v4l/dev-meta.rst
+++ b/Documentation/userspace-api/media/v4l/dev-meta.rst
@@ -65,3 +65,18 @@ to 0.
       - ``buffersize``
       - Maximum buffer size in bytes required for data. The value is set by the
         driver.
+    * - __u32
+      - ``width``
+      - Width of a line of metadata in Data units. Valid when
+	:c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set,
+	otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
+    * - __u32
+      - ``height``
+      - Number of rows of metadata. Valid when :c:type`v4l2_fmtdesc` flag
+	``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
+	:c:func:`VIDIOC_ENUM_FMT`.
+    * - __u32
+      - ``bytesperline``
+      - Offset in bytes between the beginning of two consecutive lines. Valid
+	when :c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is
+	set, otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
diff --git a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
index 000c154b0f98..a79abf4428c8 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
@@ -227,6 +227,13 @@ the ``mbus_code`` field is handled differently:
 	The application can ask to configure the quantization of the capture
 	device when calling the :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl with
 	:ref:`V4L2_PIX_FMT_FLAG_SET_CSC <v4l2-pix-fmt-flag-set-csc>` set.
+    * - ``V4L2_FMT_FLAG_META_LINE_BASED``
+      - 0x0200
+      - The metadata format is line-based. In this case the ``width``,
+	``height`` and ``bytesperline`` fields of :c:type:`v4l2_meta_format` are
+	valid. The buffer consists of ``height`` lines, each having ``width``
+	Data units of data and offset (in bytes) between the beginning of each
+	two consecutive lines is ``bytesperline``.
 
 Return Value
 ============
diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b/Documentation/userspace-api/media/videodev2.h.rst.exceptions
index 3e58aac4ef0b..bdc628e8c1d6 100644
--- a/Documentation/userspace-api/media/videodev2.h.rst.exceptions
+++ b/Documentation/userspace-api/media/videodev2.h.rst.exceptions
@@ -215,6 +215,7 @@ replace define V4L2_FMT_FLAG_CSC_XFER_FUNC fmtdesc-flags
 replace define V4L2_FMT_FLAG_CSC_YCBCR_ENC fmtdesc-flags
 replace define V4L2_FMT_FLAG_CSC_HSV_ENC fmtdesc-flags
 replace define V4L2_FMT_FLAG_CSC_QUANTIZATION fmtdesc-flags
+replace define V4L2_FMT_FLAG_META_LINE_BASED fmtdesc-flags
 
 # V4L2 timecode types
 replace define V4L2_TC_TYPE_24FPS timecode-type
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index ce4b3929ff5f..fb453b7d0c91 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -343,8 +343,9 @@ static void v4l_print_format(const void *arg, bool write_only)
 	case V4L2_BUF_TYPE_META_OUTPUT:
 		meta = &p->fmt.meta;
 		pixelformat = meta->dataformat;
-		pr_cont(", dataformat=%p4cc, buffersize=%u\n",
-			&pixelformat, meta->buffersize);
+		pr_cont(", dataformat=%p4cc, buffersize=%u, width=%u, height=%u, bytesperline=%u\n",
+			&pixelformat, meta->buffersize, meta->width,
+			meta->height, meta->bytesperline);
 		break;
 	}
 }
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 2b16b06ad278..7b0781a20dbe 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -877,6 +877,7 @@ struct v4l2_fmtdesc {
 #define V4L2_FMT_FLAG_CSC_YCBCR_ENC		0x0080
 #define V4L2_FMT_FLAG_CSC_HSV_ENC		V4L2_FMT_FLAG_CSC_YCBCR_ENC
 #define V4L2_FMT_FLAG_CSC_QUANTIZATION		0x0100
+#define V4L2_FMT_FLAG_META_LINE_BASED		0x0200
 
 	/* Frame Size and frame rate enumeration */
 /*
@@ -2420,10 +2421,19 @@ struct v4l2_sdr_format {
  * struct v4l2_meta_format - metadata format definition
  * @dataformat:		little endian four character code (fourcc)
  * @buffersize:		maximum size in bytes required for data
+ * @width:		number of data units of data per line (valid for line
+ *			based formats only, see format documentation)
+ * @height:		number of lines of data per buffer (valid for line based
+ *			formats only)
+ * @bytesperline:	offset between the beginnings of two adjacent lines in
+ *			bytes (valid for line based formats only)
  */
 struct v4l2_meta_format {
 	__u32				dataformat;
 	__u32				buffersize;
+	__u32				width;
+	__u32				height;
+	__u32				bytesperline;
 } __attribute__ ((packed));
 
 /**
-- 
2.39.2

[PATCH v6 06/28] media: uapi: ccs: Add media bus code for MIPI CCS embedded data

[Sakari Ailus]

Add new MIPI CCS embedded data media bus code
(MEDIA_BUS_FMT_CCS_EMBEDDED).

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../media/v4l/subdev-formats.rst              | 28 +++++++++++++++++++
 include/uapi/linux/media-bus-format.h         |  3 ++
 2 files changed, 31 insertions(+)

diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
index c54bf834d839..70cad81bf862 100644
--- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
+++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
@@ -8492,3 +8492,31 @@ and finally the bit number in subscript. "X" indicates a padding bit.
       - X
       - X
       - X
+
+.. _MEDIA-BUS-FMT-CCS-EMBEDDED:
+
+MIPI CCS Embedded Data Formats
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+`MIPI CCS <https://www.mipi.org/specifications/camera-command-set>`_ defines a
+metadata format for sensor embedded data, which is used to store the register
+configuration used for capturing a given frame. The format is defined in the CCS
+specification. The media bus code for this format is
+``MEDIA_BUS_FMT_CCS_EMBEDDED``.
+
+The CCS embedded data format definition includes three levels:
+
+1. Padding within CSI-2 bus :ref:`Data unit <media-glossary-data-unit>` as
+   documented in the MIPI CCS specification.
+
+2. The tagged data format as documented in the MIPI CCS specification.
+
+3. Register addresses and register documentation as documented in the MIPI CCS
+   specification.
+
+The format definition shall be used only by devices that fulfill all three
+levels above.
+
+This mbus code are only used for "2-byte simplified tagged data format" (code
+0xa) but their use may be extended further in the future, to cover other CCS
+embedded data format codes.
diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
index 9ee031397372..9805f8e73e48 100644
--- a/include/uapi/linux/media-bus-format.h
+++ b/include/uapi/linux/media-bus-format.h
@@ -182,4 +182,7 @@
 #define MEDIA_BUS_FMT_META_20			0x8006
 #define MEDIA_BUS_FMT_META_24			0x8007
 
+/* Specific metadata formats. Next is 0x9002. */
+#define MEDIA_BUS_FMT_CCS_EMBEDDED		0x9001
+
 #endif /* __LINUX_MEDIA_BUS_FORMAT_H */
-- 
2.39.2

[PATCH v6 07/28] media: Documentation: ccs: Document routing

[Sakari Ailus]

Document which routes are available for the CCS driver (source) sub-device
and what configuration are possible.

Also update copyright.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../userspace-api/media/drivers/ccs.rst       | 34 ++++++++++++++++++-
 .../media/v4l/subdev-formats.rst              |  2 ++
 2 files changed, 35 insertions(+), 1 deletion(-)

diff --git a/Documentation/userspace-api/media/drivers/ccs.rst b/Documentation/userspace-api/media/drivers/ccs.rst
index 161cb65f4d98..ad8615233bae 100644
--- a/Documentation/userspace-api/media/drivers/ccs.rst
+++ b/Documentation/userspace-api/media/drivers/ccs.rst
@@ -107,4 +107,36 @@ than in the centre.
 Shading correction needs to be enabled for luminance correction level to have an
 effect.
 
-**Copyright** |copy| 2020 Intel Corporation
+.. _media-ccs-routes:
+
+Routes
+------
+
+The CCS driver implements one or two :ref:`routes <subdev-routing>` in
+its source sub-device (scaler sub-device if exists for the device, otherwise
+binner) depending on whether the sensor supports embedded data. (All CCS
+compliant sensors do but the CCS driver supports preceding standards that did
+not require embedded data support, too.)
+
+The first route of the CCS source sub-device is for pixel data (internal pad
+1/stream 0 -> pad 0/stream 0) and the second one is for embedded data (internal
+pad 2/stream 0 -> pad 0/stream 1).
+
+Embedded data
+~~~~~~~~~~~~~
+
+MIPI CCS supports generation of camera sensor embedded data. The media bus code
+used for this format is :ref:`MEDIA_BUS_FMT_CCS_EMBEDDDED
+<MEDIA-BUS-FMT-CCS-EMBEDDED>`.
+
+The bit depth of the CCS pixel data affects how the sensor will output the
+embedded data, adding padding to align with CSI-2 bus :ref:`Data units
+<media-glossary-data-unit>` for that particular bit depth. This is indicated by
+the generic metadata format on the sensor's source sub-device's source pad.
+
+Embedded data for bit depths greater than or equal to 16 may support more dense
+packing or legacy single metadata byte per data unit, or both of these,
+depending on the device. The supported embedded data formats can be enumerated
+and configured on stream 1 of the source pad (1) of the CCS source sub-device.
+
+**Copyright** |copy| 2020, 2023 Intel Corporation
diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
index 70cad81bf862..62396a47af4d 100644
--- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
+++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
@@ -8520,3 +8520,5 @@ levels above.
 This mbus code are only used for "2-byte simplified tagged data format" (code
 0xa) but their use may be extended further in the future, to cover other CCS
 embedded data format codes.
+
+Also see :ref:`CCS driver documentation <media-ccs-routes>`.
-- 
2.39.2

[PATCH v6 08/28] media: Documentation: Additional streams generally don't harm capture

[Sakari Ailus]

Having extra streams on the source end of the link that cannot be captured
by the sink sub-device generally are not an issue, at least not on CSI-2
bus. Still document that there may be hardware specific limitations. For
example on parallel bus this might not work on all cases.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 Documentation/userspace-api/media/v4l/dev-subdev.rst | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index f375b820ab68..a387e8a15b8d 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -529,9 +529,9 @@ the its sink pad and allows to route them individually to one of its source
 pads.
 
 Subdevice drivers that support multiplexed streams are compatible with
-non-multiplexed subdev drivers, but, of course, require a routing configuration
-where the link between those two types of drivers contains only a single
-stream.
+non-multiplexed subdev drivers. However, if the driver at the sink end of a link
+does not support streams, then only the stream 0 on source end may be
+captured. There may be additional hardware specific limitations.
 
 Understanding streams
 ^^^^^^^^^^^^^^^^^^^^^
-- 
2.39.2

[PATCH v6 09/28] media: Documentation: Document embedded data guidelines for camera sensors

[Sakari Ailus]

Document how embedded data support should be implemented for camera
sensors, and when and how CCS embedded data format should be referenced.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../media/drivers/camera-sensor.rst           | 28 +++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/Documentation/userspace-api/media/drivers/camera-sensor.rst b/Documentation/userspace-api/media/drivers/camera-sensor.rst
index 919a50e8b9d9..308f391c5ca1 100644
--- a/Documentation/userspace-api/media/drivers/camera-sensor.rst
+++ b/Documentation/userspace-api/media/drivers/camera-sensor.rst
@@ -102,3 +102,31 @@ register programming sequences shall initialize the :ref:`V4L2_CID_HFLIP
 values programmed by the register sequences. The default values of these
 controls shall be 0 (disabled). Especially these controls shall not be inverted,
 independently of the sensor's mounting rotation.
+
+Embedded data
+-------------
+
+Many sensors, mostly raw sensors, support embedded data which is used to convey
+the sensor configuration for the captured frame back to the host. While CSI-2 is
+the most common bus used by such sensors, embedded data is not entirely limited
+to CSI-2 bus due to e.g. bridge devices.
+
+Embedded data support should use an internal source pad and route to the
+external pad. If embedded data output can be disabled in hardware, it should be
+possible to disable the embedded data route via ``VIDIOC_SUBDEV_S_ROUTING``
+IOCTL.
+
+In general, changing the embedded data format from the driver-configured values
+is not supported. The height of the metadata is hardware specific and the width
+is that (or less of that) of the image width, as configured on the pixel data
+stream.
+
+CCS and non-CCS embedded data
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Embedded data which is compliant with CCS definitions shall use ``CCS embedded
+data format <MEDIA-BUS-FMT-CCS-EMBEDDED>``. Device specific embedded data which
+is compliant up to MIPI CCS embedded data levels 1 or 2 only shall refer to CCS
+embedded data formats and document the level of conformance. The rest of the
+device specific embedded data format shall be documented in the context of the
+data format itself.
-- 
2.39.2

[PATCH v6 10/28] media: Documentation: v4l: Document source routes

[Sakari Ailus]

Document how internal pads are used on source routes. Use the IMX219
camera sensor as an example.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../userspace-api/media/v4l/dev-subdev.rst    | 179 ++++++++++++++++++
 1 file changed, 179 insertions(+)

diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index a387e8a15b8d..fb73a95401c3 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -553,6 +553,27 @@ A stream at a specific point in the media pipeline is identified by the
 sub-device and a (pad, stream) pair. For sub-devices that do not support
 multiplexed streams the 'stream' field is always 0.
 
+.. _v4l2-subdev-source-routes:
+
+Internal pads and source routes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Cases where a single sub-device source pad is traversed by multiple streams one
+or more of which originate from within the sub-device itself are special as
+there is no external sink pad for such routes. In those cases, the sources of
+the internally generated streams are represented by internal sink pads, which
+are sink pads that have the :ref:`MEDIA_PAD_FL_INTERNAL <MEDIA-PAD-FL-INTERNAL>`
+pad flag set.
+
+Internal pads have all the properties of an external pad, including formats and
+selections. The format in this case is the source format of the stream. An
+internal pad always has a single stream only (0).
+
+*Source routes* are routes from an internal sink pad to an external source
+pad. In most cases source routes are not modifiable but they can be activated
+and deactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE
+<v4l2-subdev-routing-flags>` flag, depending on driver capabilities.
+
 Interaction between routes, streams, formats and selections
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -668,3 +689,161 @@ To configure this pipeline, the userspace must take the following steps:
    the configurations along the stream towards the receiver, using
    :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each
    stream endpoint in each sub-device.
+
+Internal pads setup example
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A simple example of a multiplexed stream setup might be as follows:
+
+- A CCS camera sensor source sub-device, with one sink pad (0), one source pad
+  (1), an internal sink pad (2) that represents the source of embedded
+  data. There are two routes, one from the sink pad to the source, and another
+  from the internal sink pad to the source pad. The embedded data stream needs
+  to be enabled by activating the related route. The configuration of the rest
+  of the CCS sub-devices is omitted from this example.
+
+- Multiplexer bridge (Bridge). The bridge has one sink pad, connected to the
+  sensor (pad 0), and one source pad (pad 1), which outputs two streams.
+
+- Receiver in the SoC (Receiver). The receiver has a single sink pad (pad 0),
+  connected to the bridge, and two source pads (pads 1-2), going to the DMA
+  engine. The receiver demultiplexes the incoming streams to the source pads.
+
+- DMA Engines in the SoC (DMA Engine), one for each stream. Each DMA engine is
+  connected to a single source pad in the receiver.
+
+The sensor, the bridge and the receiver are modeled as V4L2 sub-devices,
+exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are
+modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
+
+To configure this pipeline, the userspace must take the following steps:
+
+1) Set up media links between entities: connect the sensors to the bridge,
+   bridge to the receiver, and the receiver to the DMA engines. This step does
+   not differ from normal non-multiplexed media controller setup.
+
+2) Configure routing
+
+.. flat-table:: Camera sensor
+    :header-rows: 1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0
+      - 1/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from the sink pad
+    * - 2/0
+      - 1/1
+      - **V4L2_SUBDEV_ROUTE_FL_ACTIVE**
+      - Metadata stream from the internal sink pad
+
+.. flat-table:: Bridge routing table
+    :header-rows: 1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0
+      - 1/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from camera sensor
+    * - 0/1
+      - 1/1
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from camera sensor
+
+.. flat-table:: Receiver routing table
+    :header-rows:  1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0
+      - 1/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from camera sensor
+    * - 0/1
+      - 2/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from camera sensor
+
+The options available in sensor's routing configuration are dictated by
+hardware capabilities: typically camera sensors always produce image data
+stream while the embedded data stream typically can be either enabled or
+disabled.
+
+3) Configure formats and selections
+
+   This example assumes that the formats are propagated from sink pad to the
+   source pad as-is. The tables contain fields of both struct v4l2_subdev_format
+   and struct v4l2_mbus_framefmt.
+
+.. flat-table:: Formats set on the sub-devices. Bold values are set, others are
+                static or propagated.
+    :header-rows: 1
+    :fill-cells:
+
+    * - Sub-device
+      - Pad/Stream
+      - Width
+      - Height
+      - Code
+    * - :rspan:`3` Camera sensor sub-device (IMX219)
+      - 1/0
+      - 3296
+      - 2480
+      - MEDIA_BUS_FMT_SRGGB10
+    * - 0/0
+      - **3296**
+      - **2480**
+      - **MEDIA_BUS_FMT_SRGGB10**
+    * - 2/0
+      - 3296
+      - 2
+      - MEDIA_BUS_FMT_IMX219_EMBEDDED
+    * - 1/1
+      - 3296
+      - 2
+      - MEDIA_BUS_FMT_META_10
+    * - :rspan:`3` Bridge
+      - 0/0
+      - **3296**
+      - **2480**
+      - **MEDIA_BUS_FMT_SRGGB10**
+    * - 1/0
+      - 3296
+      - 2480
+      - MEDIA_BUS_FMT_SRGGB10
+    * - 0/1
+      - **3296**
+      - **2**
+      - **MEDIA_BUS_FMT_META_10**
+    * - 1/1
+      - 3296
+      - 2
+      - MEDIA_BUS_FMT_META_10
+    * - :rspan:`3` Receiver
+      - 0/0
+      - **3296**
+      - **2480**
+      - **MEDIA_BUS_FMT_SRGGB10**
+    * - 1/0
+      - 3296
+      - 2480
+      - MEDIA_BUS_FMT_SRGGB10
+    * - 0/1
+      - **3296**
+      - **2**
+      - **MEDIA_BUS_FMT_META_10**
+    * - 2/0
+      - 3296
+      - 2
+      - MEDIA_BUS_FMT_META_10
+
+The embedded data format does not need to be configured as the format is
+dictated by the pixel data format in this case.
-- 
2.39.2

[PATCH v6 11/28] media: Documentation: Document S_ROUTING behaviour

[Sakari Ailus]

Document S_ROUTING behaviour for different devices.

Generally in devices that produce streams the streams are static and some
can be enabled and disabled, whereas in devices that just transport them
or write them to memory, more configurability is allowed. Document this.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../userspace-api/media/v4l/dev-subdev.rst    | 21 +++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index fb73a95401c3..83993775237f 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -593,6 +593,27 @@ Any configurations of a stream within a pad, such as format or selections,
 are independent of similar configurations on other streams. This is
 subject to change in the future.
 
+Device types and routing setup
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Different kinds of sub-devices have differing behaviour for route inactivation,
+depending on the hardware. Devices generating the streams may often allow
+enabling and disabling some of these or the configuration is fixed. If these
+routes can be disabled, not declaring these routes (or declaring them without
+``VIDIOC_SUBDEV_STREAM_FL_ACTIVE`` flag set) in ``VIDIOC_SUBDEV_S_ROUTING`` will
+disable the routes while the sub-device driver retain the streams and their
+configuration. The ``VIDIOC_SUBDEV_S_ROUTING`` will still return such routes
+back to the user in the routes array, with the ``V4L2_SUBDEV_STREAM_FL_ACTIVE``
+flag unset.
+
+Devices transporting the streams almost always have more configurability with
+respect to routing. Typically any route between the sub-device's sink and source
+pads is possible, and multiple routes (usually up to certain limited number) may
+be active simultaneously. For such devices, no routes are created by the driver
+and user-created routes are fully replaced when ``VIDIOC_SUBDEV_S_ROUTING`` is
+called on the sub-device. Such newly created routes have the device's default
+configuration for format and selection rectangles.
+
 Configuring streams
 ^^^^^^^^^^^^^^^^^^^
 
-- 
2.39.2

[PATCH v6 12/28] media: v4l: subdev: Add helpers for format, crop and compose pointers

[Sakari Ailus]

Add a helper for obtaining format, crop and compose pointers. These are
convenient for drivers, independently of the driver uses streams or not.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 54 ++++++++++++++++++++++----
 include/media/v4l2-subdev.h           | 56 +++++++++++++++++++++++++++
 2 files changed, 102 insertions(+), 8 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index d295a4e87b66..854f9d4db923 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1573,19 +1573,57 @@ v4l2_subdev_init_stream_configs(struct v4l2_subdev_stream_configs *stream_config
 	return 0;
 }
 
+struct v4l2_mbus_framefmt
+*v4l2_subdev_get_fmt_ptr(struct v4l2_subdev *sd,
+			 struct v4l2_subdev_state *state, unsigned int pad,
+			 unsigned int stream)
+{
+	if (sd->flags & V4L2_SUBDEV_FL_STREAMS)
+		return v4l2_subdev_state_get_stream_format(state, pad, stream);
+
+	if (pad < sd->entity.num_pads && stream == 0)
+		return v4l2_subdev_get_pad_format(sd, state, pad);
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_get_fmt_ptr);
+
+struct v4l2_rect
+*v4l2_subdev_get_crop_ptr(struct v4l2_subdev *sd,
+			  struct v4l2_subdev_state *state, unsigned int pad,
+			  unsigned int stream)
+{
+	if (sd->flags & V4L2_SUBDEV_FL_STREAMS)
+		return v4l2_subdev_state_get_stream_crop(state, pad, stream);
+
+	if (pad < sd->entity.num_pads && stream == 0)
+		return v4l2_subdev_get_pad_crop(sd, state, pad);
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_get_crop_ptr);
+
+struct v4l2_rect
+*v4l2_subdev_get_compose_ptr(struct v4l2_subdev *sd,
+			     struct v4l2_subdev_state *state, unsigned int pad,
+			     unsigned int stream)
+{
+	if (sd->flags & V4L2_SUBDEV_FL_STREAMS)
+		return v4l2_subdev_state_get_stream_compose(state, pad, stream);
+
+	if (pad < sd->entity.num_pads && stream == 0)
+		return v4l2_subdev_get_pad_compose(sd, state, pad);
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_get_compose_ptr);
+
 int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
 			struct v4l2_subdev_format *format)
 {
 	struct v4l2_mbus_framefmt *fmt;
 
-	if (sd->flags & V4L2_SUBDEV_FL_STREAMS)
-		fmt = v4l2_subdev_state_get_stream_format(state, format->pad,
-							  format->stream);
-	else if (format->pad < sd->entity.num_pads && format->stream == 0)
-		fmt = v4l2_subdev_get_pad_format(sd, state, format->pad);
-	else
-		fmt = NULL;
-
+	fmt = v4l2_subdev_get_fmt_ptr(sd, state, format->pad, format->stream);
 	if (!fmt)
 		return -EINVAL;
 
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 5f59ff0796b7..7c34243ffed9 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1479,6 +1479,62 @@ v4l2_subdev_lock_and_get_active_state(struct v4l2_subdev *sd)
 
 #if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
 
+/**
+ * v4l2_subdev_get_fmt_ptr - Obtain a pointer to V4L2 sub-device format for pad
+ *			     and stream
+ * @sd: subdevice
+ * @state: subdevice state
+ * @pad: the pad on the sub-device
+ * @stream: stream in the pad
+ *
+ * For given pad and stream, obtain a pointer to the mbus format from the
+ * sub-device.
+ *
+ * Returns NULL if the format is not found or the parameters are invalid.
+ */
+struct v4l2_mbus_framefmt *
+v4l2_subdev_get_fmt_ptr(struct v4l2_subdev *sd,
+			struct v4l2_subdev_state *state, unsigned int pad,
+			unsigned int stream);
+
+/**
+ * v4l2_subdev_get_crop_ptr - Obtain a pointer to V4L2 sub-device crop
+ *			      rectangle for pad and stream
+ * @sd: subdevice
+ * @state: subdevice state
+ * @pad: the pad on the sub-device
+ * @stream: stream in the pad
+ *
+ * For given pad and stream, obtain a pointer to the crop selection rectangle
+ * from the sub-device.
+ *
+ * Returns NULL if the selection rectangle is not found or the parameters are
+ * invalid.
+ */
+struct v4l2_rect *
+v4l2_subdev_get_crop_ptr(struct v4l2_subdev *sd,
+			 struct v4l2_subdev_state *state, unsigned int pad,
+			 unsigned int stream);
+
+/**
+ * v4l2_subdev_get_compose_ptr - Obtain a pointer to V4L2 sub-device compose
+ *				 rectangle for pad and stream
+ * @sd: subdevice
+ * @state: subdevice state
+ * @pad: the pad on the sub-device
+ * @stream: stream in the pad
+ *
+ * For given pad and stream, obtain a pointer to the compose selection rectangle
+ * from the sub-device.
+ *
+ * Returns NULL if the selection rectangle is not found or the parameters are
+ * invalid.
+ */
+struct v4l2_rect *
+v4l2_subdev_get_compose_ptr(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_state *state, unsigned int pad,
+			    unsigned int stream);
+
 /**
  * v4l2_subdev_get_fmt() - Fill format based on state
  * @sd: subdevice
-- 
2.39.2

[PATCH v6 13/28] media: v4l: subdev: Add a function to lock two sub-device states, use it

[Sakari Ailus]

Add two new functions, v4l2_subdev_lock_states() and
v4l2_subdev_unclock_states(), to acquire and release the state of two
sub-devices. They differ from calling v4l2_subdev_{un,}lock_state() so
that if the two states share the same lock, the lock is acquired only
once.

Also use the new functions in v4l2_subdev_link_validate().

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 12 +++-----
 include/media/v4l2-subdev.h           | 40 +++++++++++++++++++++++++++
 2 files changed, 44 insertions(+), 8 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 854f9d4db923..df9a1ae65410 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1377,17 +1377,13 @@ int v4l2_subdev_link_validate(struct media_link *link)
 
 	states_locked = sink_state && source_state;
 
-	if (states_locked) {
-		v4l2_subdev_lock_state(sink_state);
-		v4l2_subdev_lock_state(source_state);
-	}
+	if (states_locked)
+		v4l2_subdev_lock_states(sink_state, source_state);
 
 	ret = v4l2_subdev_link_validate_locked(link, states_locked);
 
-	if (states_locked) {
-		v4l2_subdev_unlock_state(sink_state);
-		v4l2_subdev_unlock_state(source_state);
-	}
+	if (states_locked)
+		v4l2_subdev_unlock_states(sink_state, source_state);
 
 	return ret;
 }
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 7c34243ffed9..e49e8af2fb52 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1418,6 +1418,46 @@ static inline void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state)
 	mutex_unlock(state->lock);
 }
 
+/**
+ * v4l2_subdev_lock_states - Lock two sub-device states
+ * @state1: One subdevice state
+ * @state2: The other subdevice state
+ *
+ * Locks the state of two sub-devices.
+ *
+ * The states must be unlocked with v4l2_subdev_unlock_states() after use.
+ *
+ * This differs from calling v4l2_subdev_lock_state() on both states so that if
+ * the states share the same lock, the lock is acquired only once (so no
+ * deadlock occurs). Note that it must be ensured the locks must always be
+ * acquired in the same order.
+ */
+static inline void v4l2_subdev_lock_states(struct v4l2_subdev_state *state1,
+					   struct v4l2_subdev_state *state2)
+{
+	mutex_lock(state1->lock);
+	if (state1->lock != state2->lock)
+		mutex_lock(state2->lock);
+}
+
+/**
+ * v4l2_subdev_unlock_states() - Unlock two sub-device states
+ * @state1: One subdevice state
+ * @state2: The other subdevice state
+ *
+ * Unlocks the state of two sub-devices.
+ *
+ * This differs from calling v4l2_subdev_unlock_state() on both states so that if
+ * the states share the same lock, the lock is released only once.
+ */
+static inline void v4l2_subdev_unlock_states(struct v4l2_subdev_state *state1,
+					     struct v4l2_subdev_state *state2)
+{
+	mutex_unlock(state1->lock);
+	if (state1->lock != state2->lock)
+		mutex_unlock(state2->lock);
+}
+
 /**
  * v4l2_subdev_get_unlocked_active_state() - Checks that the active subdev state
  *					     is unlocked and returns it
-- 
2.39.2

[PATCH v6 14/28] media: v4l: subdev: Move G_ROUTING handling below S_ROUTING

[Sakari Ailus]

Move G_ROUTING IOCTL handling below that of S_ROUTING. G_ROUTING
implementation will soon needed in handling S_ROUTING as well.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 54 +++++++++++++--------------
 1 file changed, 27 insertions(+), 27 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index df9a1ae65410..614ff0031831 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -887,33 +887,6 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 	case VIDIOC_SUBDEV_QUERYSTD:
 		return v4l2_subdev_call(sd, video, querystd, arg);
 
-	case VIDIOC_SUBDEV_G_ROUTING: {
-		struct v4l2_subdev_routing *routing = arg;
-		struct v4l2_subdev_krouting *krouting;
-
-		if (!v4l2_subdev_enable_streams_api)
-			return -ENOIOCTLCMD;
-
-		if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS))
-			return -ENOIOCTLCMD;
-
-		memset(routing->reserved, 0, sizeof(routing->reserved));
-
-		krouting = &state->routing;
-
-		if (routing->num_routes < krouting->num_routes) {
-			routing->num_routes = krouting->num_routes;
-			return -ENOSPC;
-		}
-
-		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
-		       krouting->routes,
-		       krouting->num_routes * sizeof(*krouting->routes));
-		routing->num_routes = krouting->num_routes;
-
-		return 0;
-	}
-
 	case VIDIOC_SUBDEV_S_ROUTING: {
 		struct v4l2_subdev_routing *routing = arg;
 		struct v4l2_subdev_route *routes =
@@ -962,6 +935,33 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 					routing->which, &krouting);
 	}
 
+	case VIDIOC_SUBDEV_G_ROUTING: {
+		struct v4l2_subdev_routing *routing = arg;
+		struct v4l2_subdev_krouting *krouting;
+
+		if (!v4l2_subdev_enable_streams_api)
+			return -ENOIOCTLCMD;
+
+		if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS))
+			return -ENOIOCTLCMD;
+
+		memset(routing->reserved, 0, sizeof(routing->reserved));
+
+		krouting = &state->routing;
+
+		if (routing->num_routes < krouting->num_routes) {
+			routing->num_routes = krouting->num_routes;
+			return -ENOSPC;
+		}
+
+		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
+		       krouting->routes,
+		       krouting->num_routes * sizeof(*krouting->routes));
+		routing->num_routes = krouting->num_routes;
+
+		return 0;
+	}
+
 	case VIDIOC_SUBDEV_G_CLIENT_CAP: {
 		struct v4l2_subdev_client_capability *client_cap = arg;
 
-- 
2.39.2

[PATCH v6 15/28] media: v4l: subdev: Copy argument back to user also for S_ROUTING

[Sakari Ailus]

As the user needs to know what went wrong for S_ROUTING, copy array
arguments back to the user.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/v4l2-core/v4l2-ioctl.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index fb453b7d0c91..6921a72566df 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -3419,7 +3419,8 @@ video_usercopy(struct file *file, unsigned int orig_cmd, unsigned long arg,
 	 * in case of failure, but it is not defined here as part of the
 	 * 'v4l2_ioctls' array, insert an ad-hoc check to address that.
 	 */
-	if (err < 0 && !always_copy && cmd != VIDIOC_SUBDEV_G_ROUTING)
+	if (err < 0 && !always_copy && cmd != VIDIOC_SUBDEV_G_ROUTING &&
+	    cmd != VIDIOC_SUBDEV_S_ROUTING)
 		goto out;
 
 	if (has_array_args) {
-- 
2.39.2

[PATCH v6 16/28] media: v4l: subdev: Add len_routes field to struct v4l2_subdev_routing

[Sakari Ailus]

The len_routes field is used to tell the size of the routes array in
struct v4l2_subdev_routing. This way the number of routes returned from
S_ROUTING IOCTL may be larger than the number of routes provided, in case
there are more routes returned by the driver.

Note that this changes the (now-disabled) UAPI, users must be updated.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../media/v4l/vidioc-subdev-g-routing.rst     | 31 ++++++++++++-------
 drivers/media/v4l2-core/v4l2-ioctl.c          |  4 +--
 drivers/media/v4l2-core/v4l2-subdev.c         |  6 +++-
 include/media/v4l2-subdev.h                   |  2 ++
 include/uapi/linux/v4l2-subdev.h              |  8 +++--
 5 files changed, 34 insertions(+), 17 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
index 72677a280cd6..9a9765ddc316 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
@@ -46,20 +46,26 @@ with the ``VIDIOC_SUBDEV_S_ROUTING`` ioctl, by adding or removing routes and
 setting or clearing flags of the  ``flags`` field of a
 struct :c:type:`v4l2_subdev_route`.
 
-All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called. This
-means that the userspace must reconfigure all streams after calling the ioctl
-with e.g. ``VIDIOC_SUBDEV_S_FMT``.
+All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is
+called. This means that the userspace must reconfigure all streams after calling
+the ioctl with e.g. ``VIDIOC_SUBDEV_S_FMT``.
 
 Only subdevices which have both sink and source pads can support routing.
 
-When inspecting routes through ``VIDIOC_SUBDEV_G_ROUTING`` and the application
-provided ``num_routes`` is not big enough to contain all the available routes
-the subdevice exposes, drivers return the ENOSPC error code and adjust the
-value of the ``num_routes`` field. Application should then reserve enough memory
-for all the route entries and call ``VIDIOC_SUBDEV_G_ROUTING`` again.
+The ``num_routes`` field is used to denote the number of routes set (set by user
+space on ``VIDIOC_SUBDEV_S_ROUTING`` argument) on the routing table as well as
+the number of routes returned back from both IOCTLs. The ``len_routes``
+signifies the number of routes that can fit into the ``routes`` array. The
+userspace shall set ``len_routes`` for both IOCTLs and ``num_routes`` for
+``VIDIOC_SUBDEV_S_ROUTING``.
 
-On a successful ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the
-``num_routes`` field to reflect the actual number of routes returned.
+On a ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the ``num_routes``
+field to reflect the actual number of routes known by the driver. ``num_routes``
+larger than ``len_routes`` in both IOCTLs. In this ``len_routes`` were returned
+back to the userspace. This is not an error.
+
+Also ``VIDIOC_SUBDEV_S_ROUTING`` may return more route than the user provided in
+``num_routes`` field due to e.g. hardware properties.
 
 .. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
 
@@ -74,6 +80,9 @@ On a successful ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the
       - ``which``
       - Format to modified, from enum
         :ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
+    * - __u32
+      - ``len_routes``
+      - The length of the array (as in memory reserved for the array)
     * - struct :c:type:`v4l2_subdev_route`
       - ``routes[]``
       - Array of struct :c:type:`v4l2_subdev_route` entries
@@ -81,7 +90,7 @@ On a successful ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the
       - ``num_routes``
       - Number of entries of the routes array
     * - __u32
-      - ``reserved``\ [5]
+      - ``reserved``\ [11]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index 6921a72566df..1e3da9d64958 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -3155,13 +3155,13 @@ static int check_array_args(unsigned int cmd, void *parg, size_t *array_size,
 	case VIDIOC_SUBDEV_S_ROUTING: {
 		struct v4l2_subdev_routing *routing = parg;
 
-		if (routing->num_routes > 256)
+		if (routing->len_routes > 256)
 			return -E2BIG;
 
 		*user_ptr = u64_to_user_ptr(routing->routes);
 		*kernel_ptr = (void **)&routing->routes;
 		*array_size = sizeof(struct v4l2_subdev_route)
-			    * routing->num_routes;
+			    * routing->len_routes;
 		ret = 1;
 		break;
 	}
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 614ff0031831..bd1e8205913c 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -903,6 +903,9 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 		if (routing->which != V4L2_SUBDEV_FORMAT_TRY && ro_subdev)
 			return -EPERM;
 
+		if (routing->num_routes > routing->len_routes)
+			return -EINVAL;
+
 		memset(routing->reserved, 0, sizeof(routing->reserved));
 
 		for (i = 0; i < routing->num_routes; ++i) {
@@ -929,6 +932,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 		}
 
 		krouting.num_routes = routing->num_routes;
+		krouting.len_routes = routing->len_routes;
 		krouting.routes = routes;
 
 		return v4l2_subdev_call(sd, pad, set_routing, state,
@@ -949,7 +953,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 
 		krouting = &state->routing;
 
-		if (routing->num_routes < krouting->num_routes) {
+		if (routing->len_routes < krouting->num_routes) {
 			routing->num_routes = krouting->num_routes;
 			return -ENOSPC;
 		}
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index e49e8af2fb52..baaa81a9497e 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -741,12 +741,14 @@ struct v4l2_subdev_stream_configs {
 /**
  * struct v4l2_subdev_krouting - subdev routing table
  *
+ * @len_routes: length of routes array, in routes
  * @num_routes: number of routes
  * @routes: &struct v4l2_subdev_route
  *
  * This structure contains the routing table for a subdev.
  */
 struct v4l2_subdev_krouting {
+	unsigned int len_routes;
 	unsigned int num_routes;
 	struct v4l2_subdev_route *routes;
 };
diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
index 4a195b68f28f..b57fb89caa9e 100644
--- a/include/uapi/linux/v4l2-subdev.h
+++ b/include/uapi/linux/v4l2-subdev.h
@@ -222,15 +222,17 @@ struct v4l2_subdev_route {
  * struct v4l2_subdev_routing - Subdev routing information
  *
  * @which: configuration type (from enum v4l2_subdev_format_whence)
- * @num_routes: the total number of routes in the routes array
+ * @len_routes: the length of the routes array, in routes
  * @routes: pointer to the routes array
+ * @num_routes: the total number of routes in the routes array
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_routing {
 	__u32 which;
-	__u32 num_routes;
+	__u32 len_routes;
 	__u64 routes;
-	__u32 reserved[6];
+	__u32 num_routes;
+	__u32 reserved[11];
 };
 
 /*
-- 
2.39.2

[PATCH v6 17/28] media: v4l: subdev: Return routes set using S_ROUTING

[Sakari Ailus]

Return the routes set using S_ROUTING back to the user. Also reflect this
in documentation.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../userspace-api/media/v4l/vidioc-subdev-g-routing.rst      | 5 +++--
 drivers/media/v4l2-core/v4l2-subdev.c                        | 5 ++++-
 2 files changed, 7 insertions(+), 3 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
index 9a9765ddc316..ced53ea5f23c 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
@@ -43,8 +43,9 @@ The routing configuration determines the flows of data inside an entity.
 Drivers report their current routing tables using the
 ``VIDIOC_SUBDEV_G_ROUTING`` ioctl and application may enable or disable routes
 with the ``VIDIOC_SUBDEV_S_ROUTING`` ioctl, by adding or removing routes and
-setting or clearing flags of the  ``flags`` field of a
-struct :c:type:`v4l2_subdev_route`.
+setting or clearing flags of the ``flags`` field of a struct
+:c:type:`v4l2_subdev_route`. Similarly to ``VIDIOC_SUBDEV_G_ROUTING``, also
+``VIDIOC_SUBDEV_S_ROUTING`` returns the routes back to the user.
 
 All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is
 called. This means that the userspace must reconfigure all streams after calling
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index bd1e8205913c..9a34e13dfd96 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -935,9 +935,12 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 		krouting.len_routes = routing->len_routes;
 		krouting.routes = routes;
 
-		return v4l2_subdev_call(sd, pad, set_routing, state,
+		rval = v4l2_subdev_call(sd, pad, set_routing, state,
 					routing->which, &krouting);
+		if (rval < 0)
+			return rval;
 	}
+		fallthrough;
 
 	case VIDIOC_SUBDEV_G_ROUTING: {
 		struct v4l2_subdev_routing *routing = arg;
-- 
2.39.2

[PATCH v6 18/28] media: uapi: Allow a larger number of routes than there's room for

[Sakari Ailus]

On VIDIOC_SUBDEV_[GS]_ROUTING, only return as many routes back to the user
as there's room. Do not consider it an error if more routes existed.
Simply inform the user there are more routes.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../userspace-api/media/v4l/vidioc-subdev-g-routing.rst   | 4 ----
 drivers/media/v4l2-core/v4l2-subdev.c                     | 8 ++------
 2 files changed, 2 insertions(+), 10 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
index ced53ea5f23c..99d3c15fd759 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
@@ -145,10 +145,6 @@ On success 0 is returned, on error -1 and the ``errno`` variable is set
 appropriately. The generic error codes are described at the
 :ref:`Generic Error Codes <gen-errors>` chapter.
 
-ENOSPC
-   The application provided ``num_routes`` is not big enough to contain
-   all the available routes the subdevice exposes.
-
 EINVAL
    The sink or source pad identifiers reference a non-existing pad, or reference
    pads of different types (ie. the sink_pad identifiers refers to a source pad).
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 9a34e13dfd96..dd48e7e549fb 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -956,14 +956,10 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 
 		krouting = &state->routing;
 
-		if (routing->len_routes < krouting->num_routes) {
-			routing->num_routes = krouting->num_routes;
-			return -ENOSPC;
-		}
-
 		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
 		       krouting->routes,
-		       krouting->num_routes * sizeof(*krouting->routes));
+		       min(krouting->num_routes, routing->len_routes) *
+		       sizeof(*krouting->routes));
 		routing->num_routes = krouting->num_routes;
 
 		return 0;
-- 
2.39.2

[PATCH v6 19/28] media: v4l: subdev: Add trivial set_routing support

[Sakari Ailus]

Add trivial S_ROUTING IOCTL support for drivers where routing is static.
Essentially this means returning the same information G_ROUTING call would
have done.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index dd48e7e549fb..7d7028de581a 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -894,6 +894,9 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 		struct v4l2_subdev_krouting krouting = {};
 		unsigned int i;
 
+		if (!v4l2_subdev_has_op(sd, pad, set_routing))
+			goto do_vidioc_subdev_g_routing;
+
 		if (!v4l2_subdev_enable_streams_api)
 			return -ENOIOCTLCMD;
 
@@ -939,6 +942,8 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 					routing->which, &krouting);
 		if (rval < 0)
 			return rval;
+do_vidioc_subdev_g_routing:
+		;
 	}
 		fallthrough;
 
-- 
2.39.2

[PATCH v6 20/28] media: uapi: v4l: subdev: Enable streams API

[Sakari Ailus]

Remove v4l2_subdev_enable_streams_api variable that was used to easily
enable streams API for development, and conditions that use the variable.

This patch enables the streams API for V4L2 sub-device interface which
allows transporting multiple streams on a single MC link.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 23 -----------------------
 1 file changed, 23 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 7d7028de581a..5e24a638bdba 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -26,15 +26,6 @@
 #include <media/v4l2-fh.h>
 #include <media/v4l2-ioctl.h>
 
-#if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
-/*
- * The Streams API is an experimental feature. To use the Streams API, set
- * 'v4l2_subdev_enable_streams_api' to 1 below.
- */
-
-static bool v4l2_subdev_enable_streams_api;
-#endif
-
 /*
  * Maximum stream ID is 63 for now, as we use u64 bitmask to represent a set
  * of streams.
@@ -897,9 +888,6 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 		if (!v4l2_subdev_has_op(sd, pad, set_routing))
 			goto do_vidioc_subdev_g_routing;
 
-		if (!v4l2_subdev_enable_streams_api)
-			return -ENOIOCTLCMD;
-
 		if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS))
 			return -ENOIOCTLCMD;
 
@@ -951,9 +939,6 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 		struct v4l2_subdev_routing *routing = arg;
 		struct v4l2_subdev_krouting *krouting;
 
-		if (!v4l2_subdev_enable_streams_api)
-			return -ENOIOCTLCMD;
-
 		if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS))
 			return -ENOIOCTLCMD;
 
@@ -981,14 +966,6 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 	case VIDIOC_SUBDEV_S_CLIENT_CAP: {
 		struct v4l2_subdev_client_capability *client_cap = arg;
 
-		/*
-		 * Clear V4L2_SUBDEV_CLIENT_CAP_STREAMS if streams API is not
-		 * enabled. Remove this when streams API is no longer
-		 * experimental.
-		 */
-		if (!v4l2_subdev_enable_streams_api)
-			client_cap->capabilities &= ~V4L2_SUBDEV_CLIENT_CAP_STREAMS;
-
 		/* Filter out unsupported capabilities */
 		client_cap->capabilities &= V4L2_SUBDEV_CLIENT_CAP_STREAMS;
 
-- 
2.39.2

[PATCH v6 21/28] media: ccs: No need to set streaming to false in power off

[Sakari Ailus]

Streaming will have been stopped by the sensor is powered off, and so
sensor->streaming is also false already. Do not set it as part of the
runtime suspend callback.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ccs/ccs-core.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
index 2abfd5932e02..0c518145b2c8 100644
--- a/drivers/media/i2c/ccs/ccs-core.c
+++ b/drivers/media/i2c/ccs/ccs-core.c
@@ -1700,7 +1700,6 @@ static int ccs_power_off(struct device *dev)
 	usleep_range(5000, 5000);
 	regulator_bulk_disable(ARRAY_SIZE(ccs_regulators),
 			       sensor->regulators);
-	sensor->streaming = false;
 
 	return 0;
 }
-- 
2.39.2

[PATCH v6 22/28] media: ccs: Use {enable,disable}_streams operations

[Sakari Ailus]

Switch from s_stream() video op to enable_streams() and disable_streams()
pad operations. They are preferred and required for streams support.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ccs/ccs-core.c | 183 +++++++++++++++----------------
 1 file changed, 86 insertions(+), 97 deletions(-)

diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
index 0c518145b2c8..70e7582bbf5f 100644
--- a/drivers/media/i2c/ccs/ccs-core.c
+++ b/drivers/media/i2c/ccs/ccs-core.c
@@ -1705,22 +1705,64 @@ static int ccs_power_off(struct device *dev)
 }
 
 /* -----------------------------------------------------------------------------
- * Video stream management
+ * V4L2 subdev video operations
  */
 
-static int ccs_start_streaming(struct ccs_sensor *sensor)
+static int ccs_pm_get_init(struct ccs_sensor *sensor)
 {
+	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
+	int rval;
+
+	/*
+	 * It can't use pm_runtime_resume_and_get() here, as the driver
+	 * relies at the returned value to detect if the device was already
+	 * active or not.
+	 */
+	rval = pm_runtime_get_sync(&client->dev);
+	if (rval < 0)
+		goto error;
+
+	/* Device was already active, so don't set controls */
+	if (rval == 1)
+		return 0;
+
+	/* Restore V4L2 controls to the previously suspended device */
+	rval = __v4l2_ctrl_handler_setup(&sensor->pixel_array->ctrl_handler);
+	if (rval)
+		goto error;
+
+	rval = __v4l2_ctrl_handler_setup(&sensor->src->ctrl_handler);
+	if (rval)
+		goto error;
+
+	/* Keep PM runtime usage_count incremented on success */
+	return 0;
+error:
+	pm_runtime_put(&client->dev);
+	return rval;
+}
+
+static int ccs_enable_streams(struct v4l2_subdev *subdev,
+			      struct v4l2_subdev_state *state, u32 pad,
+			      u64 streams_mask)
+{
+	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
 	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
 	unsigned int binning_mode;
 	int rval;
 
-	mutex_lock(&sensor->mutex);
+	if (pad != CCS_PAD_SRC)
+		return -EINVAL;
+
+	rval = ccs_pm_get_init(sensor);
+	if (rval)
+		return rval;
 
 	rval = ccs_write(sensor, CSI_DATA_FORMAT,
 			 (sensor->csi_format->width << 8) |
 			 sensor->csi_format->compressed);
 	if (rval)
-		goto out;
+		goto err_pm_put;
 
 	/* Binning configuration */
 	if (sensor->binning_horizontal == 1 &&
@@ -1733,38 +1775,38 @@ static int ccs_start_streaming(struct ccs_sensor *sensor)
 
 		rval = ccs_write(sensor, BINNING_TYPE, binning_type);
 		if (rval < 0)
-			goto out;
+			goto err_pm_put;
 
 		binning_mode = 1;
 	}
 	rval = ccs_write(sensor, BINNING_MODE, binning_mode);
 	if (rval < 0)
-		goto out;
+		goto err_pm_put;
 
 	/* Set up PLL */
 	rval = ccs_pll_configure(sensor);
 	if (rval)
-		goto out;
+		goto err_pm_put;
 
 	/* Analog crop start coordinates */
 	rval = ccs_write(sensor, X_ADDR_START, sensor->pa_src.left);
 	if (rval < 0)
-		goto out;
+		goto err_pm_put;
 
 	rval = ccs_write(sensor, Y_ADDR_START, sensor->pa_src.top);
 	if (rval < 0)
-		goto out;
+		goto err_pm_put;
 
 	/* Analog crop end coordinates */
 	rval = ccs_write(sensor, X_ADDR_END,
 			 sensor->pa_src.left + sensor->pa_src.width - 1);
 	if (rval < 0)
-		goto out;
+		goto err_pm_put;
 
 	rval = ccs_write(sensor, Y_ADDR_END,
 			 sensor->pa_src.top + sensor->pa_src.height - 1);
 	if (rval < 0)
-		goto out;
+		goto err_pm_put;
 
 	/*
 	 * Output from pixel array, including blanking, is set using
@@ -1777,22 +1819,22 @@ static int ccs_start_streaming(struct ccs_sensor *sensor)
 		rval = ccs_write(sensor, DIGITAL_CROP_X_OFFSET,
 				 sensor->scaler_sink.left);
 		if (rval < 0)
-			goto out;
+			goto err_pm_put;
 
 		rval = ccs_write(sensor, DIGITAL_CROP_Y_OFFSET,
 				 sensor->scaler_sink.top);
 		if (rval < 0)
-			goto out;
+			goto err_pm_put;
 
 		rval = ccs_write(sensor, DIGITAL_CROP_IMAGE_WIDTH,
 				 sensor->scaler_sink.width);
 		if (rval < 0)
-			goto out;
+			goto err_pm_put;
 
 		rval = ccs_write(sensor, DIGITAL_CROP_IMAGE_HEIGHT,
 				 sensor->scaler_sink.height);
 		if (rval < 0)
-			goto out;
+			goto err_pm_put;
 	}
 
 	/* Scaling */
@@ -1800,20 +1842,20 @@ static int ccs_start_streaming(struct ccs_sensor *sensor)
 	    != CCS_SCALING_CAPABILITY_NONE) {
 		rval = ccs_write(sensor, SCALING_MODE, sensor->scaling_mode);
 		if (rval < 0)
-			goto out;
+			goto err_pm_put;
 
 		rval = ccs_write(sensor, SCALE_M, sensor->scale_m);
 		if (rval < 0)
-			goto out;
+			goto err_pm_put;
 	}
 
 	/* Output size from sensor */
 	rval = ccs_write(sensor, X_OUTPUT_SIZE, sensor->src_src.width);
 	if (rval < 0)
-		goto out;
+		goto err_pm_put;
 	rval = ccs_write(sensor, Y_OUTPUT_SIZE, sensor->src_src.height);
 	if (rval < 0)
-		goto out;
+		goto err_pm_put;
 
 	if (CCS_LIM(sensor, FLASH_MODE_CAPABILITY) &
 	    (CCS_FLASH_MODE_CAPABILITY_SINGLE_STROBE |
@@ -1822,109 +1864,52 @@ static int ccs_start_streaming(struct ccs_sensor *sensor)
 	    sensor->hwcfg.strobe_setup->trigger != 0) {
 		rval = ccs_setup_flash_strobe(sensor);
 		if (rval)
-			goto out;
+			goto err_pm_put;
 	}
 
 	rval = ccs_call_quirk(sensor, pre_streamon);
 	if (rval) {
 		dev_err(&client->dev, "pre_streamon quirks failed\n");
-		goto out;
+		goto err_pm_put;
 	}
 
 	rval = ccs_write(sensor, MODE_SELECT, CCS_MODE_SELECT_STREAMING);
 
-out:
-	mutex_unlock(&sensor->mutex);
-
-	return rval;
-}
-
-static int ccs_stop_streaming(struct ccs_sensor *sensor)
-{
-	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
-	int rval;
-
-	mutex_lock(&sensor->mutex);
-	rval = ccs_write(sensor, MODE_SELECT, CCS_MODE_SELECT_SOFTWARE_STANDBY);
-	if (rval)
-		goto out;
-
-	rval = ccs_call_quirk(sensor, post_streamoff);
-	if (rval)
-		dev_err(&client->dev, "post_streamoff quirks failed\n");
-
-out:
-	mutex_unlock(&sensor->mutex);
-	return rval;
-}
-
-/* -----------------------------------------------------------------------------
- * V4L2 subdev video operations
- */
-
-static int ccs_pm_get_init(struct ccs_sensor *sensor)
-{
-	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
-	int rval;
-
-	/*
-	 * It can't use pm_runtime_resume_and_get() here, as the driver
-	 * relies at the returned value to detect if the device was already
-	 * active or not.
-	 */
-	rval = pm_runtime_get_sync(&client->dev);
-	if (rval < 0)
-		goto error;
-
-	/* Device was already active, so don't set controls */
-	if (rval == 1)
-		return 0;
+	sensor->streaming = true;
 
-	/* Restore V4L2 controls to the previously suspended device */
-	rval = v4l2_ctrl_handler_setup(&sensor->pixel_array->ctrl_handler);
-	if (rval)
-		goto error;
+	return 0;
 
-	rval = v4l2_ctrl_handler_setup(&sensor->src->ctrl_handler);
-	if (rval)
-		goto error;
+err_pm_put:
+	pm_runtime_mark_last_busy(&client->dev);
+	pm_runtime_put_autosuspend(&client->dev);
 
-	/* Keep PM runtime usage_count incremented on success */
-	return 0;
-error:
-	pm_runtime_put(&client->dev);
 	return rval;
 }
 
-static int ccs_set_stream(struct v4l2_subdev *subdev, int enable)
+static int ccs_disable_streams(struct v4l2_subdev *subdev,
+			       struct v4l2_subdev_state *state, u32 pad,
+			       u64 streams_mask)
 {
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
 	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
 	int rval;
 
-	if (!enable) {
-		ccs_stop_streaming(sensor);
-		sensor->streaming = false;
-		pm_runtime_mark_last_busy(&client->dev);
-		pm_runtime_put_autosuspend(&client->dev);
-
-		return 0;
-	}
+	if (pad != CCS_PAD_SRC)
+		return -EINVAL;
 
-	rval = ccs_pm_get_init(sensor);
+	rval = ccs_write(sensor, MODE_SELECT, CCS_MODE_SELECT_SOFTWARE_STANDBY);
 	if (rval)
 		return rval;
 
-	sensor->streaming = true;
+	rval = ccs_call_quirk(sensor, post_streamoff);
+	if (rval)
+		dev_err(&client->dev, "post_streamoff quirks failed\n");
 
-	rval = ccs_start_streaming(sensor);
-	if (rval < 0) {
-		sensor->streaming = false;
-		pm_runtime_mark_last_busy(&client->dev);
-		pm_runtime_put_autosuspend(&client->dev);
-	}
+	sensor->streaming = false;
+	pm_runtime_mark_last_busy(&client->dev);
+	pm_runtime_put_autosuspend(&client->dev);
 
-	return rval;
+	return 0;
 }
 
 static int ccs_pre_streamon(struct v4l2_subdev *subdev, u32 flags)
@@ -1950,7 +1935,9 @@ static int ccs_pre_streamon(struct v4l2_subdev *subdev, u32 flags)
 		}
 	}
 
+	mutex_lock(&sensor->mutex);
 	rval = ccs_pm_get_init(sensor);
+	mutex_unlock(&sensor->mutex);
 	if (rval)
 		return rval;
 
@@ -3033,7 +3020,7 @@ static int ccs_init_cfg(struct v4l2_subdev *sd,
 }
 
 static const struct v4l2_subdev_video_ops ccs_video_ops = {
-	.s_stream = ccs_set_stream,
+	.s_stream = v4l2_subdev_s_stream_helper,
 	.pre_streamon = ccs_pre_streamon,
 	.post_streamoff = ccs_post_streamoff,
 };
@@ -3045,6 +3032,8 @@ static const struct v4l2_subdev_pad_ops ccs_pad_ops = {
 	.set_fmt = ccs_set_format,
 	.get_selection = ccs_get_selection,
 	.set_selection = ccs_set_selection,
+	.enable_streams = ccs_enable_streams,
+	.disable_streams = ccs_disable_streams,
 };
 
 static const struct v4l2_subdev_sensor_ops ccs_sensor_ops = {
-- 
2.39.2

[PATCH v6 23/28] media: ccs: Track streaming state

[Sakari Ailus]

With enable_streams and disable_streams, the driver for a device where
streams are not independently started and stopped needs to maintain state
information on streams that have been requested to be started. Do that
now.

In the future, a helper function in the framework is a desirable way to do
this instead.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ccs/ccs-core.c | 13 ++++++++++---
 drivers/media/i2c/ccs/ccs.h      |  2 +-
 2 files changed, 11 insertions(+), 4 deletions(-)

diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
index 70e7582bbf5f..27ecf8688658 100644
--- a/drivers/media/i2c/ccs/ccs-core.c
+++ b/drivers/media/i2c/ccs/ccs-core.c
@@ -1754,6 +1754,11 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
 	if (pad != CCS_PAD_SRC)
 		return -EINVAL;
 
+	if (sensor->streaming) {
+		sensor->streaming |= streams_mask;
+		return 0;
+	}
+
 	rval = ccs_pm_get_init(sensor);
 	if (rval)
 		return rval;
@@ -1875,7 +1880,7 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
 
 	rval = ccs_write(sensor, MODE_SELECT, CCS_MODE_SELECT_STREAMING);
 
-	sensor->streaming = true;
+	sensor->streaming |= streams_mask;
 
 	return 0;
 
@@ -1897,6 +1902,10 @@ static int ccs_disable_streams(struct v4l2_subdev *subdev,
 	if (pad != CCS_PAD_SRC)
 		return -EINVAL;
 
+	sensor->streaming &= ~streams_mask;
+	if (sensor->streaming)
+		return 0;
+
 	rval = ccs_write(sensor, MODE_SELECT, CCS_MODE_SELECT_SOFTWARE_STANDBY);
 	if (rval)
 		return rval;
@@ -1905,7 +1914,6 @@ static int ccs_disable_streams(struct v4l2_subdev *subdev,
 	if (rval)
 		dev_err(&client->dev, "post_streamoff quirks failed\n");
 
-	sensor->streaming = false;
 	pm_runtime_mark_last_busy(&client->dev);
 	pm_runtime_put_autosuspend(&client->dev);
 
@@ -3500,7 +3508,6 @@ static int ccs_probe(struct i2c_client *client)
 		goto out_cleanup;
 	}
 
-	sensor->streaming = false;
 	sensor->dev_init_done = true;
 
 	rval = ccs_write_msr_regs(sensor);
diff --git a/drivers/media/i2c/ccs/ccs.h b/drivers/media/i2c/ccs/ccs.h
index 9c3587b2fbe7..043a679e115c 100644
--- a/drivers/media/i2c/ccs/ccs.h
+++ b/drivers/media/i2c/ccs/ccs.h
@@ -234,7 +234,7 @@ struct ccs_sensor {
 	u16 image_start; /* image data start line */
 	u16 visible_pixel_start; /* start pixel of the visible image */
 
-	bool streaming;
+	u8 streaming;
 	bool dev_init_done;
 	u8 compressed_min_bpp;
 
-- 
2.39.2

[PATCH v6 24/28] media: ccs: Move ccs_validate_csi_data_format up

[Sakari Ailus]

ccs_validate_csi_data_format() will soon be needed elsewhere, above its
current location. Move it up.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ccs/ccs-core.c | 28 ++++++++++++++--------------
 1 file changed, 14 insertions(+), 14 deletions(-)

diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
index 27ecf8688658..4924ec460d81 100644
--- a/drivers/media/i2c/ccs/ccs-core.c
+++ b/drivers/media/i2c/ccs/ccs-core.c
@@ -1967,6 +1967,20 @@ static int ccs_post_streamoff(struct v4l2_subdev *subdev)
 	return pm_runtime_put(&client->dev);
 }
 
+static const struct ccs_csi_data_format
+*ccs_validate_csi_data_format(struct ccs_sensor *sensor, u32 code)
+{
+	unsigned int i;
+
+	for (i = 0; i < ARRAY_SIZE(ccs_csi_data_formats); i++) {
+		if (sensor->mbus_frame_fmts & (1 << i) &&
+		    ccs_csi_data_formats[i].code == code)
+			return &ccs_csi_data_formats[i];
+	}
+
+	return sensor->csi_format;
+}
+
 static int ccs_enum_mbus_code(struct v4l2_subdev *subdev,
 			      struct v4l2_subdev_state *sd_state,
 			      struct v4l2_subdev_mbus_code_enum *code)
@@ -2098,20 +2112,6 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
 	}
 }
 
-static const struct ccs_csi_data_format
-*ccs_validate_csi_data_format(struct ccs_sensor *sensor, u32 code)
-{
-	unsigned int i;
-
-	for (i = 0; i < ARRAY_SIZE(ccs_csi_data_formats); i++) {
-		if (sensor->mbus_frame_fmts & (1 << i) &&
-		    ccs_csi_data_formats[i].code == code)
-			return &ccs_csi_data_formats[i];
-	}
-
-	return sensor->csi_format;
-}
-
 static int ccs_set_format_source(struct v4l2_subdev *subdev,
 				 struct v4l2_subdev_state *sd_state,
 				 struct v4l2_subdev_format *fmt)
-- 
2.39.2

[PATCH v6 25/28] media: ccs: Support frame descriptors

[Sakari Ailus]

Provide information on the frame layout using frame descriptors.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ccs/ccs-core.c  | 60 +++++++++++++++++++++++++++++++
 drivers/media/i2c/ccs/ccs-quirk.h |  7 ++++
 drivers/media/i2c/ccs/ccs.h       |  4 +++
 3 files changed, 71 insertions(+)

diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
index 4924ec460d81..75abb18ae847 100644
--- a/drivers/media/i2c/ccs/ccs-core.c
+++ b/drivers/media/i2c/ccs/ccs-core.c
@@ -25,6 +25,7 @@
 #include <linux/slab.h>
 #include <linux/smiapp.h>
 #include <linux/v4l2-mediabus.h>
+#include <media/mipi-csi2.h>
 #include <media/v4l2-fwnode.h>
 #include <media/v4l2-device.h>
 #include <uapi/linux/ccs.h>
@@ -235,6 +236,33 @@ static int ccs_read_all_limits(struct ccs_sensor *sensor)
 	return ret;
 }
 
+static u8 ccs_mipi_csi2_data_type(unsigned int bpp)
+{
+	switch (bpp) {
+	case 6:
+		return MIPI_CSI2_DT_RAW6;
+	case 7:
+		return MIPI_CSI2_DT_RAW7;
+	case 8:
+		return MIPI_CSI2_DT_RAW8;
+	case 10:
+		return MIPI_CSI2_DT_RAW10;
+	case 12:
+		return MIPI_CSI2_DT_RAW12;
+	case 14:
+		return MIPI_CSI2_DT_RAW14;
+	case 16:
+		return MIPI_CSI2_DT_RAW16;
+	case 20:
+		return MIPI_CSI2_DT_RAW20;
+	case 24:
+		return MIPI_CSI2_DT_RAW24;
+	default:
+		WARN_ON(1);
+		return 0;
+	}
+}
+
 static int ccs_read_frame_fmt(struct ccs_sensor *sensor)
 {
 	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
@@ -2616,6 +2644,37 @@ static int ccs_set_selection(struct v4l2_subdev *subdev,
 	return ret;
 }
 
+static int ccs_get_frame_desc(struct v4l2_subdev *subdev, unsigned int pad,
+				 struct v4l2_mbus_frame_desc *desc)
+{
+	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
+	struct v4l2_mbus_frame_desc_entry *entry = desc->entry;
+
+	if (ccs_has_quirk(sensor, frame_desc))
+		return ccs_call_quirk(sensor, frame_desc, desc);
+
+	switch (sensor->hwcfg.csi_signalling_mode) {
+	case CCS_CSI_SIGNALING_MODE_CSI_2_DPHY:
+	case CCS_CSI_SIGNALING_MODE_CSI_2_CPHY:
+		desc->type = V4L2_MBUS_FRAME_DESC_TYPE_CSI2;
+		break;
+	default:
+		/* FIXME: CCP2 support */
+		return -EINVAL;
+	}
+
+	entry->pixelcode = sensor->csi_format->code;
+	entry->stream = CCS_STREAM_PIXEL;
+	entry->bus.csi2.dt =
+		sensor->csi_format->width == sensor->csi_format->compressed ?
+		ccs_mipi_csi2_data_type(sensor->csi_format->compressed) :
+		CCS_DEFAULT_COMPRESSED_DT;
+	entry++;
+	desc->num_entries++;
+
+	return 0;
+}
+
 static int ccs_get_skip_frames(struct v4l2_subdev *subdev, u32 *frames)
 {
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
@@ -3042,6 +3101,7 @@ static const struct v4l2_subdev_pad_ops ccs_pad_ops = {
 	.set_selection = ccs_set_selection,
 	.enable_streams = ccs_enable_streams,
 	.disable_streams = ccs_disable_streams,
+	.get_frame_desc = ccs_get_frame_desc,
 };
 
 static const struct v4l2_subdev_sensor_ops ccs_sensor_ops = {
diff --git a/drivers/media/i2c/ccs/ccs-quirk.h b/drivers/media/i2c/ccs/ccs-quirk.h
index 0b1a64958d71..13f334fe0ac8 100644
--- a/drivers/media/i2c/ccs/ccs-quirk.h
+++ b/drivers/media/i2c/ccs/ccs-quirk.h
@@ -36,6 +36,7 @@ struct ccs_sensor;
  *			 access may be done by the caller (default read
  *			 value is zero), else negative error code on error
  * @flags: Quirk flags
+ * @frame_desc: Obtain the frame descriptor
  */
 struct ccs_quirk {
 	int (*limits)(struct ccs_sensor *sensor);
@@ -46,6 +47,8 @@ struct ccs_quirk {
 	int (*init)(struct ccs_sensor *sensor);
 	int (*reg_access)(struct ccs_sensor *sensor, bool write, u32 *reg,
 			  u32 *val);
+	int (*frame_desc)(struct ccs_sensor *sensor,
+			  struct v4l2_mbus_frame_desc *desc);
 	unsigned long flags;
 };
 
@@ -62,6 +65,10 @@ struct ccs_reg_8 {
 		.val = _val,		\
 	}
 
+#define ccs_has_quirk(sensor, _quirk)					\
+	((sensor)->minfo.quirk &&					\
+	 (sensor)->minfo.quirk->_quirk)
+
 #define ccs_call_quirk(sensor, _quirk, ...)				\
 	((sensor)->minfo.quirk &&					\
 	 (sensor)->minfo.quirk->_quirk ?				\
diff --git a/drivers/media/i2c/ccs/ccs.h b/drivers/media/i2c/ccs/ccs.h
index 043a679e115c..7fd9b6493d2b 100644
--- a/drivers/media/i2c/ccs/ccs.h
+++ b/drivers/media/i2c/ccs/ccs.h
@@ -45,6 +45,8 @@
 
 #define CCS_COLOUR_COMPONENTS		4
 
+#define CCS_DEFAULT_COMPRESSED_DT	0x30
+
 #define SMIAPP_NAME			"smiapp"
 #define CCS_NAME			"ccs"
 
@@ -174,6 +176,8 @@ struct ccs_csi_data_format {
 #define CCS_PAD_SRC			1
 #define CCS_PADS			2
 
+#define CCS_STREAM_PIXEL		0
+
 struct ccs_binning_subtype {
 	u8 horizontal:4;
 	u8 vertical:4;
-- 
2.39.2

[PATCH v6 26/28] media: ccs: Add support for embedded data stream

[Sakari Ailus]

Add support for embedded data stream, in UAPI and frame descriptor.

This patch adds also a new embedded data pad (2) to the source sub-device.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ccs/ccs-core.c | 337 +++++++++++++++++++++++++++++--
 drivers/media/i2c/ccs/ccs.h      |  18 +-
 2 files changed, 336 insertions(+), 19 deletions(-)

diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
index 75abb18ae847..2c8ff7cb63a5 100644
--- a/drivers/media/i2c/ccs/ccs-core.c
+++ b/drivers/media/i2c/ccs/ccs-core.c
@@ -1890,6 +1890,13 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
 	if (rval < 0)
 		goto err_pm_put;
 
+	/* Configure embedded data */
+	if (sensor->csi_format->compressed >= 16) {
+		rval = ccs_write(sensor, EMB_DATA_CTRL, sensor->emb_data_ctrl);
+		if (rval < 0)
+			goto err_pm_put;
+	}
+
 	if (CCS_LIM(sensor, FLASH_MODE_CAPABILITY) &
 	    (CCS_FLASH_MODE_CAPABILITY_SINGLE_STROBE |
 	     SMIAPP_FLASH_MODE_CAPABILITY_MULTIPLE_STROBE) &&
@@ -2009,6 +2016,57 @@ static const struct ccs_csi_data_format
 	return sensor->csi_format;
 }
 
+#define CCS_EMBEDDED_CODE_DEPTH(depth, half_depth)			\
+	depth,								\
+	CCS_EMB_DATA_CAPABILITY_TWO_BYTES_PER_RAW##depth,		\
+	CCS_EMB_DATA_CAPABILITY_NO_ONE_BYTE_PER_RAW##depth,		\
+	CCS_EMB_DATA_CTRL_RAW##half_depth##_PACKING_FOR_RAW##depth,	\
+	MEDIA_BUS_FMT_META_##half_depth,				\
+	MEDIA_BUS_FMT_META_##depth,					\
+
+static const struct ccs_embedded_code {
+	u8 depth;
+	u8 cap_two_bytes_per_sample;
+	u8 cap_no_legacy;
+	u8 ctrl;
+	u32 code_two_bytes;
+	u32 code_legacy;
+} ccs_embedded_codes[] = {
+	{ CCS_EMBEDDED_CODE_DEPTH(16, 8) },
+	{ CCS_EMBEDDED_CODE_DEPTH(20, 10) },
+	{ CCS_EMBEDDED_CODE_DEPTH(24, 12) },
+};
+
+static const struct ccs_embedded_code *ccs_embedded_code(unsigned int bpp)
+{
+	unsigned int i;
+
+	for (i = 0; i < ARRAY_SIZE(ccs_embedded_codes); i++)
+		if (ccs_embedded_codes[i].depth == bpp)
+			return ccs_embedded_codes + i;
+
+	WARN_ON(1);
+
+	return ccs_embedded_codes;
+}
+
+static u32
+ccs_default_embedded_code(struct ccs_sensor *sensor,
+			  const struct ccs_embedded_code *embedded_code)
+{
+	if (CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
+	    BIT(embedded_code->cap_two_bytes_per_sample))
+		return embedded_code->code_two_bytes;
+
+	if (!(CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
+	      BIT(embedded_code->cap_no_legacy)))
+		return embedded_code->code_legacy;
+
+	WARN_ON(1);
+
+	return embedded_code->code_legacy;
+}
+
 static int ccs_enum_mbus_code(struct v4l2_subdev *subdev,
 			      struct v4l2_subdev_state *sd_state,
 			      struct v4l2_subdev_mbus_code_enum *code)
@@ -2024,6 +2082,69 @@ static int ccs_enum_mbus_code(struct v4l2_subdev *subdev,
 	dev_err(&client->dev, "subdev %s, pad %u, index %u\n",
 		subdev->name, code->pad, code->index);
 
+	if (subdev == &sensor->src->sd) {
+		if (code->pad == CCS_PAD_META) {
+			if (code->index)
+				goto out;
+
+			code->code = MEDIA_BUS_FMT_CCS_EMBEDDED;
+
+			rval = 0;
+			goto out;
+		}
+		if (code->stream == CCS_STREAM_META) {
+			struct v4l2_mbus_framefmt *pix_fmt =
+				v4l2_subdev_state_get_stream_format(sd_state,
+								    CCS_PAD_SRC,
+								    CCS_STREAM_PIXEL);
+			const struct ccs_csi_data_format *csi_format =
+				ccs_validate_csi_data_format(sensor,
+							     pix_fmt->code);
+			unsigned int i = 0;
+			u32 codes[2];
+
+			switch (csi_format->compressed) {
+			case 8:
+				codes[i++] = MEDIA_BUS_FMT_META_8;
+				break;
+			case 10:
+				codes[i++] = MEDIA_BUS_FMT_META_10;
+				break;
+			case 12:
+				codes[i++] = MEDIA_BUS_FMT_META_12;
+				break;
+			case 14:
+				codes[i++] = MEDIA_BUS_FMT_META_14;
+				break;
+			case 16:
+			case 20:
+			case 24: {
+				const struct ccs_embedded_code *embedded_code =
+					ccs_embedded_code(csi_format->compressed);
+
+				if (CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
+				    BIT(embedded_code->cap_two_bytes_per_sample))
+					codes[i++] =
+						embedded_code->code_two_bytes;
+
+				if (!(CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
+				      BIT(embedded_code->cap_no_legacy)))
+					codes[i++] = embedded_code->code_legacy;
+				break;
+			}
+			default:
+				WARN_ON(1);
+			}
+
+			if (WARN_ON(i > ARRAY_SIZE(codes)) || code->index >= i)
+				goto out;
+
+			code->code = codes[code->index];
+			rval = 0;
+			goto out;
+		}
+	}
+
 	if (subdev != &sensor->src->sd || code->pad != CCS_PAD_SRC) {
 		if (code->index)
 			goto out;
@@ -2066,8 +2187,11 @@ static int __ccs_get_format(struct v4l2_subdev *subdev,
 			    struct v4l2_subdev_state *sd_state,
 			    struct v4l2_subdev_format *fmt)
 {
-	fmt->format = *v4l2_subdev_get_pad_format(subdev, sd_state, fmt->pad);
-	fmt->format.code = __ccs_get_mbus_code(subdev, fmt->pad);
+	fmt->format = *v4l2_subdev_get_fmt_ptr(subdev, sd_state, fmt->pad,
+					       fmt->stream);
+
+	if (fmt->pad != CCS_PAD_META && fmt->stream != CCS_STREAM_META)
+		fmt->format.code = __ccs_get_mbus_code(subdev, fmt->pad);
 
 	return 0;
 }
@@ -2097,10 +2221,12 @@ static void ccs_get_crop_compose(struct v4l2_subdev *subdev,
 	if (crops)
 		for (i = 0; i < subdev->entity.num_pads; i++)
 			crops[i] =
-				v4l2_subdev_get_pad_crop(subdev, sd_state, i);
+				v4l2_subdev_get_crop_ptr(subdev, sd_state, i,
+							 CCS_STREAM_PIXEL);
 	if (comps)
-		*comps = v4l2_subdev_get_pad_compose(subdev, sd_state,
-						     ssd->sink_pad);
+		*comps = v4l2_subdev_get_compose_ptr(subdev, sd_state,
+						     ssd->sink_pad,
+						     CCS_STREAM_PIXEL);
 }
 
 /* Changes require propagation only on sink pad. */
@@ -2193,6 +2319,83 @@ static int ccs_set_format_source(struct v4l2_subdev *subdev,
 	return ccs_pll_update(sensor);
 }
 
+static int ccs_set_format_meta(struct v4l2_subdev *subdev,
+			       struct v4l2_subdev_state *sd_state,
+			       struct v4l2_mbus_framefmt *fmt)
+{
+	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
+	const struct ccs_csi_data_format *csi_format;
+	struct v4l2_mbus_framefmt *pix_fmt;
+	struct v4l2_mbus_framefmt *meta_fmt;
+	struct v4l2_mbus_framefmt *meta_out_fmt;
+	u32 code;
+
+	pix_fmt = v4l2_subdev_state_get_stream_format(sd_state, CCS_PAD_SRC,
+						      CCS_STREAM_PIXEL);
+	meta_fmt = v4l2_subdev_state_get_stream_format(sd_state, CCS_PAD_META,
+						       0);
+	meta_out_fmt = v4l2_subdev_state_get_stream_format(sd_state,
+							   CCS_PAD_SRC,
+							   CCS_STREAM_META);
+
+	code = fmt ? fmt->code : meta_out_fmt->code;
+
+	meta_out_fmt->width = meta_fmt->width = pix_fmt->width;
+	meta_out_fmt->height = meta_fmt->height =
+		sensor->embedded_end - sensor->embedded_start;
+	meta_fmt->code = MEDIA_BUS_FMT_CCS_EMBEDDED;
+
+	csi_format = ccs_validate_csi_data_format(sensor, pix_fmt->code);
+
+	switch (csi_format->compressed) {
+	case 8:
+		meta_out_fmt->code = MEDIA_BUS_FMT_META_8;
+		break;
+	case 10:
+		meta_out_fmt->code = MEDIA_BUS_FMT_META_10;
+		break;
+	case 12:
+		meta_out_fmt->code = MEDIA_BUS_FMT_META_12;
+		break;
+	case 14:
+		meta_out_fmt->code = MEDIA_BUS_FMT_META_14;
+		break;
+	case 16:
+	case 20:
+	case 24: {
+		const struct ccs_embedded_code *embedded_code;
+
+		embedded_code = ccs_embedded_code(csi_format->compressed);
+		meta_out_fmt->code =
+			ccs_default_embedded_code(sensor, embedded_code);
+
+		if (!(CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
+		      BIT(embedded_code->cap_no_legacy)) &&
+		    code == embedded_code->code_legacy) {
+			meta_out_fmt->code = embedded_code->code_legacy;
+			sensor->emb_data_ctrl = 0;
+		}
+
+		if (CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
+		    BIT(embedded_code->cap_two_bytes_per_sample) &&
+		    code == embedded_code->code_two_bytes) {
+			meta_out_fmt->code = embedded_code->code_two_bytes;
+			sensor->emb_data_ctrl = embedded_code->ctrl;
+		}
+
+		break;
+	}
+	default:
+		WARN_ON(1);
+		return 0;
+	}
+
+	if (fmt)
+		*fmt = *meta_out_fmt;
+
+	return 0;
+}
+
 static int ccs_set_format(struct v4l2_subdev *subdev,
 			  struct v4l2_subdev_state *sd_state,
 			  struct v4l2_subdev_format *fmt)
@@ -2201,12 +2404,24 @@ static int ccs_set_format(struct v4l2_subdev *subdev,
 	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
 	struct v4l2_rect *crops[CCS_PADS];
 
+	if (subdev == &sensor->src->sd && fmt->pad == CCS_PAD_META)
+		return ccs_get_format(subdev, sd_state, fmt);
+
 	mutex_lock(&sensor->mutex);
 
+	if (subdev == &sensor->src->sd && fmt->stream == CCS_STREAM_META) {
+		ccs_set_format_meta(subdev, sd_state, &fmt->format);
+
+		mutex_unlock(&sensor->mutex);
+
+		return 0;
+	}
+
 	if (fmt->pad == ssd->source_pad) {
 		int rval;
 
 		rval = ccs_set_format_source(subdev, sd_state, fmt);
+		ccs_set_format_meta(subdev, sd_state, NULL);
 
 		mutex_unlock(&sensor->mutex);
 
@@ -2481,6 +2696,12 @@ static int ccs_sel_supported(struct v4l2_subdev *subdev,
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
 	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
 
+	if (sel->stream != CCS_STREAM_PIXEL)
+		return -EINVAL;
+
+	if (sel->pad == CCS_PAD_META)
+		return -EINVAL;
+
 	/* We only implement crop in three places. */
 	switch (sel->target) {
 	case V4L2_SEL_TGT_CROP:
@@ -2525,7 +2746,8 @@ static int ccs_set_crop(struct v4l2_subdev *subdev,
 
 	if (sel->pad == ssd->sink_pad) {
 		struct v4l2_mbus_framefmt *mfmt =
-			v4l2_subdev_get_pad_format(subdev, sd_state, sel->pad);
+			v4l2_subdev_get_fmt_ptr(subdev, sd_state, sel->pad,
+						CCS_STREAM_PIXEL);
 
 		src_size.width = mfmt->width;
 		src_size.height = mfmt->height;
@@ -2585,8 +2807,10 @@ static int ccs_get_selection(struct v4l2_subdev *subdev,
 			ccs_get_native_size(ssd, &sel->r);
 		} else if (sel->pad == ssd->sink_pad) {
 			struct v4l2_mbus_framefmt *sink_fmt =
-				v4l2_subdev_get_pad_format(subdev, sd_state,
-							   ssd->sink_pad);
+				v4l2_subdev_get_fmt_ptr(subdev, sd_state,
+							ssd->sink_pad,
+							CCS_STREAM_PIXEL);
+
 			sel->r.top = sel->r.left = 0;
 			sel->r.width = sink_fmt->width;
 			sel->r.height = sink_fmt->height;
@@ -2672,6 +2896,14 @@ static int ccs_get_frame_desc(struct v4l2_subdev *subdev, unsigned int pad,
 	entry++;
 	desc->num_entries++;
 
+	if (sensor->embedded_start != sensor->embedded_end) {
+		entry->pixelcode = MEDIA_BUS_FMT_CCS_EMBEDDED;
+		entry->stream = CCS_STREAM_META;
+		entry->bus.csi2.dt = MIPI_CSI2_DT_EMBEDDED_8B;
+		entry++;
+		desc->num_entries++;
+	}
+
 	return 0;
 }
 
@@ -3020,7 +3252,8 @@ static int ccs_init_subdev(struct ccs_sensor *sensor,
 	ssd->sensor = sensor;
 
 	ssd->npads = num_pads;
-	ssd->source_pad = num_pads - 1;
+	ssd->source_pad =
+		ssd == sensor->pixel_array ? CCS_PA_PAD_SRC : CCS_PAD_SRC;
 
 	v4l2_i2c_subdev_set_name(&ssd->sd, client, sensor->minfo.name, name);
 
@@ -3034,6 +3267,10 @@ static int ccs_init_subdev(struct ccs_sensor *sensor,
 		ssd->sd.owner = THIS_MODULE;
 		ssd->sd.dev = &client->dev;
 		v4l2_set_subdevdata(&ssd->sd, client);
+	} else {
+		ssd->sd.flags |= V4L2_SUBDEV_FL_STREAMS;
+		ssd->pads[CCS_PAD_META].flags =
+			MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL;
 	}
 
 	rval = media_entity_pads_init(&ssd->sd.entity, ssd->npads, ssd->pads);
@@ -3062,11 +3299,16 @@ static int ccs_init_cfg(struct v4l2_subdev *sd,
 
 	for (i = 0; i < ssd->npads; i++) {
 		struct v4l2_mbus_framefmt *fmt =
-			v4l2_subdev_get_pad_format(sd, sd_state, i);
+			v4l2_subdev_get_fmt_ptr(sd, sd_state, i,
+						CCS_STREAM_PIXEL);
 		struct v4l2_rect *crop =
-			v4l2_subdev_get_pad_crop(sd, sd_state, i);
+			v4l2_subdev_get_crop_ptr(sd, sd_state, i,
+						 CCS_STREAM_PIXEL);
 		struct v4l2_rect *comp;
 
+		if (!fmt)
+			continue;
+
 		ccs_get_native_size(ssd, crop);
 
 		fmt->width = crop->width;
@@ -3077,7 +3319,8 @@ static int ccs_init_cfg(struct v4l2_subdev *sd,
 		if (ssd == sensor->pixel_array)
 			continue;
 
-		comp = v4l2_subdev_get_pad_compose(sd, sd_state, i);
+		comp = v4l2_subdev_get_compose_ptr(sd, sd_state, i,
+						   CCS_STREAM_PIXEL);
 		*comp = *crop;
 	}
 
@@ -3086,6 +3329,47 @@ static int ccs_init_cfg(struct v4l2_subdev *sd,
 	return 0;
 }
 
+static int ccs_src_init_cfg(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_state *sd_state)
+{
+	struct v4l2_subdev_route routes[] = {
+		{
+			.sink_pad = CCS_PAD_SINK,
+			.source_pad = CCS_PAD_SRC,
+			.source_stream = CCS_STREAM_PIXEL,
+			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
+		}, {
+			.sink_pad = CCS_PAD_META,
+			.source_pad = CCS_PAD_SRC,
+			.source_stream = CCS_STREAM_META,
+			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
+		}
+	};
+	struct v4l2_subdev_krouting routing = {
+		.routes = routes,
+		.num_routes = 1,
+	};
+	struct ccs_subdev *ssd = to_ccs_subdev(sd);
+	struct ccs_sensor *sensor = ssd->sensor;
+	int rval;
+
+	if (sensor->embedded_start != sensor->embedded_end)
+		routing.num_routes++;
+
+	rval = v4l2_subdev_set_routing(sd, sd_state, &routing);
+	if (rval)
+		return rval;
+
+	rval = ccs_init_cfg(sd, sd_state);
+	if (rval)
+		return rval;
+
+	if (sensor->embedded_start != sensor->embedded_end)
+		ccs_set_format_meta(sd, sd_state, NULL);
+
+	return 0;
+}
+
 static const struct v4l2_subdev_video_ops ccs_video_ops = {
 	.s_stream = v4l2_subdev_s_stream_helper,
 	.pre_streamon = ccs_pre_streamon,
@@ -3099,6 +3383,15 @@ static const struct v4l2_subdev_pad_ops ccs_pad_ops = {
 	.set_fmt = ccs_set_format,
 	.get_selection = ccs_get_selection,
 	.set_selection = ccs_set_selection,
+};
+
+static const struct v4l2_subdev_pad_ops ccs_src_pad_ops = {
+	.init_cfg = ccs_src_init_cfg,
+	.enum_mbus_code = ccs_enum_mbus_code,
+	.get_fmt = ccs_get_format,
+	.set_fmt = ccs_set_format,
+	.get_selection = ccs_get_selection,
+	.set_selection = ccs_set_selection,
 	.enable_streams = ccs_enable_streams,
 	.disable_streams = ccs_disable_streams,
 	.get_frame_desc = ccs_get_frame_desc,
@@ -3115,6 +3408,12 @@ static const struct v4l2_subdev_ops ccs_ops = {
 	.sensor = &ccs_sensor_ops,
 };
 
+static const struct v4l2_subdev_ops ccs_src_ops = {
+	.video = &ccs_video_ops,
+	.pad = &ccs_src_pad_ops,
+	.sensor = &ccs_sensor_ops,
+};
+
 static const struct media_entity_operations ccs_entity_ops = {
 	.link_validate = v4l2_subdev_link_validate,
 };
@@ -3268,7 +3567,7 @@ static int ccs_probe(struct i2c_client *client)
 
 	sensor->src = &sensor->ssds[sensor->ssds_used];
 
-	v4l2_i2c_subdev_init(&sensor->src->sd, client, &ccs_ops);
+	v4l2_i2c_subdev_init(&sensor->src->sd, client, &ccs_src_ops);
 	sensor->src->sd.internal_ops = &ccs_internal_src_ops;
 
 	sensor->regulators = devm_kcalloc(&client->dev,
@@ -3533,11 +3832,19 @@ static int ccs_probe(struct i2c_client *client)
 		goto out_cleanup;
 	}
 
-	rval = ccs_init_subdev(sensor, sensor->scaler, " scaler", 2,
+	rval = ccs_init_subdev(sensor, sensor->scaler, " scaler",
+			       sensor->ssds_used != CCS_SUBDEVS ?
+			       CCS_PADS_NOMETA :
+			       sensor->embedded_start == sensor->embedded_end ?
+			       CCS_PADS_NOMETA : CCS_PADS,
 			       MEDIA_ENT_F_PROC_VIDEO_SCALER);
 	if (rval)
 		goto out_cleanup;
-	rval = ccs_init_subdev(sensor, sensor->binner, " binner", 2,
+	rval = ccs_init_subdev(sensor, sensor->binner, " binner",
+			       sensor->ssds_used == CCS_SUBDEVS ?
+			       CCS_PADS_NOMETA :
+			       sensor->embedded_start == sensor->embedded_end ?
+			       CCS_PADS_NOMETA : CCS_PADS,
 			       MEDIA_ENT_F_PROC_VIDEO_SCALER);
 	if (rval)
 		goto out_cleanup;
diff --git a/drivers/media/i2c/ccs/ccs.h b/drivers/media/i2c/ccs/ccs.h
index 7fd9b6493d2b..5e561cc91717 100644
--- a/drivers/media/i2c/ccs/ccs.h
+++ b/drivers/media/i2c/ccs/ccs.h
@@ -172,11 +172,18 @@ struct ccs_csi_data_format {
 #define CCS_SUBDEVS			3
 
 #define CCS_PA_PAD_SRC			0
-#define CCS_PAD_SINK			0
-#define CCS_PAD_SRC			1
-#define CCS_PADS			2
+enum {
+	CCS_PAD_SINK,
+	CCS_PAD_SRC,
+	CCS_PAD_META,
+	CCS_PADS_NOMETA = CCS_PAD_META,
+	CCS_PADS,
+};
 
-#define CCS_STREAM_PIXEL		0
+enum {
+	CCS_STREAM_PIXEL,
+	CCS_STREAM_META,
+};
 
 struct ccs_binning_subtype {
 	u8 horizontal:4;
@@ -226,6 +233,9 @@ struct ccs_sensor {
 	int default_pixel_order;
 	struct ccs_data_container sdata, mdata;
 
+	u32 embedded_mbus_code;
+	u8 emb_data_ctrl;
+
 	u8 binning_horizontal;
 	u8 binning_vertical;
 
-- 
2.39.2

[PATCH v6 27/28] media: ccs: Remove ccs_get_crop_compose helper

[Sakari Ailus]

As it's now easier to obtain the necessary information on crop and compose
rectangles after moving to sub-device state, remove the
ccs_get_crop_compose helper.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ccs/ccs-core.c | 120 ++++++++++++++-----------------
 1 file changed, 53 insertions(+), 67 deletions(-)

diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
index 2c8ff7cb63a5..9ebf6063150f 100644
--- a/drivers/media/i2c/ccs/ccs-core.c
+++ b/drivers/media/i2c/ccs/ccs-core.c
@@ -2210,25 +2210,6 @@ static int ccs_get_format(struct v4l2_subdev *subdev,
 	return rval;
 }
 
-static void ccs_get_crop_compose(struct v4l2_subdev *subdev,
-				 struct v4l2_subdev_state *sd_state,
-				 struct v4l2_rect **crops,
-				 struct v4l2_rect **comps)
-{
-	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
-	unsigned int i;
-
-	if (crops)
-		for (i = 0; i < subdev->entity.num_pads; i++)
-			crops[i] =
-				v4l2_subdev_get_crop_ptr(subdev, sd_state, i,
-							 CCS_STREAM_PIXEL);
-	if (comps)
-		*comps = v4l2_subdev_get_compose_ptr(subdev, sd_state,
-						     ssd->sink_pad,
-						     CCS_STREAM_PIXEL);
-}
-
 /* Changes require propagation only on sink pad. */
 static void ccs_propagate(struct v4l2_subdev *subdev,
 			  struct v4l2_subdev_state *sd_state, int which,
@@ -2237,9 +2218,14 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
 	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
 	struct v4l2_rect *comp, *crops[CCS_PADS];
+	unsigned int i;
 
-	ccs_get_crop_compose(subdev, sd_state, crops, &comp);
+	for (i = 0; i < subdev->entity.num_pads; i++)
+		crops[i] = v4l2_subdev_get_crop_ptr(subdev, sd_state, i,
+						    CCS_STREAM_PIXEL);
 
+	comp = v4l2_subdev_get_compose_ptr(subdev, sd_state,
+					   ssd->sink_pad, CCS_STREAM_PIXEL);
 	switch (target) {
 	case V4L2_SEL_TGT_CROP:
 		comp->width = crops[CCS_PAD_SINK]->width;
@@ -2402,7 +2388,7 @@ static int ccs_set_format(struct v4l2_subdev *subdev,
 {
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
 	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
-	struct v4l2_rect *crops[CCS_PADS];
+	struct v4l2_rect *crop;
 
 	if (subdev == &sensor->src->sd && fmt->pad == CCS_PAD_META)
 		return ccs_get_format(subdev, sd_state, fmt);
@@ -2443,12 +2429,13 @@ static int ccs_set_format(struct v4l2_subdev *subdev,
 		      CCS_LIM(sensor, MIN_Y_OUTPUT_SIZE),
 		      CCS_LIM(sensor, MAX_Y_OUTPUT_SIZE));
 
-	ccs_get_crop_compose(subdev, sd_state, crops, NULL);
+	crop = v4l2_subdev_get_crop_ptr(subdev, sd_state, ssd->sink_pad,
+					CCS_STREAM_PIXEL);
 
-	crops[ssd->sink_pad]->left = 0;
-	crops[ssd->sink_pad]->top = 0;
-	crops[ssd->sink_pad]->width = fmt->format.width;
-	crops[ssd->sink_pad]->height = fmt->format.height;
+	crop->left = 0;
+	crop->top = 0;
+	crop->width = fmt->format.width;
+	crop->height = fmt->format.height;
 	ccs_propagate(subdev, sd_state, fmt->which, V4L2_SEL_TGT_CROP);
 
 	mutex_unlock(&sensor->mutex);
@@ -2503,24 +2490,23 @@ static int scaling_goodness(struct v4l2_subdev *subdev, int w, int ask_w,
 static void ccs_set_compose_binner(struct v4l2_subdev *subdev,
 				   struct v4l2_subdev_state *sd_state,
 				   struct v4l2_subdev_selection *sel,
-				   struct v4l2_rect **crops,
+				   struct v4l2_rect *sink_crop,
 				   struct v4l2_rect *comp)
 {
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
 	unsigned int i;
 	unsigned int binh = 1, binv = 1;
 	int best = scaling_goodness(
-		subdev,
-		crops[CCS_PAD_SINK]->width, sel->r.width,
-		crops[CCS_PAD_SINK]->height, sel->r.height, sel->flags);
+		subdev,	sink_crop->width, sel->r.width,
+		sink_crop->height, sel->r.height, sel->flags);
 
 	for (i = 0; i < sensor->nbinning_subtypes; i++) {
 		int this = scaling_goodness(
 			subdev,
-			crops[CCS_PAD_SINK]->width
+			sink_crop->width
 			/ sensor->binning_subtypes[i].horizontal,
 			sel->r.width,
-			crops[CCS_PAD_SINK]->height
+			sink_crop->height
 			/ sensor->binning_subtypes[i].vertical,
 			sel->r.height, sel->flags);
 
@@ -2535,8 +2521,8 @@ static void ccs_set_compose_binner(struct v4l2_subdev *subdev,
 		sensor->binning_horizontal = binh;
 	}
 
-	sel->r.width = (crops[CCS_PAD_SINK]->width / binh) & ~1;
-	sel->r.height = (crops[CCS_PAD_SINK]->height / binv) & ~1;
+	sel->r.width = (sink_crop->width / binh) & ~1;
+	sel->r.height = (sink_crop->height / binv) & ~1;
 }
 
 /*
@@ -2551,7 +2537,7 @@ static void ccs_set_compose_binner(struct v4l2_subdev *subdev,
 static void ccs_set_compose_scaler(struct v4l2_subdev *subdev,
 				   struct v4l2_subdev_state *sd_state,
 				   struct v4l2_subdev_selection *sel,
-				   struct v4l2_rect **crops,
+				   struct v4l2_rect *sink_crop,
 				   struct v4l2_rect *comp)
 {
 	struct i2c_client *client = v4l2_get_subdevdata(subdev);
@@ -2564,17 +2550,12 @@ static void ccs_set_compose_scaler(struct v4l2_subdev *subdev,
 	unsigned int i;
 	int best = INT_MIN;
 
-	sel->r.width = min_t(unsigned int, sel->r.width,
-			     crops[CCS_PAD_SINK]->width);
-	sel->r.height = min_t(unsigned int, sel->r.height,
-			      crops[CCS_PAD_SINK]->height);
-
-	a = crops[CCS_PAD_SINK]->width
-		* CCS_LIM(sensor, SCALER_N_MIN) / sel->r.width;
-	b = crops[CCS_PAD_SINK]->height
-		* CCS_LIM(sensor, SCALER_N_MIN) / sel->r.height;
-	max_m = crops[CCS_PAD_SINK]->width
-		* CCS_LIM(sensor, SCALER_N_MIN)
+	sel->r.width = min_t(unsigned int, sel->r.width, sink_crop->width);
+	sel->r.height = min_t(unsigned int, sel->r.height, sink_crop->height);
+
+	a = sink_crop->width * CCS_LIM(sensor, SCALER_N_MIN) / sel->r.width;
+	b = sink_crop->height * CCS_LIM(sensor, SCALER_N_MIN) / sel->r.height;
+	max_m = sink_crop->width * CCS_LIM(sensor, SCALER_N_MIN)
 		/ CCS_LIM(sensor, MIN_X_OUTPUT_SIZE);
 
 	a = clamp(a, CCS_LIM(sensor, SCALER_M_MIN),
@@ -2607,10 +2588,10 @@ static void ccs_set_compose_scaler(struct v4l2_subdev *subdev,
 	for (i = 0; i < ntry; i++) {
 		int this = scaling_goodness(
 			subdev,
-			crops[CCS_PAD_SINK]->width
+			sink_crop->width
 			/ try[i] * CCS_LIM(sensor, SCALER_N_MIN),
 			sel->r.width,
-			crops[CCS_PAD_SINK]->height,
+			sink_crop->height,
 			sel->r.height,
 			sel->flags);
 
@@ -2627,12 +2608,10 @@ static void ccs_set_compose_scaler(struct v4l2_subdev *subdev,
 			continue;
 
 		this = scaling_goodness(
-			subdev, crops[CCS_PAD_SINK]->width
-			/ try[i]
+			subdev, sink_crop->width / try[i]
 			* CCS_LIM(sensor, SCALER_N_MIN),
 			sel->r.width,
-			crops[CCS_PAD_SINK]->height
-			/ try[i]
+			sink_crop->height / try[i]
 			* CCS_LIM(sensor, SCALER_N_MIN),
 			sel->r.height,
 			sel->flags);
@@ -2645,17 +2624,15 @@ static void ccs_set_compose_scaler(struct v4l2_subdev *subdev,
 	}
 
 	sel->r.width =
-		(crops[CCS_PAD_SINK]->width
-		 / scale_m
+		(sink_crop->width / scale_m
 		 * CCS_LIM(sensor, SCALER_N_MIN)) & ~1;
 	if (mode == SMIAPP_SCALING_MODE_BOTH)
 		sel->r.height =
-			(crops[CCS_PAD_SINK]->height
-			 / scale_m
+			(sink_crop->height / scale_m
 			 * CCS_LIM(sensor, SCALER_N_MIN))
 			& ~1;
 	else
-		sel->r.height = crops[CCS_PAD_SINK]->height;
+		sel->r.height = sink_crop->height;
 
 	if (sel->which == V4L2_SUBDEV_FORMAT_ACTIVE) {
 		sensor->scale_m = scale_m;
@@ -2669,17 +2646,20 @@ static int ccs_set_compose(struct v4l2_subdev *subdev,
 {
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
 	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
-	struct v4l2_rect *comp, *crops[CCS_PADS];
+	struct v4l2_rect *comp, *sink_crop;
 
-	ccs_get_crop_compose(subdev, sd_state, crops, &comp);
+	sink_crop = v4l2_subdev_get_crop_ptr(subdev, sd_state, CCS_PAD_SINK,
+					     CCS_STREAM_PIXEL);
+	comp = v4l2_subdev_get_compose_ptr(subdev, sd_state, ssd->sink_pad,
+					   CCS_STREAM_PIXEL);
 
 	sel->r.top = 0;
 	sel->r.left = 0;
 
 	if (ssd == sensor->binner)
-		ccs_set_compose_binner(subdev, sd_state, sel, crops, comp);
+		ccs_set_compose_binner(subdev, sd_state, sel, sink_crop, comp);
 	else
-		ccs_set_compose_scaler(subdev, sd_state, sel, crops, comp);
+		ccs_set_compose_scaler(subdev, sd_state, sel, sink_crop, comp);
 
 	*comp = sel->r;
 	ccs_propagate(subdev, sd_state, sel->which, V4L2_SEL_TGT_COMPOSE);
@@ -2740,9 +2720,12 @@ static int ccs_set_crop(struct v4l2_subdev *subdev,
 {
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
 	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
-	struct v4l2_rect src_size = { 0 }, *crops[CCS_PADS], *comp;
+	struct v4l2_rect src_size = { 0 }, *crop, *comp;
 
-	ccs_get_crop_compose(subdev, sd_state, crops, &comp);
+	crop = v4l2_subdev_get_crop_ptr(subdev, sd_state, sel->pad,
+					CCS_STREAM_PIXEL);
+	comp = v4l2_subdev_get_compose_ptr(subdev, sd_state, ssd->sink_pad,
+					   CCS_STREAM_PIXEL);
 
 	if (sel->pad == ssd->sink_pad) {
 		struct v4l2_mbus_framefmt *mfmt =
@@ -2766,7 +2749,7 @@ static int ccs_set_crop(struct v4l2_subdev *subdev,
 	sel->r.left = min_t(int, sel->r.left, src_size.width - sel->r.width);
 	sel->r.top = min_t(int, sel->r.top, src_size.height - sel->r.height);
 
-	*crops[sel->pad] = sel->r;
+	*crop = sel->r;
 
 	if (ssd != sensor->pixel_array && sel->pad == CCS_PAD_SINK)
 		ccs_propagate(subdev, sd_state, sel->which, V4L2_SEL_TGT_CROP);
@@ -2791,14 +2774,17 @@ static int ccs_get_selection(struct v4l2_subdev *subdev,
 {
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
 	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
-	struct v4l2_rect *comp, *crops[CCS_PADS];
+	struct v4l2_rect *crop, *comp;
 	int ret;
 
 	ret = ccs_sel_supported(subdev, sel);
 	if (ret)
 		return ret;
 
-	ccs_get_crop_compose(subdev, sd_state, crops, &comp);
+	crop = v4l2_subdev_get_crop_ptr(subdev, sd_state, sel->pad,
+					CCS_STREAM_PIXEL);
+	comp = v4l2_subdev_get_compose_ptr(subdev, sd_state, ssd->sink_pad,
+					   CCS_STREAM_PIXEL);
 
 	switch (sel->target) {
 	case V4L2_SEL_TGT_CROP_BOUNDS:
@@ -2820,7 +2806,7 @@ static int ccs_get_selection(struct v4l2_subdev *subdev,
 		break;
 	case V4L2_SEL_TGT_CROP:
 	case V4L2_SEL_TGT_COMPOSE_BOUNDS:
-		sel->r = *crops[sel->pad];
+		sel->r = *crop;
 		break;
 	case V4L2_SEL_TGT_COMPOSE:
 		sel->r = *comp;
-- 
2.39.2

[PATCH v6 28/28] media: ccs: Rely on sub-device state locking

[Sakari Ailus]

Rely on sub-device state locking to serialise access to driver's data
structures. The driver-provided mutex is used as the state lock for all
driver sub-devices.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ccs/ccs-core.c | 163 +++++++++++++++----------------
 drivers/media/i2c/ccs/ccs.h      |   1 -
 2 files changed, 77 insertions(+), 87 deletions(-)

diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
index 9ebf6063150f..89eda03a99a1 100644
--- a/drivers/media/i2c/ccs/ccs-core.c
+++ b/drivers/media/i2c/ccs/ccs-core.c
@@ -531,12 +531,13 @@ static int ccs_pll_update(struct ccs_sensor *sensor)
  *
  */
 
-static void __ccs_update_exposure_limits(struct ccs_sensor *sensor)
+static void __ccs_update_exposure_limits(struct ccs_sensor *sensor,
+					 struct v4l2_rect *pa_src)
 {
 	struct v4l2_ctrl *ctrl = sensor->exposure;
 	int max;
 
-	max = sensor->pa_src.height + sensor->vblank->val -
+	max = pa_src->height + sensor->vblank->val -
 		CCS_LIM(sensor, COARSE_INTEGRATION_TIME_MAX_MARGIN);
 
 	__v4l2_ctrl_modify_range(ctrl, ctrl->minimum, max, ctrl->step, max);
@@ -639,12 +640,21 @@ static int ccs_set_ctrl(struct v4l2_ctrl *ctrl)
 		container_of(ctrl->handler, struct ccs_subdev, ctrl_handler)
 			->sensor;
 	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
+	struct v4l2_subdev_state *state;
+	struct v4l2_rect *pa_src;
 	int pm_status;
 	u32 orient = 0;
 	unsigned int i;
 	int exposure;
 	int rval;
 
+	if (ctrl->id == V4L2_CID_VBLANK || ctrl->id == V4L2_CID_HBLANK) {
+		state = v4l2_subdev_get_locked_active_state(&sensor->pixel_array->sd);
+		pa_src = v4l2_subdev_get_crop_ptr(&sensor->pixel_array->sd,
+						  state, CCS_PA_PAD_SRC,
+						  CCS_STREAM_PIXEL);
+	}
+
 	switch (ctrl->id) {
 	case V4L2_CID_HFLIP:
 	case V4L2_CID_VFLIP:
@@ -663,7 +673,7 @@ static int ccs_set_ctrl(struct v4l2_ctrl *ctrl)
 	case V4L2_CID_VBLANK:
 		exposure = sensor->exposure->val;
 
-		__ccs_update_exposure_limits(sensor);
+		__ccs_update_exposure_limits(sensor, pa_src);
 
 		if (exposure > sensor->exposure->maximum) {
 			sensor->exposure->val =	sensor->exposure->maximum;
@@ -755,12 +765,12 @@ static int ccs_set_ctrl(struct v4l2_ctrl *ctrl)
 		break;
 	case V4L2_CID_VBLANK:
 		rval = ccs_write(sensor, FRAME_LENGTH_LINES,
-				 sensor->pa_src.height + ctrl->val);
+				 pa_src->height + ctrl->val);
 
 		break;
 	case V4L2_CID_HBLANK:
 		rval = ccs_write(sensor, LINE_LENGTH_PCK,
-				 sensor->pa_src.width + ctrl->val);
+				 pa_src->width + ctrl->val);
 
 		break;
 	case V4L2_CID_TEST_PATTERN:
@@ -1215,7 +1225,8 @@ static int ccs_get_mbus_formats(struct ccs_sensor *sensor)
 	return 0;
 }
 
-static void ccs_update_blanking(struct ccs_sensor *sensor)
+static void ccs_update_blanking(struct ccs_sensor *sensor,
+				struct v4l2_rect *pa_src)
 {
 	struct v4l2_ctrl *vblank = sensor->vblank;
 	struct v4l2_ctrl *hblank = sensor->hblank;
@@ -1238,21 +1249,26 @@ static void ccs_update_blanking(struct ccs_sensor *sensor)
 
 	min = max_t(int,
 		    CCS_LIM(sensor, MIN_FRAME_BLANKING_LINES),
-		    min_fll - sensor->pa_src.height);
-	max = max_fll -	sensor->pa_src.height;
+		    min_fll - pa_src->height);
+	max = max_fll -	pa_src->height;
 
 	__v4l2_ctrl_modify_range(vblank, min, max, vblank->step, min);
 
-	min = max_t(int, min_llp - sensor->pa_src.width, min_lbp);
-	max = max_llp - sensor->pa_src.width;
+	min = max_t(int, min_llp - pa_src->width, min_lbp);
+	max = max_llp - pa_src->width;
 
 	__v4l2_ctrl_modify_range(hblank, min, max, hblank->step, min);
 
-	__ccs_update_exposure_limits(sensor);
+	__ccs_update_exposure_limits(sensor, pa_src);
 }
 
 static int ccs_pll_blanking_update(struct ccs_sensor *sensor)
 {
+	struct v4l2_subdev_state *state =
+		v4l2_subdev_get_locked_active_state(&sensor->pixel_array->sd);
+	struct v4l2_rect *pa_src =
+		v4l2_subdev_get_crop_ptr(&sensor->pixel_array->sd, state,
+					 CCS_PA_PAD_SRC, CCS_STREAM_PIXEL);
 	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
 	int rval;
 
@@ -1261,15 +1277,15 @@ static int ccs_pll_blanking_update(struct ccs_sensor *sensor)
 		return rval;
 
 	/* Output from pixel array, including blanking */
-	ccs_update_blanking(sensor);
+	ccs_update_blanking(sensor, pa_src);
 
 	dev_dbg(&client->dev, "vblank\t\t%d\n", sensor->vblank->val);
 	dev_dbg(&client->dev, "hblank\t\t%d\n", sensor->hblank->val);
 
 	dev_dbg(&client->dev, "real timeperframe\t100/%d\n",
 		sensor->pll.pixel_rate_pixel_array /
-		((sensor->pa_src.width + sensor->hblank->val) *
-		 (sensor->pa_src.height + sensor->vblank->val) / 100));
+		((pa_src->width + sensor->hblank->val) *
+		 (pa_src->height + sensor->vblank->val) / 100));
 
 	return 0;
 }
@@ -1775,6 +1791,16 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
 			      u64 streams_mask)
 {
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
+	struct v4l2_subdev_state *pa_state =
+		v4l2_subdev_get_locked_active_state(&sensor->pixel_array->sd);
+	struct v4l2_subdev_state *src_state =
+		v4l2_subdev_get_locked_active_state(&sensor->src->sd);
+	struct v4l2_rect *pa_src =
+		v4l2_subdev_get_crop_ptr(&sensor->pixel_array->sd, pa_state,
+					 CCS_PA_PAD_SRC, CCS_STREAM_PIXEL);
+	struct v4l2_rect *src_src =
+		v4l2_subdev_get_crop_ptr(&sensor->src->sd, src_state,
+					 CCS_PAD_SRC, CCS_STREAM_PIXEL);
 	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
 	unsigned int binning_mode;
 	int rval;
@@ -1822,22 +1848,20 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
 		goto err_pm_put;
 
 	/* Analog crop start coordinates */
-	rval = ccs_write(sensor, X_ADDR_START, sensor->pa_src.left);
+	rval = ccs_write(sensor, X_ADDR_START, pa_src->left);
 	if (rval < 0)
 		goto err_pm_put;
 
-	rval = ccs_write(sensor, Y_ADDR_START, sensor->pa_src.top);
+	rval = ccs_write(sensor, Y_ADDR_START, pa_src->top);
 	if (rval < 0)
 		goto err_pm_put;
 
 	/* Analog crop end coordinates */
-	rval = ccs_write(sensor, X_ADDR_END,
-			 sensor->pa_src.left + sensor->pa_src.width - 1);
+	rval = ccs_write(sensor, X_ADDR_END, pa_src->left + pa_src->width - 1);
 	if (rval < 0)
 		goto err_pm_put;
 
-	rval = ccs_write(sensor, Y_ADDR_END,
-			 sensor->pa_src.top + sensor->pa_src.height - 1);
+	rval = ccs_write(sensor, Y_ADDR_END, pa_src->top + pa_src->height - 1);
 	if (rval < 0)
 		goto err_pm_put;
 
@@ -1849,23 +1873,31 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
 	/* Digital crop */
 	if (CCS_LIM(sensor, DIGITAL_CROP_CAPABILITY)
 	    == CCS_DIGITAL_CROP_CAPABILITY_INPUT_CROP) {
+		struct v4l2_subdev_state *scaler_state =
+			v4l2_subdev_get_locked_active_state(&sensor->scaler->sd);
+		struct v4l2_rect *scaler_sink =
+			v4l2_subdev_get_crop_ptr(&sensor->scaler->sd,
+						 scaler_state,
+						 sensor->scaler->sink_pad,
+						 CCS_STREAM_PIXEL);
+
 		rval = ccs_write(sensor, DIGITAL_CROP_X_OFFSET,
-				 sensor->scaler_sink.left);
+				 scaler_sink->left);
 		if (rval < 0)
 			goto err_pm_put;
 
 		rval = ccs_write(sensor, DIGITAL_CROP_Y_OFFSET,
-				 sensor->scaler_sink.top);
+				 scaler_sink->top);
 		if (rval < 0)
 			goto err_pm_put;
 
 		rval = ccs_write(sensor, DIGITAL_CROP_IMAGE_WIDTH,
-				 sensor->scaler_sink.width);
+				 scaler_sink->width);
 		if (rval < 0)
 			goto err_pm_put;
 
 		rval = ccs_write(sensor, DIGITAL_CROP_IMAGE_HEIGHT,
-				 sensor->scaler_sink.height);
+				 scaler_sink->height);
 		if (rval < 0)
 			goto err_pm_put;
 	}
@@ -1883,10 +1915,10 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
 	}
 
 	/* Output size from sensor */
-	rval = ccs_write(sensor, X_OUTPUT_SIZE, sensor->src_src.width);
+	rval = ccs_write(sensor, X_OUTPUT_SIZE, src_src->width);
 	if (rval < 0)
 		goto err_pm_put;
-	rval = ccs_write(sensor, Y_OUTPUT_SIZE, sensor->src_src.height);
+	rval = ccs_write(sensor, Y_OUTPUT_SIZE, src_src->height);
 	if (rval < 0)
 		goto err_pm_put;
 
@@ -2075,9 +2107,6 @@ static int ccs_enum_mbus_code(struct v4l2_subdev *subdev,
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
 	unsigned int i;
 	int idx = -1;
-	int rval = -EINVAL;
-
-	mutex_lock(&sensor->mutex);
 
 	dev_err(&client->dev, "subdev %s, pad %u, index %u\n",
 		subdev->name, code->pad, code->index);
@@ -2085,12 +2114,11 @@ static int ccs_enum_mbus_code(struct v4l2_subdev *subdev,
 	if (subdev == &sensor->src->sd) {
 		if (code->pad == CCS_PAD_META) {
 			if (code->index)
-				goto out;
+				return -EINVAL;
 
 			code->code = MEDIA_BUS_FMT_CCS_EMBEDDED;
 
-			rval = 0;
-			goto out;
+			return 0;
 		}
 		if (code->stream == CCS_STREAM_META) {
 			struct v4l2_mbus_framefmt *pix_fmt =
@@ -2137,21 +2165,21 @@ static int ccs_enum_mbus_code(struct v4l2_subdev *subdev,
 			}
 
 			if (WARN_ON(i > ARRAY_SIZE(codes)) || code->index >= i)
-				goto out;
+				return -EINVAL;
 
 			code->code = codes[code->index];
-			rval = 0;
-			goto out;
+
+			return 0;
 		}
 	}
 
 	if (subdev != &sensor->src->sd || code->pad != CCS_PAD_SRC) {
 		if (code->index)
-			goto out;
+			return -EINVAL;
 
 		code->code = sensor->internal_csi_format->code;
-		rval = 0;
-		goto out;
+
+		return 0;
 	}
 
 	for (i = 0; i < ARRAY_SIZE(ccs_csi_data_formats); i++) {
@@ -2162,18 +2190,14 @@ static int ccs_enum_mbus_code(struct v4l2_subdev *subdev,
 			code->code = ccs_csi_data_formats[i].code;
 			dev_err(&client->dev, "found index %u, i %u, code %x\n",
 				code->index, i, code->code);
-			rval = 0;
-			break;
+			return 0;
 		}
 	}
 
-out:
-	mutex_unlock(&sensor->mutex);
-
-	return rval;
+	return -EINVAL;
 }
 
-static u32 __ccs_get_mbus_code(struct v4l2_subdev *subdev, unsigned int pad)
+static u32 ccs_get_mbus_code(struct v4l2_subdev *subdev, unsigned int pad)
 {
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
 
@@ -2183,33 +2207,19 @@ static u32 __ccs_get_mbus_code(struct v4l2_subdev *subdev, unsigned int pad)
 		return sensor->internal_csi_format->code;
 }
 
-static int __ccs_get_format(struct v4l2_subdev *subdev,
-			    struct v4l2_subdev_state *sd_state,
-			    struct v4l2_subdev_format *fmt)
+static int ccs_get_format(struct v4l2_subdev *subdev,
+			  struct v4l2_subdev_state *sd_state,
+			  struct v4l2_subdev_format *fmt)
 {
 	fmt->format = *v4l2_subdev_get_fmt_ptr(subdev, sd_state, fmt->pad,
 					       fmt->stream);
 
 	if (fmt->pad != CCS_PAD_META && fmt->stream != CCS_STREAM_META)
-		fmt->format.code = __ccs_get_mbus_code(subdev, fmt->pad);
+		fmt->format.code = ccs_get_mbus_code(subdev, fmt->pad);
 
 	return 0;
 }
 
-static int ccs_get_format(struct v4l2_subdev *subdev,
-			  struct v4l2_subdev_state *sd_state,
-			  struct v4l2_subdev_format *fmt)
-{
-	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
-	int rval;
-
-	mutex_lock(&sensor->mutex);
-	rval = __ccs_get_format(subdev, sd_state, fmt);
-	mutex_unlock(&sensor->mutex);
-
-	return rval;
-}
-
 /* Changes require propagation only on sink pad. */
 static void ccs_propagate(struct v4l2_subdev *subdev,
 			  struct v4l2_subdev_state *sd_state, int which,
@@ -2235,7 +2245,6 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
 				sensor->scale_m = CCS_LIM(sensor, SCALER_N_MIN);
 				sensor->scaling_mode =
 					CCS_SCALING_MODE_NO_SCALING;
-				sensor->scaler_sink = *comp;
 			} else if (ssd == sensor->binner) {
 				sensor->binning_horizontal = 1;
 				sensor->binning_vertical = 1;
@@ -2244,8 +2253,6 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
 		fallthrough;
 	case V4L2_SEL_TGT_COMPOSE:
 		*crops[CCS_PAD_SRC] = *comp;
-		if (which == V4L2_SUBDEV_FORMAT_ACTIVE && ssd == sensor->src)
-			sensor->src_src = *crops[CCS_PAD_SRC];
 		break;
 	default:
 		WARN_ON_ONCE(1);
@@ -2264,7 +2271,7 @@ static int ccs_set_format_source(struct v4l2_subdev *subdev,
 	unsigned int i;
 	int rval;
 
-	rval = __ccs_get_format(subdev, sd_state, fmt);
+	rval = ccs_get_format(subdev, sd_state, fmt);
 	if (rval)
 		return rval;
 
@@ -2393,13 +2400,9 @@ static int ccs_set_format(struct v4l2_subdev *subdev,
 	if (subdev == &sensor->src->sd && fmt->pad == CCS_PAD_META)
 		return ccs_get_format(subdev, sd_state, fmt);
 
-	mutex_lock(&sensor->mutex);
-
 	if (subdev == &sensor->src->sd && fmt->stream == CCS_STREAM_META) {
 		ccs_set_format_meta(subdev, sd_state, &fmt->format);
 
-		mutex_unlock(&sensor->mutex);
-
 		return 0;
 	}
 
@@ -2409,13 +2412,12 @@ static int ccs_set_format(struct v4l2_subdev *subdev,
 		rval = ccs_set_format_source(subdev, sd_state, fmt);
 		ccs_set_format_meta(subdev, sd_state, NULL);
 
-		mutex_unlock(&sensor->mutex);
-
 		return rval;
 	}
 
 	/* Sink pad. Width and height are changeable here. */
-	fmt->format.code = __ccs_get_mbus_code(subdev, fmt->pad);
+	fmt->format.code = ccs_get_mbus_code(subdev, fmt->pad);
+
 	fmt->format.width &= ~1;
 	fmt->format.height &= ~1;
 	fmt->format.field = V4L2_FIELD_NONE;
@@ -2438,8 +2440,6 @@ static int ccs_set_format(struct v4l2_subdev *subdev,
 	crop->height = fmt->format.height;
 	ccs_propagate(subdev, sd_state, fmt->which, V4L2_SEL_TGT_CROP);
 
-	mutex_unlock(&sensor->mutex);
-
 	return 0;
 }
 
@@ -2753,9 +2753,6 @@ static int ccs_set_crop(struct v4l2_subdev *subdev,
 
 	if (ssd != sensor->pixel_array && sel->pad == CCS_PAD_SINK)
 		ccs_propagate(subdev, sd_state, sel->which, V4L2_SEL_TGT_CROP);
-	else if (sel->which == V4L2_SUBDEV_FORMAT_ACTIVE &&
-		 ssd == sensor->pixel_array)
-		sensor->pa_src = sel->r;
 
 	return 0;
 }
@@ -2827,8 +2824,6 @@ static int ccs_set_selection(struct v4l2_subdev *subdev,
 	if (ret)
 		return ret;
 
-	mutex_lock(&sensor->mutex);
-
 	sel->r.left = max(0, sel->r.left & ~1);
 	sel->r.top = max(0, sel->r.top & ~1);
 	sel->r.width = CCS_ALIGN_DIM(sel->r.width, sel->flags);
@@ -2850,7 +2845,6 @@ static int ccs_set_selection(struct v4l2_subdev *subdev,
 		ret = -EINVAL;
 	}
 
-	mutex_unlock(&sensor->mutex);
 	return ret;
 }
 
@@ -3235,6 +3229,7 @@ static int ccs_init_subdev(struct ccs_sensor *sensor,
 
 	ssd->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
 	ssd->sd.entity.function = function;
+	ssd->sd.state_lock = &sensor->mutex;
 	ssd->sensor = sensor;
 
 	ssd->npads = num_pads;
@@ -3281,8 +3276,6 @@ static int ccs_init_cfg(struct v4l2_subdev *sd,
 	struct ccs_sensor *sensor = ssd->sensor;
 	unsigned int i;
 
-	mutex_lock(&sensor->mutex);
-
 	for (i = 0; i < ssd->npads; i++) {
 		struct v4l2_mbus_framefmt *fmt =
 			v4l2_subdev_get_fmt_ptr(sd, sd_state, i,
@@ -3310,8 +3303,6 @@ static int ccs_init_cfg(struct v4l2_subdev *sd,
 		*comp = *crop;
 	}
 
-	mutex_unlock(&sensor->mutex);
-
 	return 0;
 }
 
diff --git a/drivers/media/i2c/ccs/ccs.h b/drivers/media/i2c/ccs/ccs.h
index 5e561cc91717..47c77f3284c8 100644
--- a/drivers/media/i2c/ccs/ccs.h
+++ b/drivers/media/i2c/ccs/ccs.h
@@ -228,7 +228,6 @@ struct ccs_sensor {
 	u32 mbus_frame_fmts;
 	const struct ccs_csi_data_format *csi_format;
 	const struct ccs_csi_data_format *internal_csi_format;
-	struct v4l2_rect pa_src, scaler_sink, src_src;
 	u32 default_mbus_frame_fmts;
 	int default_pixel_order;
 	struct ccs_data_container sdata, mdata;
-- 
2.39.2

Re: [PATCH v6 00/28] Generic line based metadata support, internal pads

[Tomi Valkeinen]

Hi!

Thanks for working on this. I think this series is very important.

On 03/10/2023 14:52, Sakari Ailus wrote:
> Hi folks,
> 
> Here are a few patches to add support generic, line based metadata as well
> as internal pads. While the amount of code is not very large, to the
> contrary it is quite small actually IMO, I presume what this is about and
> why it is being proposed requires some explaining.
> 
> Metadata mbus codes and formats have existed for some time in V4L2. They
> however have been only used by drivers that produce the data itself and
> effectively this metadata has always been statistics of some sort (at
> least when it comes to ISPs). What is different here is that we intend to
> add support for metadata originating from camera sensors.
> 
> Camera sensors produce different kinds of metadata, embedded data (usually
> register address--value pairs used to capture the frame, in a more or less
> sensor specific format), histograms (in a very sensor specific format),
> dark pixels etc. The number of these formats is probably going to be about
> as large as image data formats if not larger, as the image data formats
> are much better standardised but a smaller subset of them will be
> supported by V4L2, at least initially but possibly much more in the long
> run.
> 
> Having this many device specific formats would be a major problem for all
> the other drivers along that pipeline (not to mention the users of those
> drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2
> receiver drivers that have DMA: the poor driver developer would not only
> need to know camera sensor specific formats but to choose the specific
> packing of that format suitable for the DMA used by the hardware. It is
> unlikely many of these would ever get tested while being present on the
> driver API. Also adding new sensors with new embedded data formats would
> involve updating all bridge and CSI-2 receiver drivers. I don't expect
> this to be a workable approach.
> 
> Instead what I'm proposing is to use specific metadata formats on the
> sensor devices only, on internal pads (more about those soon) of the
> sensors, only visible in the UAPI, and then generic mbus formats along the

What do you mean with "only visible in the UAPI"?

> pipeline and finally generic V4L2 metadata formats on the DMAs (specific
> to bit depth and packing). This would unsnarl the two, defining what data
> there is (specific mbus code) and how that is transported and packed
> (generic mbus codes and V4L2 formats).
> 
> The user space would be required to "know" the path of that data from the
> sensor's internal pad to the V4L2 video node. I do not see this as these
> devices require at least some knowledge of the pipeline, i.e. hardware at
> hand. Separating what the data means and how it is packed may even be
> beneficial: it allows separating code that interprets the data (sensor
> internal mbus code) from the code that accesses it (packing).
> 
> These formats are in practice line based, meaning that there may be
> padding at the end of the line, depending on the bus as well as the DMA.
> If non-line based formats are needed, it is always possible to set the
> "height" field to 1.
> 
> The internal (source) pads are an alternative to source routes [1]. The
> source routes were not universally liked and I do have to say I like
> re-using existing interface concepts (pads and everything you can do with
> pads, including access format, selections etc.) wherever it makes sense,
> instead of duplicating functionality.
> 
> Effectively internal source pads behave mostly just like sink pads, but
> they describe a flow of data that originates from a sub-device instead of
> arriving to a sub-device. The SUBDEV_S_ROUTING IOCTLs are used to enable
> and disable routes from internal source pads to sub-device's source pads.
> The subdev format IOCTLs are usable, too, so one can find which subdev
> format is available on given internal source pad.

I think the internal pads require a bit more praise, as they can be used 
for other things too. E.g. the ds90ub953 FPD-Link serializer has a test 
pattern generator, which can be modeled very nicely with internal pads. 
The internal pad represents the TPG, and the user can use routing to 
choose if the output of the device is sourced from the normal input or 
from the TPG. And one can set the format on the TPG pad, thus 
configuring the TPG.

> This set depends on these patches:
> 
> <URL:https://lore.kernel.org/linux-media/20231002105557.28972-1-sakari.ailus@linux.intel.com/T/#t>

Hmm, it's a bit odd for a generic series to depend on a device specific 
series. That makes backporting these more difficult. Why do these depend 
on ov2740 and css patches?

> I've also pushed these here and I'll keep updating the branch, I've also
> included untested OV2740 patches:
> 
> <URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=metadata>
> 
> Questions and comments are most welcome.
> 
> Preliminary media-ctl and yavta patches can be found here:
> 
> <URL:https://git.retiisi.eu/?p=~sailus/yavta.git;a=shortlog;h=refs/heads/metadata>
> <URL:https://git.retiisi.eu/?p=~sailus/v4l-utils.git;a=shortlog;h=refs/heads/metadata>
> 
> I have used IMX219 as an example on routing in a sensor driver in this
> version. I also hope I can add OV2740 support in the next version after
> testing the patches.

Sorry, I didn't understand that. You have used imx219 for what, where?

  Tomi

> [1] <URL:https://lore.kernel.org/linux-media/20220831141357.1396081-20-tomi.valkeinen@ideasonboard.com/>
> 
> since v5:
> 
> - Rebase on new set of preparation patches.
> 
> - Switch CCS driver from s_stream to enable_streams/disable_streams. Keep
>    streaming state information --- the sensor remains in streaming state if
>    any of the streams is enabled.
> 
> - Fix setting mbus code on embedded data in get_frame_desc() op in the CCS
>    driver.
> 
> since v4:
> 
> - Add a patch to acquire two sub-device states that may use the same lock.
> 
> - Add a patch for CCS driver to remove ccs_get_crop_compose() helper.
> 
> - Add a patch for CCS driver moving acquiring and releasing the mutex to
>    the s_stream callback.
> 
> - Add a patch for CCS driver to rely on sub-device state locking using a
>    single driver-provided lock.
> 
> - Fixed calculating minimum number of routes in copying the routes
>    (thanks, Laurent).
> 
> - Moved a label in S_ROUTING handling to make Clang happy (hopefully).
> 
> - Fixed setting emb_data_ctrl register for CCS embedded data support.
> 
> - Rebase on Laurent's cleanup patches.
> 
> - Wrap a few long lines.
> 
> - Write in embedded data documentation sensor drivers generally don't
>    allow configuring it.
> 
> since v3:
> 
> - Separate preparation patches from this set.
> 
> - Add a definition for "Data unit", a pixel that is not image data and use
>    it instead in format documentation.
> 
> - Fix more numbered lists in dev-subdev.rst.
> 
> - Remove a redundant definition for V4L2_META_FMT_GENERIC_CSI2_2_24 ---
>    V4L2_META_FMT_GENERIC_CSI2_12 can be used instead.
> 
> - Use "X" instead of "p" to denote padding in format documentation.
> 
> - Use IMX219 in examples instead of CCS.
> 
> - Document that the generic V4L2 CSI-2 metadata formats use padding
>    defined in CSI-2 spec and packing defined in CCS spec.
> 
> - Add patches to align [GS]_ROUTING behaviour with V4L2. This means mainly
>    returning configured routes as part of S_ROUTING as well. "len_routes"
>    field is added to denote the length of the array and having more routes
>    than fits in the array is no longer an error. Also added more reserved
>    fields.
> 
> - Added trivial support for S_ROUTING (via G_ROUTING implementation) for
>    use in drivers with static-only routes.
> 
> - Added helper functions to obtain mbus format as well as crop and compose
>    rectangles that are streams-independent.
> 
> - Added a patch to define generic CSI-2 long packet types.
> 
> - Removed MEDIA_BUS_FMT_IS_META() macro. It didn't seem useful in the end.
> 
> - Use a single CCS embedded data format. The bit depth can be selected
>    using the meta stream on the source pad.
> 
> - Fix mbus code numbers (there were holes due to removed redundant
>    formats).
> 
> - Fix generic mbus code documentation (byte was being used instead of
>    bit).
> 
> - Fix spelling of "length".
> 
> - Added a patch to remove v4l2_subdev_enable_streams_api that disables
>    streams API. This should be merged once libcamera support for streams
>    works nicely.
> 
> - Don't use strings in printing frame descriptor flags.
> 
> - Warn on string truncation in printing frame descriptor.
> 
> since v2:
> 
> - Add a better example, with formats.
> 
> - Add CCS static data media bus codes.
> 
> - Added an example demonstrating the use of internal pads. --- Is the
>    level of detail enough for the purpose?
> 
> - Improved documentation.
> 
> - Added a macro to tell whether a format is a metadata format.
>    (Documentation could be added.)
> 
> - A small ReST syntax fix in the same section.
> 
> - Drop leftovers of a patch checking for the INTERNAL_SOURCE flag.
> 
> since v1:
> 
> - Make the new pad flag just "INTERNAL", requiring either SINK or SOURCE
>    pad flag to accompany it. Removed the union in struct v4l2_subdev_route.
> 
> - Add the term "stream" to MC glossary.
> 
> - Improved and fixed documentation (according to comments).
> 
> - Note these formats are little endian.
> 
> - Remove 1X8 from the names of the mbus codes. These formats have generally
>    8 bits per pixel.
> 
> - Fix mbus code numbering (had holes in RFC).
> 
> - Add new metadata fields to debug prints.
> 
> - Fix a minor documentation build issue.
> 
> Sakari Ailus (28):
>    media: mc: Add INTERNAL pad flag
>    media: uapi: Add generic serial metadata mbus formats
>    media: uapi: Document which mbus format fields are valid for metadata
>    media: uapi: Add generic 8-bit metadata format definitions
>    media: v4l: Support line-based metadata capture
>    media: uapi: ccs: Add media bus code for MIPI CCS embedded data
>    media: Documentation: ccs: Document routing
>    media: Documentation: Additional streams generally don't harm capture
>    media: Documentation: Document embedded data guidelines for camera
>      sensors
>    media: Documentation: v4l: Document source routes
>    media: Documentation: Document S_ROUTING behaviour
>    media: v4l: subdev: Add helpers for format, crop and compose pointers
>    media: v4l: subdev: Add a function to lock two sub-device states, use
>      it
>    media: v4l: subdev: Move G_ROUTING handling below S_ROUTING
>    media: v4l: subdev: Copy argument back to user also for S_ROUTING
>    media: v4l: subdev: Add len_routes field to struct v4l2_subdev_routing
>    media: v4l: subdev: Return routes set using S_ROUTING
>    media: uapi: Allow a larger number of routes than there's room for
>    media: v4l: subdev: Add trivial set_routing support
>    media: uapi: v4l: subdev: Enable streams API
>    media: ccs: No need to set streaming to false in power off
>    media: ccs: Use {enable,disable}_streams operations
>    media: ccs: Track streaming state
>    media: ccs: Move ccs_validate_csi_data_format up
>    media: ccs: Support frame descriptors
>    media: ccs: Add support for embedded data stream
>    media: ccs: Remove ccs_get_crop_compose helper
>    media: ccs: Rely on sub-device state locking
> 
>   .../media/drivers/camera-sensor.rst           |  28 +
>   .../userspace-api/media/drivers/ccs.rst       |  34 +-
>   .../userspace-api/media/glossary.rst          |  14 +
>   .../media/mediactl/media-types.rst            |   6 +
>   .../userspace-api/media/v4l/dev-meta.rst      |  15 +
>   .../userspace-api/media/v4l/dev-subdev.rst    | 208 ++++-
>   .../userspace-api/media/v4l/meta-formats.rst  |   1 +
>   .../media/v4l/metafmt-generic.rst             | 304 +++++++
>   .../media/v4l/subdev-formats.rst              | 288 ++++++
>   .../media/v4l/vidioc-enum-fmt.rst             |   7 +
>   .../media/v4l/vidioc-subdev-g-routing.rst     |  40 +-
>   .../media/videodev2.h.rst.exceptions          |   1 +
>   drivers/media/i2c/ccs/ccs-core.c              | 859 ++++++++++++------
>   drivers/media/i2c/ccs/ccs-quirk.h             |   7 +
>   drivers/media/i2c/ccs/ccs.h                   |  23 +-
>   drivers/media/mc/mc-entity.c                  |  10 +-
>   drivers/media/v4l2-core/v4l2-ioctl.c          |  19 +-
>   drivers/media/v4l2-core/v4l2-subdev.c         | 145 +--
>   include/media/v4l2-subdev.h                   |  98 ++
>   include/uapi/linux/media-bus-format.h         |  12 +
>   include/uapi/linux/media.h                    |   1 +
>   include/uapi/linux/v4l2-mediabus.h            |  18 +-
>   include/uapi/linux/v4l2-subdev.h              |   8 +-
>   include/uapi/linux/videodev2.h                |  18 +
>   24 files changed, 1799 insertions(+), 365 deletions(-)
>   create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
> 
> 
> base-commit: 0fa78064f6e3a354616fb24462864900c0db3191

Re: [PATCH v6 00/28] Generic line based metadata support, internal pads

[Sakari Ailus]

Hyvää huomenta!

On Thu, Oct 05, 2023 at 11:05:54AM +0300, Tomi Valkeinen wrote:
> Hi!
> 
> Thanks for working on this. I think this series is very important.
> 
> On 03/10/2023 14:52, Sakari Ailus wrote:
> > Hi folks,
> > 
> > Here are a few patches to add support generic, line based metadata as well
> > as internal pads. While the amount of code is not very large, to the
> > contrary it is quite small actually IMO, I presume what this is about and
> > why it is being proposed requires some explaining.
> > 
> > Metadata mbus codes and formats have existed for some time in V4L2. They
> > however have been only used by drivers that produce the data itself and
> > effectively this metadata has always been statistics of some sort (at
> > least when it comes to ISPs). What is different here is that we intend to
> > add support for metadata originating from camera sensors.
> > 
> > Camera sensors produce different kinds of metadata, embedded data (usually
> > register address--value pairs used to capture the frame, in a more or less
> > sensor specific format), histograms (in a very sensor specific format),
> > dark pixels etc. The number of these formats is probably going to be about
> > as large as image data formats if not larger, as the image data formats
> > are much better standardised but a smaller subset of them will be
> > supported by V4L2, at least initially but possibly much more in the long
> > run.
> > 
> > Having this many device specific formats would be a major problem for all
> > the other drivers along that pipeline (not to mention the users of those
> > drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2
> > receiver drivers that have DMA: the poor driver developer would not only
> > need to know camera sensor specific formats but to choose the specific
> > packing of that format suitable for the DMA used by the hardware. It is
> > unlikely many of these would ever get tested while being present on the
> > driver API. Also adding new sensors with new embedded data formats would
> > involve updating all bridge and CSI-2 receiver drivers. I don't expect
> > this to be a workable approach.
> > 
> > Instead what I'm proposing is to use specific metadata formats on the
> > sensor devices only, on internal pads (more about those soon) of the
> > sensors, only visible in the UAPI, and then generic mbus formats along the
> 
> What do you mean with "only visible in the UAPI"?

Other drivers won't bother with specific metadata formats: they are only
present on the internal pads while external pads have generic formats.

> 
> > pipeline and finally generic V4L2 metadata formats on the DMAs (specific
> > to bit depth and packing). This would unsnarl the two, defining what data
> > there is (specific mbus code) and how that is transported and packed
> > (generic mbus codes and V4L2 formats).
> > 
> > The user space would be required to "know" the path of that data from the
> > sensor's internal pad to the V4L2 video node. I do not see this as these
> > devices require at least some knowledge of the pipeline, i.e. hardware at
> > hand. Separating what the data means and how it is packed may even be
> > beneficial: it allows separating code that interprets the data (sensor
> > internal mbus code) from the code that accesses it (packing).
> > 
> > These formats are in practice line based, meaning that there may be
> > padding at the end of the line, depending on the bus as well as the DMA.
> > If non-line based formats are needed, it is always possible to set the
> > "height" field to 1.
> > 
> > The internal (source) pads are an alternative to source routes [1]. The
> > source routes were not universally liked and I do have to say I like
> > re-using existing interface concepts (pads and everything you can do with
> > pads, including access format, selections etc.) wherever it makes sense,
> > instead of duplicating functionality.
> > 
> > Effectively internal source pads behave mostly just like sink pads, but
> > they describe a flow of data that originates from a sub-device instead of
> > arriving to a sub-device. The SUBDEV_S_ROUTING IOCTLs are used to enable
> > and disable routes from internal source pads to sub-device's source pads.
> > The subdev format IOCTLs are usable, too, so one can find which subdev
> > format is available on given internal source pad.
> 
> I think the internal pads require a bit more praise, as they can be used for
> other things too. E.g. the ds90ub953 FPD-Link serializer has a test pattern
> generator, which can be modeled very nicely with internal pads. The internal
> pad represents the TPG, and the user can use routing to choose if the output
> of the device is sourced from the normal input or from the TPG. And one can
> set the format on the TPG pad, thus configuring the TPG.

Well, yes, indeed.

Could you review especially the documentation patches to ensure we're
aligned on this?

> 
> > This set depends on these patches:
> > 
> > <URL:https://lore.kernel.org/linux-media/20231002105557.28972-1-sakari.ailus@linux.intel.com/T/#t>
> 
> Hmm, it's a bit odd for a generic series to depend on a device specific
> series. That makes backporting these more difficult. Why do these depend on
> ov2740 and css patches?

Patchset-wise that is the dependency, individual patches may be backported
without backporting _all_ driver patches in the previous set. However, if
you need those drivers as well, then you'll need to backport these patches,
too.

> 
> > I've also pushed these here and I'll keep updating the branch, I've also
> > included untested OV2740 patches:
> > 
> > <URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=metadata>
> > 
> > Questions and comments are most welcome.
> > 
> > Preliminary media-ctl and yavta patches can be found here:
> > 
> > <URL:https://git.retiisi.eu/?p=~sailus/yavta.git;a=shortlog;h=refs/heads/metadata>
> > <URL:https://git.retiisi.eu/?p=~sailus/v4l-utils.git;a=shortlog;h=refs/heads/metadata>
> > 
> > I have used IMX219 as an example on routing in a sensor driver in this
> > version. I also hope I can add OV2740 support in the next version after
> > testing the patches.
> 
> Sorry, I didn't understand that. You have used imx219 for what, where?

In an example only. I'm expecting Laurent to come up with actual driver
patches I can add to the set later (or merge at the same time at least).
:-)

> 
>  Tomi
> 
> > [1] <URL:https://lore.kernel.org/linux-media/20220831141357.1396081-20-tomi.valkeinen@ideasonboard.com/>
> > 
> > since v5:
> > 
> > - Rebase on new set of preparation patches.
> > 
> > - Switch CCS driver from s_stream to enable_streams/disable_streams. Keep
> >    streaming state information --- the sensor remains in streaming state if
> >    any of the streams is enabled.
> > 
> > - Fix setting mbus code on embedded data in get_frame_desc() op in the CCS
> >    driver.
> > 
> > since v4:
> > 
> > - Add a patch to acquire two sub-device states that may use the same lock.
> > 
> > - Add a patch for CCS driver to remove ccs_get_crop_compose() helper.
> > 
> > - Add a patch for CCS driver moving acquiring and releasing the mutex to
> >    the s_stream callback.
> > 
> > - Add a patch for CCS driver to rely on sub-device state locking using a
> >    single driver-provided lock.
> > 
> > - Fixed calculating minimum number of routes in copying the routes
> >    (thanks, Laurent).
> > 
> > - Moved a label in S_ROUTING handling to make Clang happy (hopefully).
> > 
> > - Fixed setting emb_data_ctrl register for CCS embedded data support.
> > 
> > - Rebase on Laurent's cleanup patches.
> > 
> > - Wrap a few long lines.
> > 
> > - Write in embedded data documentation sensor drivers generally don't
> >    allow configuring it.
> > 
> > since v3:
> > 
> > - Separate preparation patches from this set.
> > 
> > - Add a definition for "Data unit", a pixel that is not image data and use
> >    it instead in format documentation.
> > 
> > - Fix more numbered lists in dev-subdev.rst.
> > 
> > - Remove a redundant definition for V4L2_META_FMT_GENERIC_CSI2_2_24 ---
> >    V4L2_META_FMT_GENERIC_CSI2_12 can be used instead.
> > 
> > - Use "X" instead of "p" to denote padding in format documentation.
> > 
> > - Use IMX219 in examples instead of CCS.
> > 
> > - Document that the generic V4L2 CSI-2 metadata formats use padding
> >    defined in CSI-2 spec and packing defined in CCS spec.
> > 
> > - Add patches to align [GS]_ROUTING behaviour with V4L2. This means mainly
> >    returning configured routes as part of S_ROUTING as well. "len_routes"
> >    field is added to denote the length of the array and having more routes
> >    than fits in the array is no longer an error. Also added more reserved
> >    fields.
> > 
> > - Added trivial support for S_ROUTING (via G_ROUTING implementation) for
> >    use in drivers with static-only routes.
> > 
> > - Added helper functions to obtain mbus format as well as crop and compose
> >    rectangles that are streams-independent.
> > 
> > - Added a patch to define generic CSI-2 long packet types.
> > 
> > - Removed MEDIA_BUS_FMT_IS_META() macro. It didn't seem useful in the end.
> > 
> > - Use a single CCS embedded data format. The bit depth can be selected
> >    using the meta stream on the source pad.
> > 
> > - Fix mbus code numbers (there were holes due to removed redundant
> >    formats).
> > 
> > - Fix generic mbus code documentation (byte was being used instead of
> >    bit).
> > 
> > - Fix spelling of "length".
> > 
> > - Added a patch to remove v4l2_subdev_enable_streams_api that disables
> >    streams API. This should be merged once libcamera support for streams
> >    works nicely.
> > 
> > - Don't use strings in printing frame descriptor flags.
> > 
> > - Warn on string truncation in printing frame descriptor.
> > 
> > since v2:
> > 
> > - Add a better example, with formats.
> > 
> > - Add CCS static data media bus codes.
> > 
> > - Added an example demonstrating the use of internal pads. --- Is the
> >    level of detail enough for the purpose?
> > 
> > - Improved documentation.
> > 
> > - Added a macro to tell whether a format is a metadata format.
> >    (Documentation could be added.)
> > 
> > - A small ReST syntax fix in the same section.
> > 
> > - Drop leftovers of a patch checking for the INTERNAL_SOURCE flag.
> > 
> > since v1:
> > 
> > - Make the new pad flag just "INTERNAL", requiring either SINK or SOURCE
> >    pad flag to accompany it. Removed the union in struct v4l2_subdev_route.
> > 
> > - Add the term "stream" to MC glossary.
> > 
> > - Improved and fixed documentation (according to comments).
> > 
> > - Note these formats are little endian.
> > 
> > - Remove 1X8 from the names of the mbus codes. These formats have generally
> >    8 bits per pixel.
> > 
> > - Fix mbus code numbering (had holes in RFC).
> > 
> > - Add new metadata fields to debug prints.
> > 
> > - Fix a minor documentation build issue.
> > 
> > Sakari Ailus (28):
> >    media: mc: Add INTERNAL pad flag
> >    media: uapi: Add generic serial metadata mbus formats
> >    media: uapi: Document which mbus format fields are valid for metadata
> >    media: uapi: Add generic 8-bit metadata format definitions
> >    media: v4l: Support line-based metadata capture
> >    media: uapi: ccs: Add media bus code for MIPI CCS embedded data
> >    media: Documentation: ccs: Document routing
> >    media: Documentation: Additional streams generally don't harm capture
> >    media: Documentation: Document embedded data guidelines for camera
> >      sensors
> >    media: Documentation: v4l: Document source routes
> >    media: Documentation: Document S_ROUTING behaviour
> >    media: v4l: subdev: Add helpers for format, crop and compose pointers
> >    media: v4l: subdev: Add a function to lock two sub-device states, use
> >      it
> >    media: v4l: subdev: Move G_ROUTING handling below S_ROUTING
> >    media: v4l: subdev: Copy argument back to user also for S_ROUTING
> >    media: v4l: subdev: Add len_routes field to struct v4l2_subdev_routing
> >    media: v4l: subdev: Return routes set using S_ROUTING
> >    media: uapi: Allow a larger number of routes than there's room for
> >    media: v4l: subdev: Add trivial set_routing support
> >    media: uapi: v4l: subdev: Enable streams API
> >    media: ccs: No need to set streaming to false in power off
> >    media: ccs: Use {enable,disable}_streams operations
> >    media: ccs: Track streaming state
> >    media: ccs: Move ccs_validate_csi_data_format up
> >    media: ccs: Support frame descriptors
> >    media: ccs: Add support for embedded data stream
> >    media: ccs: Remove ccs_get_crop_compose helper
> >    media: ccs: Rely on sub-device state locking
> > 
> >   .../media/drivers/camera-sensor.rst           |  28 +
> >   .../userspace-api/media/drivers/ccs.rst       |  34 +-
> >   .../userspace-api/media/glossary.rst          |  14 +
> >   .../media/mediactl/media-types.rst            |   6 +
> >   .../userspace-api/media/v4l/dev-meta.rst      |  15 +
> >   .../userspace-api/media/v4l/dev-subdev.rst    | 208 ++++-
> >   .../userspace-api/media/v4l/meta-formats.rst  |   1 +
> >   .../media/v4l/metafmt-generic.rst             | 304 +++++++
> >   .../media/v4l/subdev-formats.rst              | 288 ++++++
> >   .../media/v4l/vidioc-enum-fmt.rst             |   7 +
> >   .../media/v4l/vidioc-subdev-g-routing.rst     |  40 +-
> >   .../media/videodev2.h.rst.exceptions          |   1 +
> >   drivers/media/i2c/ccs/ccs-core.c              | 859 ++++++++++++------
> >   drivers/media/i2c/ccs/ccs-quirk.h             |   7 +
> >   drivers/media/i2c/ccs/ccs.h                   |  23 +-
> >   drivers/media/mc/mc-entity.c                  |  10 +-
> >   drivers/media/v4l2-core/v4l2-ioctl.c          |  19 +-
> >   drivers/media/v4l2-core/v4l2-subdev.c         | 145 +--
> >   include/media/v4l2-subdev.h                   |  98 ++
> >   include/uapi/linux/media-bus-format.h         |  12 +
> >   include/uapi/linux/media.h                    |   1 +
> >   include/uapi/linux/v4l2-mediabus.h            |  18 +-
> >   include/uapi/linux/v4l2-subdev.h              |   8 +-
> >   include/uapi/linux/videodev2.h                |  18 +
> >   24 files changed, 1799 insertions(+), 365 deletions(-)
> >   create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > 
> > 
> > base-commit: 0fa78064f6e3a354616fb24462864900c0db3191
> 

-- 
Regards,

Sakari Ailus

Re: [PATCH v6 01/28] media: mc: Add INTERNAL pad flag

[Hans Verkuil]

On 03/10/2023 13:52, Sakari Ailus wrote:
> Internal source pads will be used as routing endpoints in V4L2
> [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
> 
> Also prevent creating links to pads that have been flagged as internal and
> initialising source pads with internal flag set.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  Documentation/userspace-api/media/glossary.rst         |  6 ++++++
>  .../userspace-api/media/mediactl/media-types.rst       |  6 ++++++
>  drivers/media/mc/mc-entity.c                           | 10 ++++++++--
>  include/uapi/linux/media.h                             |  1 +
>  4 files changed, 21 insertions(+), 2 deletions(-)
> 
> diff --git a/Documentation/userspace-api/media/glossary.rst b/Documentation/userspace-api/media/glossary.rst
> index 96a360edbf3b..f7b99a4527c7 100644
> --- a/Documentation/userspace-api/media/glossary.rst
> +++ b/Documentation/userspace-api/media/glossary.rst
> @@ -173,6 +173,12 @@ Glossary
>  	An integrated circuit that integrates all components of a computer
>  	or other electronic systems.
>  
> +_media-glossary-stream:
> +    Stream
> +	A distinct flow of data (image data or metadata) over a media pipeline
> +	from source to sink. A source may be e.g. an image sensor and a sink
> +	e.g. a memory buffer.

Hmm, I think this is a bit confusing. I think it would be better to replace
"from source to sink" with "from the initial source to the final sink".

The original text doesn't make it clear that there can be many hops in between.

So also: "A source" -> "An initial source", and "a sink" -> "a final sink".

Note that "media pipeline" isn't defined either. Should that be added?

Finally, I think this should be a separate patch as it has nothing to do with
adding the INTERNAL pad flag.

> +
>      V4L2 API
>  	**V4L2 userspace API**
>  
> diff --git a/Documentation/userspace-api/media/mediactl/media-types.rst b/Documentation/userspace-api/media/mediactl/media-types.rst
> index 0ffeece1e0c8..28941da27790 100644
> --- a/Documentation/userspace-api/media/mediactl/media-types.rst
> +++ b/Documentation/userspace-api/media/mediactl/media-types.rst
> @@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
>  .. _MEDIA-PAD-FL-SINK:
>  .. _MEDIA-PAD-FL-SOURCE:
>  .. _MEDIA-PAD-FL-MUST-CONNECT:
> +.. _MEDIA-PAD-FL-INTERNAL:
>  
>  .. flat-table:: Media pad flags
>      :header-rows:  0
> @@ -382,6 +383,11 @@ Types and flags used to represent the media graph elements
>  	  when this flag isn't set; the absence of the flag doesn't imply
>  	  there is none.
>  
> +    *  -  ``MEDIA_PAD_FL_INTERNAL``
> +       -  The internal flag indicates an internal pad that has no external
> +	  connections. Such a pad shall not be connected with a link. The
> +	  internal flag indicates that the :ref:``stream
> +	  <media-glossary-stream>`` either starts or ends in the entity.

This suggests that INTERNAL can be used for both sinks and sources, but...

>  
>  One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
>  must be set for every pad.
> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> index 543a392f8635..f5f290781021 100644
> --- a/drivers/media/mc/mc-entity.c
> +++ b/drivers/media/mc/mc-entity.c
> @@ -213,7 +213,9 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
>  		iter->index = i++;
>  
>  		if (hweight32(iter->flags & (MEDIA_PAD_FL_SINK |
> -					     MEDIA_PAD_FL_SOURCE)) != 1) {
> +					     MEDIA_PAD_FL_SOURCE)) != 1 ||
> +		    (iter->flags & MEDIA_PAD_FL_INTERNAL &&
> +		     !(iter->flags & MEDIA_PAD_FL_SINK))) {

...this appears to limit it to just sinks.

Regards,

	Hans

>  			ret = -EINVAL;
>  			break;
>  		}
> @@ -1075,7 +1077,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
>  
>  	for (i = 0; i < entity->num_pads; i++) {
>  		if ((entity->pads[i].flags &
> -		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
> +		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
> +		      MEDIA_PAD_FL_INTERNAL)) != pad_type)
>  			continue;
>  
>  		if (entity->pads[i].sig_type == sig_type)
> @@ -1100,6 +1103,9 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
>  		return -EINVAL;
>  	if (WARN_ON(!(sink->pads[sink_pad].flags & MEDIA_PAD_FL_SINK)))
>  		return -EINVAL;
> +	if (WARN_ON(source->pads[source_pad].flags & MEDIA_PAD_FL_INTERNAL) ||
> +	    WARN_ON(source->pads[sink_pad].flags & MEDIA_PAD_FL_INTERNAL))
> +		return -EINVAL;
>  
>  	link = media_add_link(&source->links);
>  	if (link == NULL)
> diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
> index 1c80b1d6bbaf..80cfd12a43fc 100644
> --- a/include/uapi/linux/media.h
> +++ b/include/uapi/linux/media.h
> @@ -208,6 +208,7 @@ struct media_entity_desc {
>  #define MEDIA_PAD_FL_SINK			(1U << 0)
>  #define MEDIA_PAD_FL_SOURCE			(1U << 1)
>  #define MEDIA_PAD_FL_MUST_CONNECT		(1U << 2)
> +#define MEDIA_PAD_FL_INTERNAL			(1U << 3)
>  
>  struct media_pad_desc {
>  	__u32 entity;		/* entity ID */

Re: [PATCH v6 05/28] media: v4l: Support line-based metadata capture

[Hans Verkuil]

On 03/10/2023 13:52, Sakari Ailus wrote:
> many camera sensors, among other devices, transmit embedded data and image
> data for each CSI-2 frame. This embedded data typically contains register
> configuration of the sensor that has been used to capture the image data
> of the same frame.
> 
> The embedded data is received by the CSI-2 receiver and has the same
> properties as the image data, including that it is line based: it has
> width, height and bytesperline (stride).
> 
> Add these fields to struct v4l2_meta_format and document them.
> 
> Also add V4L2_FMT_FLAG_META_LINE_BASED to tell a given format is
> line-based i.e. these fields of struct v4l2_meta_format are valid for it.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../userspace-api/media/v4l/dev-meta.rst          | 15 +++++++++++++++
>  .../userspace-api/media/v4l/vidioc-enum-fmt.rst   |  7 +++++++
>  .../media/videodev2.h.rst.exceptions              |  1 +
>  drivers/media/v4l2-core/v4l2-ioctl.c              |  5 +++--
>  include/uapi/linux/videodev2.h                    | 10 ++++++++++
>  5 files changed, 36 insertions(+), 2 deletions(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/dev-meta.rst b/Documentation/userspace-api/media/v4l/dev-meta.rst
> index 0e7e1ee1471a..4dfd79e0a705 100644
> --- a/Documentation/userspace-api/media/v4l/dev-meta.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-meta.rst
> @@ -65,3 +65,18 @@ to 0.
>        - ``buffersize``
>        - Maximum buffer size in bytes required for data. The value is set by the
>          driver.
> +    * - __u32
> +      - ``width``
> +      - Width of a line of metadata in Data units. Valid when
> +	:c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set,
> +	otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
> +    * - __u32
> +      - ``height``
> +      - Number of rows of metadata. Valid when :c:type`v4l2_fmtdesc` flag
> +	``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
> +	:c:func:`VIDIOC_ENUM_FMT`.
> +    * - __u32
> +      - ``bytesperline``
> +      - Offset in bytes between the beginning of two consecutive lines. Valid
> +	when :c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is
> +	set, otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> index 000c154b0f98..a79abf4428c8 100644
> --- a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> +++ b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> @@ -227,6 +227,13 @@ the ``mbus_code`` field is handled differently:
>  	The application can ask to configure the quantization of the capture
>  	device when calling the :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl with
>  	:ref:`V4L2_PIX_FMT_FLAG_SET_CSC <v4l2-pix-fmt-flag-set-csc>` set.
> +    * - ``V4L2_FMT_FLAG_META_LINE_BASED``
> +      - 0x0200
> +      - The metadata format is line-based. In this case the ``width``,
> +	``height`` and ``bytesperline`` fields of :c:type:`v4l2_meta_format` are
> +	valid. The buffer consists of ``height`` lines, each having ``width``
> +	Data units of data and offset (in bytes) between the beginning of each

offset -> the offset

Would 'stride' be a better name for this? The v4l2_pix_format documentation for
this field says: "Distance in bytes between the leftmost pixels in two adjacent lines."

> +	two consecutive lines is ``bytesperline``.
>  
>  Return Value
>  ============
> diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b/Documentation/userspace-api/media/videodev2.h.rst.exceptions
> index 3e58aac4ef0b..bdc628e8c1d6 100644
> --- a/Documentation/userspace-api/media/videodev2.h.rst.exceptions
> +++ b/Documentation/userspace-api/media/videodev2.h.rst.exceptions
> @@ -215,6 +215,7 @@ replace define V4L2_FMT_FLAG_CSC_XFER_FUNC fmtdesc-flags
>  replace define V4L2_FMT_FLAG_CSC_YCBCR_ENC fmtdesc-flags
>  replace define V4L2_FMT_FLAG_CSC_HSV_ENC fmtdesc-flags
>  replace define V4L2_FMT_FLAG_CSC_QUANTIZATION fmtdesc-flags
> +replace define V4L2_FMT_FLAG_META_LINE_BASED fmtdesc-flags
>  
>  # V4L2 timecode types
>  replace define V4L2_TC_TYPE_24FPS timecode-type
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index ce4b3929ff5f..fb453b7d0c91 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -343,8 +343,9 @@ static void v4l_print_format(const void *arg, bool write_only)
>  	case V4L2_BUF_TYPE_META_OUTPUT:
>  		meta = &p->fmt.meta;
>  		pixelformat = meta->dataformat;
> -		pr_cont(", dataformat=%p4cc, buffersize=%u\n",
> -			&pixelformat, meta->buffersize);
> +		pr_cont(", dataformat=%p4cc, buffersize=%u, width=%u, height=%u, bytesperline=%u\n",
> +			&pixelformat, meta->buffersize, meta->width,
> +			meta->height, meta->bytesperline);
>  		break;
>  	}
>  }
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index 2b16b06ad278..7b0781a20dbe 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -877,6 +877,7 @@ struct v4l2_fmtdesc {
>  #define V4L2_FMT_FLAG_CSC_YCBCR_ENC		0x0080
>  #define V4L2_FMT_FLAG_CSC_HSV_ENC		V4L2_FMT_FLAG_CSC_YCBCR_ENC
>  #define V4L2_FMT_FLAG_CSC_QUANTIZATION		0x0100
> +#define V4L2_FMT_FLAG_META_LINE_BASED		0x0200
>  
>  	/* Frame Size and frame rate enumeration */
>  /*
> @@ -2420,10 +2421,19 @@ struct v4l2_sdr_format {
>   * struct v4l2_meta_format - metadata format definition
>   * @dataformat:		little endian four character code (fourcc)
>   * @buffersize:		maximum size in bytes required for data
> + * @width:		number of data units of data per line (valid for line

I'd drop "of data".

> + *			based formats only, see format documentation)
> + * @height:		number of lines of data per buffer (valid for line based

Ditto.

> + *			formats only)
> + * @bytesperline:	offset between the beginnings of two adjacent lines in
> + *			bytes (valid for line based formats only)

I would use 'distance' instead of 'offset'.

>   */
>  struct v4l2_meta_format {
>  	__u32				dataformat;
>  	__u32				buffersize;
> +	__u32				width;
> +	__u32				height;
> +	__u32				bytesperline;
>  } __attribute__ ((packed));
>  
>  /**

Regards,

	Hans

Re: [PATCH v6 09/28] media: Documentation: Document embedded data guidelines for camera sensors

[Hans Verkuil]

On 03/10/2023 13:52, Sakari Ailus wrote:
> Document how embedded data support should be implemented for camera
> sensors, and when and how CCS embedded data format should be referenced.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../media/drivers/camera-sensor.rst           | 28 +++++++++++++++++++
>  1 file changed, 28 insertions(+)
> 
> diff --git a/Documentation/userspace-api/media/drivers/camera-sensor.rst b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> index 919a50e8b9d9..308f391c5ca1 100644
> --- a/Documentation/userspace-api/media/drivers/camera-sensor.rst
> +++ b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> @@ -102,3 +102,31 @@ register programming sequences shall initialize the :ref:`V4L2_CID_HFLIP
>  values programmed by the register sequences. The default values of these
>  controls shall be 0 (disabled). Especially these controls shall not be inverted,
>  independently of the sensor's mounting rotation.
> +
> +Embedded data
> +-------------
> +
> +Many sensors, mostly raw sensors, support embedded data which is used to convey
> +the sensor configuration for the captured frame back to the host. While CSI-2 is
> +the most common bus used by such sensors, embedded data is not entirely limited
> +to CSI-2 bus due to e.g. bridge devices.

I'm not quite sure what you mean with "embedded data is not entirely limited
to CSI-2 bus due to e.g. bridge devices": do you just want to say: "embedded data
can be available on other bus types as well."?

> +
> +Embedded data support should use an internal source pad and route to the

should or shall/must?

Regards,

	Hans

> +external pad. If embedded data output can be disabled in hardware, it should be
> +possible to disable the embedded data route via ``VIDIOC_SUBDEV_S_ROUTING``
> +IOCTL.
> +
> +In general, changing the embedded data format from the driver-configured values
> +is not supported. The height of the metadata is hardware specific and the width
> +is that (or less of that) of the image width, as configured on the pixel data
> +stream.
> +
> +CCS and non-CCS embedded data
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Embedded data which is compliant with CCS definitions shall use ``CCS embedded
> +data format <MEDIA-BUS-FMT-CCS-EMBEDDED>``. Device specific embedded data which
> +is compliant up to MIPI CCS embedded data levels 1 or 2 only shall refer to CCS
> +embedded data formats and document the level of conformance. The rest of the
> +device specific embedded data format shall be documented in the context of the
> +data format itself.

Re: [PATCH v6 09/28] media: Documentation: Document embedded data guidelines for camera sensors

[Hans Verkuil]

On 03/10/2023 13:52, Sakari Ailus wrote:
> Document how embedded data support should be implemented for camera
> sensors, and when and how CCS embedded data format should be referenced.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../media/drivers/camera-sensor.rst           | 28 +++++++++++++++++++
>  1 file changed, 28 insertions(+)
> 
> diff --git a/Documentation/userspace-api/media/drivers/camera-sensor.rst b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> index 919a50e8b9d9..308f391c5ca1 100644
> --- a/Documentation/userspace-api/media/drivers/camera-sensor.rst
> +++ b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> @@ -102,3 +102,31 @@ register programming sequences shall initialize the :ref:`V4L2_CID_HFLIP
>  values programmed by the register sequences. The default values of these
>  controls shall be 0 (disabled). Especially these controls shall not be inverted,
>  independently of the sensor's mounting rotation.
> +
> +Embedded data
> +-------------
> +
> +Many sensors, mostly raw sensors, support embedded data which is used to convey
> +the sensor configuration for the captured frame back to the host. While CSI-2 is
> +the most common bus used by such sensors, embedded data is not entirely limited
> +to CSI-2 bus due to e.g. bridge devices.
> +
> +Embedded data support should use an internal source pad and route to the

"internal source pad" -> "internal sink pad"?

> +external pad. If embedded data output can be disabled in hardware, it should be

"external pad" -> "external source pad"?

Or perhaps "a source pad of the entity"?

Regards,

	Hans

> +possible to disable the embedded data route via ``VIDIOC_SUBDEV_S_ROUTING``
> +IOCTL.
> +
> +In general, changing the embedded data format from the driver-configured values
> +is not supported. The height of the metadata is hardware specific and the width
> +is that (or less of that) of the image width, as configured on the pixel data
> +stream.
> +
> +CCS and non-CCS embedded data
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Embedded data which is compliant with CCS definitions shall use ``CCS embedded
> +data format <MEDIA-BUS-FMT-CCS-EMBEDDED>``. Device specific embedded data which
> +is compliant up to MIPI CCS embedded data levels 1 or 2 only shall refer to CCS
> +embedded data formats and document the level of conformance. The rest of the
> +device specific embedded data format shall be documented in the context of the
> +data format itself.

Re: [PATCH v6 01/28] media: mc: Add INTERNAL pad flag

[Sakari Ailus]

Hi Hans,

Thanks for the review!

On Thu, Oct 05, 2023 at 11:52:08AM +0200, Hans Verkuil wrote:
> On 03/10/2023 13:52, Sakari Ailus wrote:
> > Internal source pads will be used as routing endpoints in V4L2
> > [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
> > 
> > Also prevent creating links to pads that have been flagged as internal and
> > initialising source pads with internal flag set.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  Documentation/userspace-api/media/glossary.rst         |  6 ++++++
> >  .../userspace-api/media/mediactl/media-types.rst       |  6 ++++++
> >  drivers/media/mc/mc-entity.c                           | 10 ++++++++--
> >  include/uapi/linux/media.h                             |  1 +
> >  4 files changed, 21 insertions(+), 2 deletions(-)
> > 
> > diff --git a/Documentation/userspace-api/media/glossary.rst b/Documentation/userspace-api/media/glossary.rst
> > index 96a360edbf3b..f7b99a4527c7 100644
> > --- a/Documentation/userspace-api/media/glossary.rst
> > +++ b/Documentation/userspace-api/media/glossary.rst
> > @@ -173,6 +173,12 @@ Glossary
> >  	An integrated circuit that integrates all components of a computer
> >  	or other electronic systems.
> >  
> > +_media-glossary-stream:
> > +    Stream
> > +	A distinct flow of data (image data or metadata) over a media pipeline
> > +	from source to sink. A source may be e.g. an image sensor and a sink
> > +	e.g. a memory buffer.
> 
> Hmm, I think this is a bit confusing. I think it would be better to replace
> "from source to sink" with "from the initial source to the final sink".

Seems fine to me. I'll use that in v7.

> 
> The original text doesn't make it clear that there can be many hops in
> between.
> 
> So also: "A source" -> "An initial source", and "a sink" -> "a final sink".
> 
> Note that "media pipeline" isn't defined either. Should that be added?

I can add that, too...

> 
> Finally, I think this should be a separate patch as it has nothing to do with
> adding the INTERNAL pad flag.

I agree, will split in v7.

> 
> > +
> >      V4L2 API
> >  	**V4L2 userspace API**
> >  
> > diff --git a/Documentation/userspace-api/media/mediactl/media-types.rst b/Documentation/userspace-api/media/mediactl/media-types.rst
> > index 0ffeece1e0c8..28941da27790 100644
> > --- a/Documentation/userspace-api/media/mediactl/media-types.rst
> > +++ b/Documentation/userspace-api/media/mediactl/media-types.rst
> > @@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
> >  .. _MEDIA-PAD-FL-SINK:
> >  .. _MEDIA-PAD-FL-SOURCE:
> >  .. _MEDIA-PAD-FL-MUST-CONNECT:
> > +.. _MEDIA-PAD-FL-INTERNAL:
> >  
> >  .. flat-table:: Media pad flags
> >      :header-rows:  0
> > @@ -382,6 +383,11 @@ Types and flags used to represent the media graph elements
> >  	  when this flag isn't set; the absence of the flag doesn't imply
> >  	  there is none.
> >  
> > +    *  -  ``MEDIA_PAD_FL_INTERNAL``
> > +       -  The internal flag indicates an internal pad that has no external
> > +	  connections. Such a pad shall not be connected with a link. The
> > +	  internal flag indicates that the :ref:``stream
> > +	  <media-glossary-stream>`` either starts or ends in the entity.
> 
> This suggests that INTERNAL can be used for both sinks and sources, but...

Right. The INTERNAL flag in UAPI shouldn't be limited to sources, but at
the same time we don't have use for it in sinks. I'd prefer to leave this
open in the future. We could of course say that explicitly.

> 
> >  
> >  One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
> >  must be set for every pad.
> > diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> > index 543a392f8635..f5f290781021 100644
> > --- a/drivers/media/mc/mc-entity.c
> > +++ b/drivers/media/mc/mc-entity.c
> > @@ -213,7 +213,9 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
> >  		iter->index = i++;
> >  
> >  		if (hweight32(iter->flags & (MEDIA_PAD_FL_SINK |
> > -					     MEDIA_PAD_FL_SOURCE)) != 1) {
> > +					     MEDIA_PAD_FL_SOURCE)) != 1 ||
> > +		    (iter->flags & MEDIA_PAD_FL_INTERNAL &&
> > +		     !(iter->flags & MEDIA_PAD_FL_SINK))) {
> 
> ...this appears to limit it to just sinks.

Yes, this is enforced so we won't get drivers that accidentally would use
the INTERNAL flag on source pads.

> 
> Regards,
> 
> 	Hans
> 
> >  			ret = -EINVAL;
> >  			break;
> >  		}
> > @@ -1075,7 +1077,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
> >  
> >  	for (i = 0; i < entity->num_pads; i++) {
> >  		if ((entity->pads[i].flags &
> > -		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
> > +		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
> > +		      MEDIA_PAD_FL_INTERNAL)) != pad_type)
> >  			continue;
> >  
> >  		if (entity->pads[i].sig_type == sig_type)
> > @@ -1100,6 +1103,9 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
> >  		return -EINVAL;
> >  	if (WARN_ON(!(sink->pads[sink_pad].flags & MEDIA_PAD_FL_SINK)))
> >  		return -EINVAL;
> > +	if (WARN_ON(source->pads[source_pad].flags & MEDIA_PAD_FL_INTERNAL) ||
> > +	    WARN_ON(source->pads[sink_pad].flags & MEDIA_PAD_FL_INTERNAL))
> > +		return -EINVAL;
> >  
> >  	link = media_add_link(&source->links);
> >  	if (link == NULL)
> > diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
> > index 1c80b1d6bbaf..80cfd12a43fc 100644
> > --- a/include/uapi/linux/media.h
> > +++ b/include/uapi/linux/media.h
> > @@ -208,6 +208,7 @@ struct media_entity_desc {
> >  #define MEDIA_PAD_FL_SINK			(1U << 0)
> >  #define MEDIA_PAD_FL_SOURCE			(1U << 1)
> >  #define MEDIA_PAD_FL_MUST_CONNECT		(1U << 2)
> > +#define MEDIA_PAD_FL_INTERNAL			(1U << 3)
> >  
> >  struct media_pad_desc {
> >  	__u32 entity;		/* entity ID */
> 

-- 
Regards,

Sakari Ailus

Re: [PATCH v6 10/28] media: Documentation: v4l: Document source routes

[Hans Verkuil]

On 03/10/2023 14:07, Sakari Ailus wrote:
> Document how internal pads are used on source routes. Use the IMX219
> camera sensor as an example.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../userspace-api/media/v4l/dev-subdev.rst    | 179 ++++++++++++++++++
>  1 file changed, 179 insertions(+)
> 
> diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> index a387e8a15b8d..fb73a95401c3 100644
> --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> @@ -553,6 +553,27 @@ A stream at a specific point in the media pipeline is identified by the
>  sub-device and a (pad, stream) pair. For sub-devices that do not support
>  multiplexed streams the 'stream' field is always 0.
>  
> +.. _v4l2-subdev-source-routes:
> +
> +Internal pads and source routes
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +Cases where a single sub-device source pad is traversed by multiple streams one

Comma after "streams"

> +or more of which originate from within the sub-device itself are special as

and comma after "itself".

> +there is no external sink pad for such routes. In those cases, the sources of
> +the internally generated streams are represented by internal sink pads, which
> +are sink pads that have the :ref:`MEDIA_PAD_FL_INTERNAL <MEDIA-PAD-FL-INTERNAL>`
> +pad flag set.
> +
> +Internal pads have all the properties of an external pad, including formats and

Perhaps the glossary should have definitions for "internal pad" and "external pad"?
Or define it somewhere in:

https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/mediactl/media-controller.html

> +selections. The format in this case is the source format of the stream. An
> +internal pad always has a single stream only (0).
> +
> +*Source routes* are routes from an internal sink pad to an external source
> +pad. In most cases source routes are not modifiable but they can be activated
> +and deactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +<v4l2-subdev-routing-flags>` flag, depending on driver capabilities.
> +
>  Interaction between routes, streams, formats and selections
>  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>  
> @@ -668,3 +689,161 @@ To configure this pipeline, the userspace must take the following steps:
>     the configurations along the stream towards the receiver, using
>     :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each
>     stream endpoint in each sub-device.
> +
> +Internal pads setup example
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +A simple example of a multiplexed stream setup might be as follows:
> +
> +- A CCS camera sensor source sub-device, with one sink pad (0), one source pad
> +  (1), an internal sink pad (2) that represents the source of embedded
> +  data. There are two routes, one from the sink pad to the source, and another
> +  from the internal sink pad to the source pad. The embedded data stream needs
> +  to be enabled by activating the related route. The configuration of the rest
> +  of the CCS sub-devices is omitted from this example.
> +
> +- Multiplexer bridge (Bridge). The bridge has one sink pad, connected to the
> +  sensor (pad 0), and one source pad (pad 1), which outputs two streams.

I think this should mention "CSI" somewhere, since this transmits over the CSI bus.

> +
> +- Receiver in the SoC (Receiver). The receiver has a single sink pad (pad 0),
> +  connected to the bridge, and two source pads (pads 1-2), going to the DMA
> +  engine. The receiver demultiplexes the incoming streams to the source pads.

Ditto: this receives data from the CSI bus.

> +
> +- DMA Engines in the SoC (DMA Engine), one for each stream. Each DMA engine is
> +  connected to a single source pad in the receiver.
> +
> +The sensor, the bridge and the receiver are modeled as V4L2 sub-devices,
> +exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are
> +modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
> +
> +To configure this pipeline, the userspace must take the following steps:
> +
> +1) Set up media links between entities: connect the sensors to the bridge,
> +   bridge to the receiver, and the receiver to the DMA engines. This step does
> +   not differ from normal non-multiplexed media controller setup.
> +
> +2) Configure routing
> +
> +.. flat-table:: Camera sensor
> +    :header-rows: 1
> +
> +    * - Sink Pad/Stream
> +      - Source Pad/Stream
> +      - Routing Flags
> +      - Comments
> +    * - 0/0
> +      - 1/0
> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +      - Pixel data stream from the sink pad
> +    * - 2/0
> +      - 1/1
> +      - **V4L2_SUBDEV_ROUTE_FL_ACTIVE**
> +      - Metadata stream from the internal sink pad
> +
> +.. flat-table:: Bridge routing table
> +    :header-rows: 1
> +
> +    * - Sink Pad/Stream
> +      - Source Pad/Stream
> +      - Routing Flags
> +      - Comments
> +    * - 0/0
> +      - 1/0
> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +      - Pixel data stream from camera sensor
> +    * - 0/1
> +      - 1/1
> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +      - Metadata stream from camera sensor
> +
> +.. flat-table:: Receiver routing table
> +    :header-rows:  1
> +
> +    * - Sink Pad/Stream
> +      - Source Pad/Stream
> +      - Routing Flags
> +      - Comments
> +    * - 0/0
> +      - 1/0
> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +      - Pixel data stream from camera sensor
> +    * - 0/1
> +      - 2/0
> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +      - Metadata stream from camera sensor
> +
> +The options available in sensor's routing configuration are dictated by
> +hardware capabilities: typically camera sensors always produce image data

image data -> an image data

> +stream while the embedded data stream typically can be either enabled or
> +disabled.
> +
> +3) Configure formats and selections
> +
> +   This example assumes that the formats are propagated from sink pad to the
> +   source pad as-is. The tables contain fields of both struct v4l2_subdev_format
> +   and struct v4l2_mbus_framefmt.
> +
> +.. flat-table:: Formats set on the sub-devices. Bold values are set, others are
> +                static or propagated.
> +    :header-rows: 1
> +    :fill-cells:
> +
> +    * - Sub-device
> +      - Pad/Stream
> +      - Width
> +      - Height
> +      - Code
> +    * - :rspan:`3` Camera sensor sub-device (IMX219)
> +      - 1/0
> +      - 3296
> +      - 2480
> +      - MEDIA_BUS_FMT_SRGGB10
> +    * - 0/0
> +      - **3296**
> +      - **2480**
> +      - **MEDIA_BUS_FMT_SRGGB10**
> +    * - 2/0
> +      - 3296
> +      - 2
> +      - MEDIA_BUS_FMT_IMX219_EMBEDDED
> +    * - 1/1
> +      - 3296
> +      - 2
> +      - MEDIA_BUS_FMT_META_10
> +    * - :rspan:`3` Bridge
> +      - 0/0
> +      - **3296**
> +      - **2480**
> +      - **MEDIA_BUS_FMT_SRGGB10**
> +    * - 1/0
> +      - 3296
> +      - 2480
> +      - MEDIA_BUS_FMT_SRGGB10
> +    * - 0/1
> +      - **3296**
> +      - **2**
> +      - **MEDIA_BUS_FMT_META_10**
> +    * - 1/1
> +      - 3296
> +      - 2
> +      - MEDIA_BUS_FMT_META_10
> +    * - :rspan:`3` Receiver
> +      - 0/0
> +      - **3296**
> +      - **2480**
> +      - **MEDIA_BUS_FMT_SRGGB10**
> +    * - 1/0
> +      - 3296
> +      - 2480
> +      - MEDIA_BUS_FMT_SRGGB10
> +    * - 0/1
> +      - **3296**
> +      - **2**
> +      - **MEDIA_BUS_FMT_META_10**
> +    * - 2/0
> +      - 3296
> +      - 2
> +      - MEDIA_BUS_FMT_META_10
> +
> +The embedded data format does not need to be configured as the format is
> +dictated by the pixel data format in this case.

"pixel data format"? That doesn't sound right for embedded data.

Regards,

	Hans

Re: [PATCH v6 15/28] media: v4l: subdev: Copy argument back to user also for S_ROUTING

[Hans Verkuil]

On 03/10/2023 14:08, Sakari Ailus wrote:
> As the user needs to know what went wrong for S_ROUTING, copy array
> arguments back to the user.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  drivers/media/v4l2-core/v4l2-ioctl.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index fb453b7d0c91..6921a72566df 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -3419,7 +3419,8 @@ video_usercopy(struct file *file, unsigned int orig_cmd, unsigned long arg,
>  	 * in case of failure, but it is not defined here as part of the
>  	 * 'v4l2_ioctls' array, insert an ad-hoc check to address that.
>  	 */
> -	if (err < 0 && !always_copy && cmd != VIDIOC_SUBDEV_G_ROUTING)
> +	if (err < 0 && !always_copy && cmd != VIDIOC_SUBDEV_G_ROUTING &&
> +	    cmd != VIDIOC_SUBDEV_S_ROUTING)
>  		goto out;
>  
>  	if (has_array_args) {

This is ugly.

How about this:

	if (cmd == VIDIOC_SUBDEV_G_ROUTING || cmd == VIDIOC_SUBDEV_S_ROUTING)
		always_copy = true;

	if (err < 0 && !always_copy)
		goto out;

The first 'if' can also be a 'switch', I have no preference.

This cmd check can also be done earlier in the function, right after
the call to video_get_user(). It might be a better place.

Regards,

	Hans

Re: [PATCH v6 16/28] media: v4l: subdev: Add len_routes field to struct v4l2_subdev_routing

[Hans Verkuil]

On 03/10/2023 14:08, Sakari Ailus wrote:
> The len_routes field is used to tell the size of the routes array in
> struct v4l2_subdev_routing. This way the number of routes returned from
> S_ROUTING IOCTL may be larger than the number of routes provided, in case
> there are more routes returned by the driver.
> 
> Note that this changes the (now-disabled) UAPI, users must be updated.

With "now-disabled" you mean "still disabled", right?

So:

"Note that this uAPI is still disabled in the code, so this change can
safely be done. Anyone who manually patched the code to enable this uAPI
must update their code."

> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../media/v4l/vidioc-subdev-g-routing.rst     | 31 ++++++++++++-------
>  drivers/media/v4l2-core/v4l2-ioctl.c          |  4 +--
>  drivers/media/v4l2-core/v4l2-subdev.c         |  6 +++-
>  include/media/v4l2-subdev.h                   |  2 ++
>  include/uapi/linux/v4l2-subdev.h              |  8 +++--
>  5 files changed, 34 insertions(+), 17 deletions(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> index 72677a280cd6..9a9765ddc316 100644
> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> @@ -46,20 +46,26 @@ with the ``VIDIOC_SUBDEV_S_ROUTING`` ioctl, by adding or removing routes and
>  setting or clearing flags of the  ``flags`` field of a
>  struct :c:type:`v4l2_subdev_route`.
>  
> -All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called. This
> -means that the userspace must reconfigure all streams after calling the ioctl
> -with e.g. ``VIDIOC_SUBDEV_S_FMT``.
> +All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is
> +called. This means that the userspace must reconfigure all streams after calling
> +the ioctl with e.g. ``VIDIOC_SUBDEV_S_FMT``.
>  
>  Only subdevices which have both sink and source pads can support routing.
>  
> -When inspecting routes through ``VIDIOC_SUBDEV_G_ROUTING`` and the application
> -provided ``num_routes`` is not big enough to contain all the available routes
> -the subdevice exposes, drivers return the ENOSPC error code and adjust the
> -value of the ``num_routes`` field. Application should then reserve enough memory
> -for all the route entries and call ``VIDIOC_SUBDEV_G_ROUTING`` again.
> +The ``num_routes`` field is used to denote the number of routes set (set by user
> +space on ``VIDIOC_SUBDEV_S_ROUTING`` argument) on the routing table as well as
> +the number of routes returned back from both IOCTLs. The ``len_routes``
> +signifies the number of routes that can fit into the ``routes`` array. The
> +userspace shall set ``len_routes`` for both IOCTLs and ``num_routes`` for
> +``VIDIOC_SUBDEV_S_ROUTING``.
>  
> -On a successful ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the
> -``num_routes`` field to reflect the actual number of routes returned.
> +On a ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the ``num_routes``
> +field to reflect the actual number of routes known by the driver. ``num_routes``
> +larger than ``len_routes`` in both IOCTLs. In this ``len_routes`` were returned
> +back to the userspace. This is not an error.

This paragraph is garbled.

> +
> +Also ``VIDIOC_SUBDEV_S_ROUTING`` may return more route than the user provided in
> +``num_routes`` field due to e.g. hardware properties.
>  
>  .. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
>  
> @@ -74,6 +80,9 @@ On a successful ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the
>        - ``which``
>        - Format to modified, from enum
>          :ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
> +    * - __u32
> +      - ``len_routes``
> +      - The length of the array (as in memory reserved for the array)

So is this in bytes or in number of route entries?

I think 'len_routes' is a terribly confusing name.

How about 'max_num_routes'? Or 'max_route_elems/entries'?

>      * - struct :c:type:`v4l2_subdev_route`
>        - ``routes[]``
>        - Array of struct :c:type:`v4l2_subdev_route` entries
> @@ -81,7 +90,7 @@ On a successful ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the
>        - ``num_routes``
>        - Number of entries of the routes array

This is now a very confusing field description.

How about:

'max_num_routes': Maximum number of entries that can fit in the routes array
'num_routes': Actual number of entries stored in the routes array.

>      * - __u32
> -      - ``reserved``\ [5]
> +      - ``reserved``\ [11]
>        - Reserved for future extensions. Applications and drivers must set
>  	the array to zero.
>  
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index 6921a72566df..1e3da9d64958 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -3155,13 +3155,13 @@ static int check_array_args(unsigned int cmd, void *parg, size_t *array_size,
>  	case VIDIOC_SUBDEV_S_ROUTING: {
>  		struct v4l2_subdev_routing *routing = parg;
>  
> -		if (routing->num_routes > 256)
> +		if (routing->len_routes > 256)
>  			return -E2BIG;
>  
>  		*user_ptr = u64_to_user_ptr(routing->routes);
>  		*kernel_ptr = (void **)&routing->routes;
>  		*array_size = sizeof(struct v4l2_subdev_route)
> -			    * routing->num_routes;
> +			    * routing->len_routes;
>  		ret = 1;
>  		break;
>  	}
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 614ff0031831..bd1e8205913c 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -903,6 +903,9 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>  		if (routing->which != V4L2_SUBDEV_FORMAT_TRY && ro_subdev)
>  			return -EPERM;
>  
> +		if (routing->num_routes > routing->len_routes)
> +			return -EINVAL;
> +
>  		memset(routing->reserved, 0, sizeof(routing->reserved));
>  
>  		for (i = 0; i < routing->num_routes; ++i) {
> @@ -929,6 +932,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>  		}
>  
>  		krouting.num_routes = routing->num_routes;
> +		krouting.len_routes = routing->len_routes;
>  		krouting.routes = routes;
>  
>  		return v4l2_subdev_call(sd, pad, set_routing, state,
> @@ -949,7 +953,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>  
>  		krouting = &state->routing;
>  
> -		if (routing->num_routes < krouting->num_routes) {
> +		if (routing->len_routes < krouting->num_routes) {
>  			routing->num_routes = krouting->num_routes;
>  			return -ENOSPC;
>  		}
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index e49e8af2fb52..baaa81a9497e 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -741,12 +741,14 @@ struct v4l2_subdev_stream_configs {
>  /**
>   * struct v4l2_subdev_krouting - subdev routing table
>   *
> + * @len_routes: length of routes array, in routes
>   * @num_routes: number of routes
>   * @routes: &struct v4l2_subdev_route
>   *
>   * This structure contains the routing table for a subdev.
>   */
>  struct v4l2_subdev_krouting {
> +	unsigned int len_routes;
>  	unsigned int num_routes;
>  	struct v4l2_subdev_route *routes;
>  };
> diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
> index 4a195b68f28f..b57fb89caa9e 100644
> --- a/include/uapi/linux/v4l2-subdev.h
> +++ b/include/uapi/linux/v4l2-subdev.h
> @@ -222,15 +222,17 @@ struct v4l2_subdev_route {
>   * struct v4l2_subdev_routing - Subdev routing information
>   *
>   * @which: configuration type (from enum v4l2_subdev_format_whence)
> - * @num_routes: the total number of routes in the routes array
> + * @len_routes: the length of the routes array, in routes
>   * @routes: pointer to the routes array
> + * @num_routes: the total number of routes in the routes array
>   * @reserved: drivers and applications must zero this array
>   */
>  struct v4l2_subdev_routing {
>  	__u32 which;
> -	__u32 num_routes;
> +	__u32 len_routes;
>  	__u64 routes;
> -	__u32 reserved[6];
> +	__u32 num_routes;
> +	__u32 reserved[11];
>  };
>  
>  /*

I'm going with a NACK for the 'len_routes' name, it's a really, really bad name.

So:

Nacked-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>

Regards,

	Hans

Re: [PATCH v6 01/28] media: mc: Add INTERNAL pad flag

[Tomi Valkeinen]

On 03/10/2023 14:52, Sakari Ailus wrote:
> Internal source pads will be used as routing endpoints in V4L2
> [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.

It would probably be good to explain here and in the comments/docs, that 
a sink pad is a source pad =). When you say above "internal source pad", 
it's actually MEDIA_PAD_FL_INTERNAL | MEDIA_PAD_FL_SINK. I think this 
will confuse people time and time again, so it's probably good to 
explain it in as many places as possible.

> Also prevent creating links to pads that have been flagged as internal and
> initialising source pads with internal flag set.
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>   Documentation/userspace-api/media/glossary.rst         |  6 ++++++
>   .../userspace-api/media/mediactl/media-types.rst       |  6 ++++++
>   drivers/media/mc/mc-entity.c                           | 10 ++++++++--
>   include/uapi/linux/media.h                             |  1 +
>   4 files changed, 21 insertions(+), 2 deletions(-)
> 
> diff --git a/Documentation/userspace-api/media/glossary.rst b/Documentation/userspace-api/media/glossary.rst
> index 96a360edbf3b..f7b99a4527c7 100644
> --- a/Documentation/userspace-api/media/glossary.rst
> +++ b/Documentation/userspace-api/media/glossary.rst
> @@ -173,6 +173,12 @@ Glossary
>   	An integrated circuit that integrates all components of a computer
>   	or other electronic systems.
>   
> +_media-glossary-stream:
> +    Stream
> +	A distinct flow of data (image data or metadata) over a media pipeline
> +	from source to sink. A source may be e.g. an image sensor and a sink
> +	e.g. a memory buffer.
> +
>       V4L2 API
>   	**V4L2 userspace API**
>   
> diff --git a/Documentation/userspace-api/media/mediactl/media-types.rst b/Documentation/userspace-api/media/mediactl/media-types.rst
> index 0ffeece1e0c8..28941da27790 100644
> --- a/Documentation/userspace-api/media/mediactl/media-types.rst
> +++ b/Documentation/userspace-api/media/mediactl/media-types.rst
> @@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
>   .. _MEDIA-PAD-FL-SINK:
>   .. _MEDIA-PAD-FL-SOURCE:
>   .. _MEDIA-PAD-FL-MUST-CONNECT:
> +.. _MEDIA-PAD-FL-INTERNAL:
>   
>   .. flat-table:: Media pad flags
>       :header-rows:  0
> @@ -382,6 +383,11 @@ Types and flags used to represent the media graph elements
>   	  when this flag isn't set; the absence of the flag doesn't imply
>   	  there is none.
>   
> +    *  -  ``MEDIA_PAD_FL_INTERNAL``
> +       -  The internal flag indicates an internal pad that has no external
> +	  connections. Such a pad shall not be connected with a link. The
> +	  internal flag indicates that the :ref:``stream
> +	  <media-glossary-stream>`` either starts or ends in the entity.
>   
>   One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
>   must be set for every pad.
> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> index 543a392f8635..f5f290781021 100644
> --- a/drivers/media/mc/mc-entity.c
> +++ b/drivers/media/mc/mc-entity.c
> @@ -213,7 +213,9 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
>   		iter->index = i++;
>   
>   		if (hweight32(iter->flags & (MEDIA_PAD_FL_SINK |
> -					     MEDIA_PAD_FL_SOURCE)) != 1) {
> +					     MEDIA_PAD_FL_SOURCE)) != 1 ||
> +		    (iter->flags & MEDIA_PAD_FL_INTERNAL &&
> +		     !(iter->flags & MEDIA_PAD_FL_SINK))) {
>   			ret = -EINVAL;
>   			break;

Would it make sense to have this "only internal-sinks supported"-check 
as a separate check, with an error print? It is a valid thing to do, we 
just want to disable it for the time being.

>   		}
> @@ -1075,7 +1077,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
>   
>   	for (i = 0; i < entity->num_pads; i++) {
>   		if ((entity->pads[i].flags &
> -		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
> +		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
> +		      MEDIA_PAD_FL_INTERNAL)) != pad_type)
>   			continue;
>   
>   		if (entity->pads[i].sig_type == sig_type)
> @@ -1100,6 +1103,9 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
>   		return -EINVAL;
>   	if (WARN_ON(!(sink->pads[sink_pad].flags & MEDIA_PAD_FL_SINK)))
>   		return -EINVAL;
> +	if (WARN_ON(source->pads[source_pad].flags & MEDIA_PAD_FL_INTERNAL) ||
> +	    WARN_ON(source->pads[sink_pad].flags & MEDIA_PAD_FL_INTERNAL))
> +		return -EINVAL;
>   
>   	link = media_add_link(&source->links);
>   	if (link == NULL)
> diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
> index 1c80b1d6bbaf..80cfd12a43fc 100644
> --- a/include/uapi/linux/media.h
> +++ b/include/uapi/linux/media.h
> @@ -208,6 +208,7 @@ struct media_entity_desc {
>   #define MEDIA_PAD_FL_SINK			(1U << 0)
>   #define MEDIA_PAD_FL_SOURCE			(1U << 1)
>   #define MEDIA_PAD_FL_MUST_CONNECT		(1U << 2)
> +#define MEDIA_PAD_FL_INTERNAL			(1U << 3)
>   
>   struct media_pad_desc {
>   	__u32 entity;		/* entity ID */

Re: [PATCH v6 17/28] media: v4l: subdev: Return routes set using S_ROUTING

[Hans Verkuil]

On 03/10/2023 14:08, Sakari Ailus wrote:
> Return the routes set using S_ROUTING back to the user. Also reflect this
> in documentation.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../userspace-api/media/v4l/vidioc-subdev-g-routing.rst      | 5 +++--
>  drivers/media/v4l2-core/v4l2-subdev.c                        | 5 ++++-
>  2 files changed, 7 insertions(+), 3 deletions(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> index 9a9765ddc316..ced53ea5f23c 100644
> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> @@ -43,8 +43,9 @@ The routing configuration determines the flows of data inside an entity.
>  Drivers report their current routing tables using the
>  ``VIDIOC_SUBDEV_G_ROUTING`` ioctl and application may enable or disable routes
>  with the ``VIDIOC_SUBDEV_S_ROUTING`` ioctl, by adding or removing routes and
> -setting or clearing flags of the  ``flags`` field of a
> -struct :c:type:`v4l2_subdev_route`.
> +setting or clearing flags of the ``flags`` field of a struct
> +:c:type:`v4l2_subdev_route`. Similarly to ``VIDIOC_SUBDEV_G_ROUTING``, also
> +``VIDIOC_SUBDEV_S_ROUTING`` returns the routes back to the user.
>  
>  All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is
>  called. This means that the userspace must reconfigure all streams after calling
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index bd1e8205913c..9a34e13dfd96 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -935,9 +935,12 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>  		krouting.len_routes = routing->len_routes;
>  		krouting.routes = routes;
>  
> -		return v4l2_subdev_call(sd, pad, set_routing, state,
> +		rval = v4l2_subdev_call(sd, pad, set_routing, state,
>  					routing->which, &krouting);
> +		if (rval < 0)
> +			return rval;
>  	}
> +		fallthrough;

Admittedly, it is a bit hard to see from just this patch, but I think
the actual code that VIDIOC_SUBDEV_S_ROUTING has to do here is just
a very few lines from VIDIOC_SUBDEV_G_ROUTING. I think it is better
to explicitly have them here, then doing a fallthrough.

Most of what is in VIDIOC_SUBDEV_G_ROUTING are just a bunch of checks
that are already done in VIDIOC_SUBDEV_S_ROUTING.

Regards,

	Hans

>  
>  	case VIDIOC_SUBDEV_G_ROUTING: {
>  		struct v4l2_subdev_routing *routing = arg;

Re: [PATCH v6 18/28] media: uapi: Allow a larger number of routes than there's room for

[Hans Verkuil]

On 03/10/2023 14:08, Sakari Ailus wrote:
> On VIDIOC_SUBDEV_[GS]_ROUTING, only return as many routes back to the user
> as there's room. Do not consider it an error if more routes existed.
> Simply inform the user there are more routes.

I'm not convinced by this change. Can userspace do anything useful with only
a partially filled routes array?

Just silently allowing bad data is asking for problems, IMHO.

Regards,

	Hans

> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../userspace-api/media/v4l/vidioc-subdev-g-routing.rst   | 4 ----
>  drivers/media/v4l2-core/v4l2-subdev.c                     | 8 ++------
>  2 files changed, 2 insertions(+), 10 deletions(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> index ced53ea5f23c..99d3c15fd759 100644
> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> @@ -145,10 +145,6 @@ On success 0 is returned, on error -1 and the ``errno`` variable is set
>  appropriately. The generic error codes are described at the
>  :ref:`Generic Error Codes <gen-errors>` chapter.
>  
> -ENOSPC
> -   The application provided ``num_routes`` is not big enough to contain
> -   all the available routes the subdevice exposes.
> -
>  EINVAL
>     The sink or source pad identifiers reference a non-existing pad, or reference
>     pads of different types (ie. the sink_pad identifiers refers to a source pad).
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 9a34e13dfd96..dd48e7e549fb 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -956,14 +956,10 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>  
>  		krouting = &state->routing;
>  
> -		if (routing->len_routes < krouting->num_routes) {
> -			routing->num_routes = krouting->num_routes;
> -			return -ENOSPC;
> -		}
> -
>  		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
>  		       krouting->routes,
> -		       krouting->num_routes * sizeof(*krouting->routes));
> +		       min(krouting->num_routes, routing->len_routes) *
> +		       sizeof(*krouting->routes));
>  		routing->num_routes = krouting->num_routes;
>  
>  		return 0;

Re: [PATCH v6 01/28] media: mc: Add INTERNAL pad flag

[Hans Verkuil]

On 05/10/2023 12:22, Sakari Ailus wrote:
> Hi Hans,
> 
> Thanks for the review!
> 
> On Thu, Oct 05, 2023 at 11:52:08AM +0200, Hans Verkuil wrote:
>> On 03/10/2023 13:52, Sakari Ailus wrote:
>>> Internal source pads will be used as routing endpoints in V4L2
>>> [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
>>>
>>> Also prevent creating links to pads that have been flagged as internal and
>>> initialising source pads with internal flag set.
>>>
>>> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
>>> ---
>>>  Documentation/userspace-api/media/glossary.rst         |  6 ++++++
>>>  .../userspace-api/media/mediactl/media-types.rst       |  6 ++++++
>>>  drivers/media/mc/mc-entity.c                           | 10 ++++++++--
>>>  include/uapi/linux/media.h                             |  1 +
>>>  4 files changed, 21 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/Documentation/userspace-api/media/glossary.rst b/Documentation/userspace-api/media/glossary.rst
>>> index 96a360edbf3b..f7b99a4527c7 100644
>>> --- a/Documentation/userspace-api/media/glossary.rst
>>> +++ b/Documentation/userspace-api/media/glossary.rst
>>> @@ -173,6 +173,12 @@ Glossary
>>>  	An integrated circuit that integrates all components of a computer
>>>  	or other electronic systems.
>>>  
>>> +_media-glossary-stream:
>>> +    Stream
>>> +	A distinct flow of data (image data or metadata) over a media pipeline
>>> +	from source to sink. A source may be e.g. an image sensor and a sink
>>> +	e.g. a memory buffer.
>>
>> Hmm, I think this is a bit confusing. I think it would be better to replace
>> "from source to sink" with "from the initial source to the final sink".
> 
> Seems fine to me. I'll use that in v7.
> 
>>
>> The original text doesn't make it clear that there can be many hops in
>> between.
>>
>> So also: "A source" -> "An initial source", and "a sink" -> "a final sink".
>>
>> Note that "media pipeline" isn't defined either. Should that be added?
> 
> I can add that, too...
> 
>>
>> Finally, I think this should be a separate patch as it has nothing to do with
>> adding the INTERNAL pad flag.
> 
> I agree, will split in v7.
> 
>>
>>> +
>>>      V4L2 API
>>>  	**V4L2 userspace API**
>>>  
>>> diff --git a/Documentation/userspace-api/media/mediactl/media-types.rst b/Documentation/userspace-api/media/mediactl/media-types.rst
>>> index 0ffeece1e0c8..28941da27790 100644
>>> --- a/Documentation/userspace-api/media/mediactl/media-types.rst
>>> +++ b/Documentation/userspace-api/media/mediactl/media-types.rst
>>> @@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
>>>  .. _MEDIA-PAD-FL-SINK:
>>>  .. _MEDIA-PAD-FL-SOURCE:
>>>  .. _MEDIA-PAD-FL-MUST-CONNECT:
>>> +.. _MEDIA-PAD-FL-INTERNAL:
>>>  
>>>  .. flat-table:: Media pad flags
>>>      :header-rows:  0
>>> @@ -382,6 +383,11 @@ Types and flags used to represent the media graph elements
>>>  	  when this flag isn't set; the absence of the flag doesn't imply
>>>  	  there is none.
>>>  
>>> +    *  -  ``MEDIA_PAD_FL_INTERNAL``
>>> +       -  The internal flag indicates an internal pad that has no external
>>> +	  connections. Such a pad shall not be connected with a link. The
>>> +	  internal flag indicates that the :ref:``stream
>>> +	  <media-glossary-stream>`` either starts or ends in the entity.
>>
>> This suggests that INTERNAL can be used for both sinks and sources, but...
> 
> Right. The INTERNAL flag in UAPI shouldn't be limited to sources, but at
> the same time we don't have use for it in sinks. I'd prefer to leave this
> open in the future. We could of course say that explicitly.

I think it is better to mention that it is for streams that start in the
entity only, but that in the future it might be extended to support streams
that end in the entity.

When we add support for that, we need to update the documentation as well,
at minimum with one or more examples of how that would be used.

Regards,

	Hans

> 
>>
>>>  
>>>  One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
>>>  must be set for every pad.
>>> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
>>> index 543a392f8635..f5f290781021 100644
>>> --- a/drivers/media/mc/mc-entity.c
>>> +++ b/drivers/media/mc/mc-entity.c
>>> @@ -213,7 +213,9 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
>>>  		iter->index = i++;
>>>  
>>>  		if (hweight32(iter->flags & (MEDIA_PAD_FL_SINK |
>>> -					     MEDIA_PAD_FL_SOURCE)) != 1) {
>>> +					     MEDIA_PAD_FL_SOURCE)) != 1 ||
>>> +		    (iter->flags & MEDIA_PAD_FL_INTERNAL &&
>>> +		     !(iter->flags & MEDIA_PAD_FL_SINK))) {
>>
>> ...this appears to limit it to just sinks.
> 
> Yes, this is enforced so we won't get drivers that accidentally would use
> the INTERNAL flag on source pads.
> 
>>
>> Regards,
>>
>> 	Hans
>>
>>>  			ret = -EINVAL;
>>>  			break;
>>>  		}
>>> @@ -1075,7 +1077,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
>>>  
>>>  	for (i = 0; i < entity->num_pads; i++) {
>>>  		if ((entity->pads[i].flags &
>>> -		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
>>> +		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
>>> +		      MEDIA_PAD_FL_INTERNAL)) != pad_type)
>>>  			continue;
>>>  
>>>  		if (entity->pads[i].sig_type == sig_type)
>>> @@ -1100,6 +1103,9 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
>>>  		return -EINVAL;
>>>  	if (WARN_ON(!(sink->pads[sink_pad].flags & MEDIA_PAD_FL_SINK)))
>>>  		return -EINVAL;
>>> +	if (WARN_ON(source->pads[source_pad].flags & MEDIA_PAD_FL_INTERNAL) ||
>>> +	    WARN_ON(source->pads[sink_pad].flags & MEDIA_PAD_FL_INTERNAL))
>>> +		return -EINVAL;
>>>  
>>>  	link = media_add_link(&source->links);
>>>  	if (link == NULL)
>>> diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
>>> index 1c80b1d6bbaf..80cfd12a43fc 100644
>>> --- a/include/uapi/linux/media.h
>>> +++ b/include/uapi/linux/media.h
>>> @@ -208,6 +208,7 @@ struct media_entity_desc {
>>>  #define MEDIA_PAD_FL_SINK			(1U << 0)
>>>  #define MEDIA_PAD_FL_SOURCE			(1U << 1)
>>>  #define MEDIA_PAD_FL_MUST_CONNECT		(1U << 2)
>>> +#define MEDIA_PAD_FL_INTERNAL			(1U << 3)
>>>  
>>>  struct media_pad_desc {
>>>  	__u32 entity;		/* entity ID */
>>
> 

Re: [PATCH v6 00/28] Generic line based metadata support, internal pads

[Tomi Valkeinen]

On 05/10/2023 12:04, Sakari Ailus wrote:
> Hyvää huomenta!
> 
> On Thu, Oct 05, 2023 at 11:05:54AM +0300, Tomi Valkeinen wrote:
>> Hi!
>>
>> Thanks for working on this. I think this series is very important.
>>
>> On 03/10/2023 14:52, Sakari Ailus wrote:
>>> Hi folks,
>>>
>>> Here are a few patches to add support generic, line based metadata as well
>>> as internal pads. While the amount of code is not very large, to the
>>> contrary it is quite small actually IMO, I presume what this is about and
>>> why it is being proposed requires some explaining.
>>>
>>> Metadata mbus codes and formats have existed for some time in V4L2. They
>>> however have been only used by drivers that produce the data itself and
>>> effectively this metadata has always been statistics of some sort (at
>>> least when it comes to ISPs). What is different here is that we intend to
>>> add support for metadata originating from camera sensors.
>>>
>>> Camera sensors produce different kinds of metadata, embedded data (usually
>>> register address--value pairs used to capture the frame, in a more or less
>>> sensor specific format), histograms (in a very sensor specific format),
>>> dark pixels etc. The number of these formats is probably going to be about
>>> as large as image data formats if not larger, as the image data formats
>>> are much better standardised but a smaller subset of them will be
>>> supported by V4L2, at least initially but possibly much more in the long
>>> run.
>>>
>>> Having this many device specific formats would be a major problem for all
>>> the other drivers along that pipeline (not to mention the users of those
>>> drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2
>>> receiver drivers that have DMA: the poor driver developer would not only
>>> need to know camera sensor specific formats but to choose the specific
>>> packing of that format suitable for the DMA used by the hardware. It is
>>> unlikely many of these would ever get tested while being present on the
>>> driver API. Also adding new sensors with new embedded data formats would
>>> involve updating all bridge and CSI-2 receiver drivers. I don't expect
>>> this to be a workable approach.
>>>
>>> Instead what I'm proposing is to use specific metadata formats on the
>>> sensor devices only, on internal pads (more about those soon) of the
>>> sensors, only visible in the UAPI, and then generic mbus formats along the
>>
>> What do you mean with "only visible in the UAPI"?
> 
> Other drivers won't bother with specific metadata formats: they are only
> present on the internal pads while external pads have generic formats.
> 
>>
>>> pipeline and finally generic V4L2 metadata formats on the DMAs (specific
>>> to bit depth and packing). This would unsnarl the two, defining what data
>>> there is (specific mbus code) and how that is transported and packed
>>> (generic mbus codes and V4L2 formats).
>>>
>>> The user space would be required to "know" the path of that data from the
>>> sensor's internal pad to the V4L2 video node. I do not see this as these
>>> devices require at least some knowledge of the pipeline, i.e. hardware at
>>> hand. Separating what the data means and how it is packed may even be
>>> beneficial: it allows separating code that interprets the data (sensor
>>> internal mbus code) from the code that accesses it (packing).
>>>
>>> These formats are in practice line based, meaning that there may be
>>> padding at the end of the line, depending on the bus as well as the DMA.
>>> If non-line based formats are needed, it is always possible to set the
>>> "height" field to 1.
>>>
>>> The internal (source) pads are an alternative to source routes [1]. The
>>> source routes were not universally liked and I do have to say I like
>>> re-using existing interface concepts (pads and everything you can do with
>>> pads, including access format, selections etc.) wherever it makes sense,
>>> instead of duplicating functionality.
>>>
>>> Effectively internal source pads behave mostly just like sink pads, but
>>> they describe a flow of data that originates from a sub-device instead of
>>> arriving to a sub-device. The SUBDEV_S_ROUTING IOCTLs are used to enable
>>> and disable routes from internal source pads to sub-device's source pads.
>>> The subdev format IOCTLs are usable, too, so one can find which subdev
>>> format is available on given internal source pad.
>>
>> I think the internal pads require a bit more praise, as they can be used for
>> other things too. E.g. the ds90ub953 FPD-Link serializer has a test pattern
>> generator, which can be modeled very nicely with internal pads. The internal
>> pad represents the TPG, and the user can use routing to choose if the output
>> of the device is sourced from the normal input or from the TPG. And one can
>> set the format on the TPG pad, thus configuring the TPG.
> 
> Well, yes, indeed.
> 
> Could you review especially the documentation patches to ensure we're
> aligned on this?

Sure.

>>
>>> This set depends on these patches:
>>>
>>> <URL:https://lore.kernel.org/linux-media/20231002105557.28972-1-sakari.ailus@linux.intel.com/T/#t>
>>
>> Hmm, it's a bit odd for a generic series to depend on a device specific
>> series. That makes backporting these more difficult. Why do these depend on
>> ov2740 and css patches?
> 
> Patchset-wise that is the dependency, individual patches may be backported
> without backporting _all_ driver patches in the previous set. However, if
> you need those drivers as well, then you'll need to backport these patches,
> too.

Ok. I don't like the structure of these serieses that much.

I haven't reviewed the patches yet, so maybe there are reasons for the 
structure, but I'd rather have, in this order, preferably as separate 
serieses:

- cleanups/fixes not directly related to ccs, embedded data or internal pads
- internal pads
- embedded data
- driver changes

That would help in backporting, but it'd also help with the reviews as 
the series would be more concentrated on a single topic.

  Tomi

Re: [PATCH v6 03/28] media: uapi: Document which mbus format fields are valid for metadata

[Tomi Valkeinen]

On 03/10/2023 14:52, Sakari Ailus wrote:
> Now that metadata mbus formats have been added, it is necessary to define
> which fields in struct v4l2_mbus_format are applicable to them (not many).
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>   include/uapi/linux/v4l2-mediabus.h | 18 ++++++++++++------
>   1 file changed, 12 insertions(+), 6 deletions(-)
> 
> diff --git a/include/uapi/linux/v4l2-mediabus.h b/include/uapi/linux/v4l2-mediabus.h
> index 6b07b73473b5..3cadb3b58b85 100644
> --- a/include/uapi/linux/v4l2-mediabus.h
> +++ b/include/uapi/linux/v4l2-mediabus.h
> @@ -19,12 +19,18 @@
>    * @width:	image width
>    * @height:	image height
>    * @code:	data format code (from enum v4l2_mbus_pixelcode)
> - * @field:	used interlacing type (from enum v4l2_field)
> - * @colorspace:	colorspace of the data (from enum v4l2_colorspace)
> - * @ycbcr_enc:	YCbCr encoding of the data (from enum v4l2_ycbcr_encoding)
> - * @hsv_enc:	HSV encoding of the data (from enum v4l2_hsv_encoding)
> - * @quantization: quantization of the data (from enum v4l2_quantization)
> - * @xfer_func:  transfer function of the data (from enum v4l2_xfer_func)
> + * @field:	used interlacing type (from enum v4l2_field), not applicable
> + *		to metadata mbus codes

This one does say "zero", like the others do. Is that on purpose? 
Annoyingly V4L2_FIELD_NONE is 1, not 0, but perhaps zeroing this would 
still be best?

> + * @colorspace:	colorspace of the data (from enum v4l2_colorspace), zero on
> + *		metadata mbus codes
> + * @ycbcr_enc:	YCbCr encoding of the data (from enum v4l2_ycbcr_encoding), zero
> + *		on metadata mbus codes
> + * @hsv_enc:	HSV encoding of the data (from enum v4l2_hsv_encoding), zero on
> + *		metadata mbus codes
> + * @quantization: quantization of the data (from enum v4l2_quantization), zero
> + *		on metadata mbus codes
> + * @xfer_func:  transfer function of the data (from enum v4l2_xfer_func), zero
> + *		on metadata mbus codes
>    * @flags:	flags (V4L2_MBUS_FRAMEFMT_*)
>    * @reserved:  reserved bytes that can be later used
>    */

Re: [PATCH v6 08/28] media: Documentation: Additional streams generally don't harm capture

[Tomi Valkeinen]

On 03/10/2023 14:52, Sakari Ailus wrote:
> Having extra streams on the source end of the link that cannot be captured
> by the sink sub-device generally are not an issue, at least not on CSI-2
> bus. Still document that there may be hardware specific limitations. For
> example on parallel bus this might not work on all cases.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>   Documentation/userspace-api/media/v4l/dev-subdev.rst | 6 +++---
>   1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> index f375b820ab68..a387e8a15b8d 100644
> --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> @@ -529,9 +529,9 @@ the its sink pad and allows to route them individually to one of its source
>   pads.
>   
>   Subdevice drivers that support multiplexed streams are compatible with
> -non-multiplexed subdev drivers, but, of course, require a routing configuration
> -where the link between those two types of drivers contains only a single
> -stream.
> +non-multiplexed subdev drivers. However, if the driver at the sink end of a link
> +does not support streams, then only the stream 0 on source end may be
> +captured. There may be additional hardware specific limitations.
>   
>   Understanding streams
>   ^^^^^^^^^^^^^^^^^^^^^

Reviewed-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>

  Tomi

Re: [PATCH v6 09/28] media: Documentation: Document embedded data guidelines for camera sensors

[Tomi Valkeinen]

On 05/10/2023 13:10, Hans Verkuil wrote:
> On 03/10/2023 13:52, Sakari Ailus wrote:
>> Document how embedded data support should be implemented for camera
>> sensors, and when and how CCS embedded data format should be referenced.
>>
>> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
>> ---
>>   .../media/drivers/camera-sensor.rst           | 28 +++++++++++++++++++
>>   1 file changed, 28 insertions(+)
>>
>> diff --git a/Documentation/userspace-api/media/drivers/camera-sensor.rst b/Documentation/userspace-api/media/drivers/camera-sensor.rst
>> index 919a50e8b9d9..308f391c5ca1 100644
>> --- a/Documentation/userspace-api/media/drivers/camera-sensor.rst
>> +++ b/Documentation/userspace-api/media/drivers/camera-sensor.rst
>> @@ -102,3 +102,31 @@ register programming sequences shall initialize the :ref:`V4L2_CID_HFLIP
>>   values programmed by the register sequences. The default values of these
>>   controls shall be 0 (disabled). Especially these controls shall not be inverted,
>>   independently of the sensor's mounting rotation.
>> +
>> +Embedded data
>> +-------------
>> +
>> +Many sensors, mostly raw sensors, support embedded data which is used to convey
>> +the sensor configuration for the captured frame back to the host. While CSI-2 is
>> +the most common bus used by such sensors, embedded data is not entirely limited
>> +to CSI-2 bus due to e.g. bridge devices.
> 
> I'm not quite sure what you mean with "embedded data is not entirely limited
> to CSI-2 bus due to e.g. bridge devices": do you just want to say: "embedded data
> can be available on other bus types as well."?

Right, nothing CSI-2 specific here. I have a parallel sensor that sends 
embedded data as the last (or was it first?) few lines of the frame.

Also, is this about sensors only, and more specifically, about 
generating the embedded data? If yes, why mention bridge devices (and 
should the title be "Embedded data generation" or such)? If not, is it 
about metadata in general, in which case bridges come into play?

>> +
>> +Embedded data support should use an internal source pad and route to the
> 
> should or shall/must?

Is the above necessary? I don't see anything wrong with, say, having a 
sensor represented with multiple subdevs, and one subdev is used for the 
embedded data generation.

Basically, anything that can be represented with internal pads can also 
be done with multiple subdevs, although I think you don't want to go the 
multiple subdev route unless absolutely necessary.

> Regards,
> 
> 	Hans
> 
>> +external pad. If embedded data output can be disabled in hardware, it should be
>> +possible to disable the embedded data route via ``VIDIOC_SUBDEV_S_ROUTING``
>> +IOCTL.
>> +
>> +In general, changing the embedded data format from the driver-configured values
>> +is not supported. The height of the metadata is hardware specific and the width
>> +is that (or less of that) of the image width, as configured on the pixel data
>> +stream.
>> +
>> +CCS and non-CCS embedded data
>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> +
>> +Embedded data which is compliant with CCS definitions shall use ``CCS embedded
>> +data format <MEDIA-BUS-FMT-CCS-EMBEDDED>``. Device specific embedded data which
>> +is compliant up to MIPI CCS embedded data levels 1 or 2 only shall refer to CCS
>> +embedded data formats and document the level of conformance. The rest of the
>> +device specific embedded data format shall be documented in the context of the
>> +data format itself.
> 

Re: [PATCH v6 10/28] media: Documentation: v4l: Document source routes

[Tomi Valkeinen]

On 03/10/2023 15:07, Sakari Ailus wrote:
> Document how internal pads are used on source routes. Use the IMX219
> camera sensor as an example.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>   .../userspace-api/media/v4l/dev-subdev.rst    | 179 ++++++++++++++++++
>   1 file changed, 179 insertions(+)
> 
> diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> index a387e8a15b8d..fb73a95401c3 100644
> --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> @@ -553,6 +553,27 @@ A stream at a specific point in the media pipeline is identified by the
>   sub-device and a (pad, stream) pair. For sub-devices that do not support
>   multiplexed streams the 'stream' field is always 0.
>   
> +.. _v4l2-subdev-source-routes:
> +
> +Internal pads and source routes
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +Cases where a single sub-device source pad is traversed by multiple streams one
> +or more of which originate from within the sub-device itself are special as
> +there is no external sink pad for such routes. In those cases, the sources of
> +the internally generated streams are represented by internal sink pads, which
> +are sink pads that have the :ref:`MEDIA_PAD_FL_INTERNAL <MEDIA-PAD-FL-INTERNAL>`
> +pad flag set.
> +
> +Internal pads have all the properties of an external pad, including formats and
> +selections. The format in this case is the source format of the stream. An
> +internal pad always has a single stream only (0).
> +
> +*Source routes* are routes from an internal sink pad to an external source
> +pad. In most cases source routes are not modifiable but they can be activated
> +and deactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +<v4l2-subdev-routing-flags>` flag, depending on driver capabilities.
> +
>   Interaction between routes, streams, formats and selections
>   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   
> @@ -668,3 +689,161 @@ To configure this pipeline, the userspace must take the following steps:
>      the configurations along the stream towards the receiver, using
>      :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each
>      stream endpoint in each sub-device.
> +
> +Internal pads setup example
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +A simple example of a multiplexed stream setup might be as follows:
> +
> +- A CCS camera sensor source sub-device, with one sink pad (0), one source pad
> +  (1), an internal sink pad (2) that represents the source of embedded

Wouldn't a simpler and better example be a sensor with a single 
sub-device, with two internal pads and one external pad? Now your 
example is a partial example, as the sensor subdevice has a external 
sink pad connected to something which is not covered in the example.

Or is part of the point here to show an example with a subdevice with 
both internal and external sink pads?

> +  data. There are two routes, one from the sink pad to the source, and another
> +  from the internal sink pad to the source pad. The embedded data stream needs
> +  to be enabled by activating the related route. The configuration of the rest
> +  of the CCS sub-devices is omitted from this example.
> +
> +- Multiplexer bridge (Bridge). The bridge has one sink pad, connected to the
> +  sensor (pad 0), and one source pad (pad 1), which outputs two streams.

What does it multiplex if there's a single input and a single output?

> +- Receiver in the SoC (Receiver). The receiver has a single sink pad (pad 0),
> +  connected to the bridge, and two source pads (pads 1-2), going to the DMA
> +  engine. The receiver demultiplexes the incoming streams to the source pads.
> +
> +- DMA Engines in the SoC (DMA Engine), one for each stream. Each DMA engine is
> +  connected to a single source pad in the receiver.
> +
> +The sensor, the bridge and the receiver are modeled as V4L2 sub-devices,
> +exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are
> +modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
> +
> +To configure this pipeline, the userspace must take the following steps:
> +
> +1) Set up media links between entities: connect the sensors to the bridge,
> +   bridge to the receiver, and the receiver to the DMA engines. This step does
> +   not differ from normal non-multiplexed media controller setup.
> +
> +2) Configure routing
> +
> +.. flat-table:: Camera sensor
> +    :header-rows: 1
> +
> +    * - Sink Pad/Stream
> +      - Source Pad/Stream
> +      - Routing Flags
> +      - Comments
> +    * - 0/0
> +      - 1/0
> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +      - Pixel data stream from the sink pad
> +    * - 2/0
> +      - 1/1
> +      - **V4L2_SUBDEV_ROUTE_FL_ACTIVE**

Why is this one bold?

> +      - Metadata stream from the internal sink pad
> +
> +.. flat-table:: Bridge routing table
> +    :header-rows: 1
> +
> +    * - Sink Pad/Stream
> +      - Source Pad/Stream
> +      - Routing Flags
> +      - Comments
> +    * - 0/0
> +      - 1/0
> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +      - Pixel data stream from camera sensor
> +    * - 0/1
> +      - 1/1
> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +      - Metadata stream from camera sensor
> +
> +.. flat-table:: Receiver routing table
> +    :header-rows:  1
> +
> +    * - Sink Pad/Stream
> +      - Source Pad/Stream
> +      - Routing Flags
> +      - Comments
> +    * - 0/0
> +      - 1/0
> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +      - Pixel data stream from camera sensor
> +    * - 0/1
> +      - 2/0
> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +      - Metadata stream from camera sensor
> +
> +The options available in sensor's routing configuration are dictated by
> +hardware capabilities: typically camera sensors always produce image data
> +stream while the embedded data stream typically can be either enabled or
> +disabled.
> +
> +3) Configure formats and selections
> +
> +   This example assumes that the formats are propagated from sink pad to the
> +   source pad as-is. The tables contain fields of both struct v4l2_subdev_format
> +   and struct v4l2_mbus_framefmt.
> +
> +.. flat-table:: Formats set on the sub-devices. Bold values are set, others are
> +                static or propagated.
> +    :header-rows: 1
> +    :fill-cells:
> +
> +    * - Sub-device
> +      - Pad/Stream
> +      - Width
> +      - Height
> +      - Code
> +    * - :rspan:`3` Camera sensor sub-device (IMX219)
> +      - 1/0
> +      - 3296
> +      - 2480
> +      - MEDIA_BUS_FMT_SRGGB10
> +    * - 0/0

I think the 0/0 stream should be the first one in the table.

> +      - **3296**
> +      - **2480**
> +      - **MEDIA_BUS_FMT_SRGGB10**
> +    * - 2/0
> +      - 3296
> +      - 2
> +      - MEDIA_BUS_FMT_IMX219_EMBEDDED
> +    * - 1/1
> +      - 3296
> +      - 2
> +      - MEDIA_BUS_FMT_META_10
> +    * - :rspan:`3` Bridge
> +      - 0/0
> +      - **3296**
> +      - **2480**
> +      - **MEDIA_BUS_FMT_SRGGB10**
> +    * - 1/0
> +      - 3296
> +      - 2480
> +      - MEDIA_BUS_FMT_SRGGB10
> +    * - 0/1
> +      - **3296**
> +      - **2**
> +      - **MEDIA_BUS_FMT_META_10**
> +    * - 1/1
> +      - 3296
> +      - 2
> +      - MEDIA_BUS_FMT_META_10
> +    * - :rspan:`3` Receiver
> +      - 0/0
> +      - **3296**
> +      - **2480**
> +      - **MEDIA_BUS_FMT_SRGGB10**
> +    * - 1/0
> +      - 3296
> +      - 2480
> +      - MEDIA_BUS_FMT_SRGGB10
> +    * - 0/1
> +      - **3296**
> +      - **2**
> +      - **MEDIA_BUS_FMT_META_10**
> +    * - 2/0
> +      - 3296
> +      - 2
> +      - MEDIA_BUS_FMT_META_10
> +
> +The embedded data format does not need to be configured as the format is
> +dictated by the pixel data format in this case.

Re: [PATCH v6 11/28] media: Documentation: Document S_ROUTING behaviour

[Tomi Valkeinen]

On 03/10/2023 15:07, Sakari Ailus wrote:
> Document S_ROUTING behaviour for different devices.
> 
> Generally in devices that produce streams the streams are static and some

I'm not sure what "static" means here. "Generally in devices that 
produce streams there is a fixed amount of streams..."?

> can be enabled and disabled, whereas in devices that just transport them
> or write them to memory, more configurability is allowed. Document this.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>   .../userspace-api/media/v4l/dev-subdev.rst    | 21 +++++++++++++++++++
>   1 file changed, 21 insertions(+)
> 
> diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> index fb73a95401c3..83993775237f 100644
> --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> @@ -593,6 +593,27 @@ Any configurations of a stream within a pad, such as format or selections,
>   are independent of similar configurations on other streams. This is
>   subject to change in the future.
>   
> +Device types and routing setup
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +Different kinds of sub-devices have differing behaviour for route inactivation,

Would "activation" convey the same, but be a bit clearer?

> +depending on the hardware. Devices generating the streams may often allow

Maybe drop the "often".

> +enabling and disabling some of these or the configuration is fixed. If these

"some of these" -> "some of the routes".

> +routes can be disabled, not declaring these routes (or declaring them without

Here also, I think "the routes" is more readable than repeating "these 
routes".

> +``VIDIOC_SUBDEV_STREAM_FL_ACTIVE`` flag set) in ``VIDIOC_SUBDEV_S_ROUTING`` will

Why is the flag sentence in parenthesis? Aren't both options of the same 
value?

> +disable the routes while the sub-device driver retain the streams and their

What does this mean? That even if the user disables a route, the driver 
must keep the configuration that was set earlier related to that route?

> +configuration. The ``VIDIOC_SUBDEV_S_ROUTING`` will still return such routes
> +back to the user in the routes array, with the ``V4L2_SUBDEV_STREAM_FL_ACTIVE``
> +flag unset.

So a generating device should always return all its routes with both 
G_ROUTING and S_ROUTING, right? But with disabled routes not having 
VIDIOC_SUBDEV_STREAM_FL_ACTIVE. The text doesn't mention G_ROUTING at all.

> +Devices transporting the streams almost always have more configurability with
> +respect to routing. Typically any route between the sub-device's sink and source
> +pads is possible, and multiple routes (usually up to certain limited number) may
> +be active simultaneously. For such devices, no routes are created by the driver
> +and user-created routes are fully replaced when ``VIDIOC_SUBDEV_S_ROUTING`` is
> +called on the sub-device. Such newly created routes have the device's default
> +configuration for format and selection rectangles.
> +

I think this paragraph is ok. But could this whole section be 
restructured a bit, as the previous paragraph gets quite confusing. Maybe:

First paragraph to explain the two different kinds of devices, and 
perhaps a mention that a route is considered disabled if either it does 
not exist in the routing table or if VIDIOC_SUBDEV_STREAM_FL_ACTIVE is 
not set.

Then a paragraph for generating devices, and then a paragraph for 
transporting devices.

>   Configuring streams
>   ^^^^^^^^^^^^^^^^^^^
>   

Re: [PATCH v6 12/28] media: v4l: subdev: Add helpers for format, crop and compose pointers

[Tomi Valkeinen]

On 03/10/2023 15:07, Sakari Ailus wrote:
> Add a helper for obtaining format, crop and compose pointers. These are
> convenient for drivers, independently of the driver uses streams or not.

If we go with these, should we deprecate 
v4l2_subdev_state_get_stream_format() and v4l2_subdev_get_pad_format()?

Having three different ways to get the fmt seems a bit excessive.

Should we add 'num_pads' to the state? That would remove the need pass 
the subdevice to these functions.

> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>   drivers/media/v4l2-core/v4l2-subdev.c | 54 ++++++++++++++++++++++----
>   include/media/v4l2-subdev.h           | 56 +++++++++++++++++++++++++++
>   2 files changed, 102 insertions(+), 8 deletions(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index d295a4e87b66..854f9d4db923 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -1573,19 +1573,57 @@ v4l2_subdev_init_stream_configs(struct v4l2_subdev_stream_configs *stream_config
>   	return 0;
>   }
>   
> +struct v4l2_mbus_framefmt
> +*v4l2_subdev_get_fmt_ptr(struct v4l2_subdev *sd,
> +			 struct v4l2_subdev_state *state, unsigned int pad,
> +			 unsigned int stream)
> +{
> +	if (sd->flags & V4L2_SUBDEV_FL_STREAMS)
> +		return v4l2_subdev_state_get_stream_format(state, pad, stream);
> +
> +	if (pad < sd->entity.num_pads && stream == 0)
> +		return v4l2_subdev_get_pad_format(sd, state, pad);
> +
> +	return NULL;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_get_fmt_ptr);
> +
> +struct v4l2_rect
> +*v4l2_subdev_get_crop_ptr(struct v4l2_subdev *sd,
> +			  struct v4l2_subdev_state *state, unsigned int pad,
> +			  unsigned int stream)
> +{
> +	if (sd->flags & V4L2_SUBDEV_FL_STREAMS)
> +		return v4l2_subdev_state_get_stream_crop(state, pad, stream);
> +
> +	if (pad < sd->entity.num_pads && stream == 0)
> +		return v4l2_subdev_get_pad_crop(sd, state, pad);
> +
> +	return NULL;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_get_crop_ptr);
> +
> +struct v4l2_rect
> +*v4l2_subdev_get_compose_ptr(struct v4l2_subdev *sd,
> +			     struct v4l2_subdev_state *state, unsigned int pad,
> +			     unsigned int stream)
> +{
> +	if (sd->flags & V4L2_SUBDEV_FL_STREAMS)
> +		return v4l2_subdev_state_get_stream_compose(state, pad, stream);
> +
> +	if (pad < sd->entity.num_pads && stream == 0)
> +		return v4l2_subdev_get_pad_compose(sd, state, pad);
> +
> +	return NULL;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_get_compose_ptr);
> +
>   int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
>   			struct v4l2_subdev_format *format)
>   {
>   	struct v4l2_mbus_framefmt *fmt;
>   
> -	if (sd->flags & V4L2_SUBDEV_FL_STREAMS)
> -		fmt = v4l2_subdev_state_get_stream_format(state, format->pad,
> -							  format->stream);
> -	else if (format->pad < sd->entity.num_pads && format->stream == 0)
> -		fmt = v4l2_subdev_get_pad_format(sd, state, format->pad);
> -	else
> -		fmt = NULL;
> -
> +	fmt = v4l2_subdev_get_fmt_ptr(sd, state, format->pad, format->stream);
>   	if (!fmt)
>   		return -EINVAL;
>   
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 5f59ff0796b7..7c34243ffed9 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -1479,6 +1479,62 @@ v4l2_subdev_lock_and_get_active_state(struct v4l2_subdev *sd)
>   
>   #if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
>   
> +/**
> + * v4l2_subdev_get_fmt_ptr - Obtain a pointer to V4L2 sub-device format for pad
> + *			     and stream
> + * @sd: subdevice
> + * @state: subdevice state
> + * @pad: the pad on the sub-device
> + * @stream: stream in the pad
> + *
> + * For given pad and stream, obtain a pointer to the mbus format from the
> + * sub-device.
> + *
> + * Returns NULL if the format is not found or the parameters are invalid.
> + */
> +struct v4l2_mbus_framefmt *
> +v4l2_subdev_get_fmt_ptr(struct v4l2_subdev *sd,
> +			struct v4l2_subdev_state *state, unsigned int pad,
> +			unsigned int stream);
> +
> +/**
> + * v4l2_subdev_get_crop_ptr - Obtain a pointer to V4L2 sub-device crop
> + *			      rectangle for pad and stream
> + * @sd: subdevice
> + * @state: subdevice state
> + * @pad: the pad on the sub-device
> + * @stream: stream in the pad
> + *
> + * For given pad and stream, obtain a pointer to the crop selection rectangle
> + * from the sub-device.
> + *
> + * Returns NULL if the selection rectangle is not found or the parameters are
> + * invalid.
> + */
> +struct v4l2_rect *
> +v4l2_subdev_get_crop_ptr(struct v4l2_subdev *sd,
> +			 struct v4l2_subdev_state *state, unsigned int pad,
> +			 unsigned int stream);
> +
> +/**
> + * v4l2_subdev_get_compose_ptr - Obtain a pointer to V4L2 sub-device compose
> + *				 rectangle for pad and stream
> + * @sd: subdevice
> + * @state: subdevice state
> + * @pad: the pad on the sub-device
> + * @stream: stream in the pad
> + *
> + * For given pad and stream, obtain a pointer to the compose selection rectangle
> + * from the sub-device.
> + *
> + * Returns NULL if the selection rectangle is not found or the parameters are
> + * invalid.
> + */
> +struct v4l2_rect *
> +v4l2_subdev_get_compose_ptr(struct v4l2_subdev *sd,
> +			    struct v4l2_subdev_state *state, unsigned int pad,
> +			    unsigned int stream);
> +
>   /**
>    * v4l2_subdev_get_fmt() - Fill format based on state
>    * @sd: subdevice

Re: [PATCH v6 09/28] media: Documentation: Document embedded data guidelines for camera sensors

[Sakari Ailus]

Hi Hans,

On Thu, Oct 05, 2023 at 12:10:29PM +0200, Hans Verkuil wrote:
> On 03/10/2023 13:52, Sakari Ailus wrote:
> > Document how embedded data support should be implemented for camera
> > sensors, and when and how CCS embedded data format should be referenced.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  .../media/drivers/camera-sensor.rst           | 28 +++++++++++++++++++
> >  1 file changed, 28 insertions(+)
> > 
> > diff --git a/Documentation/userspace-api/media/drivers/camera-sensor.rst b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> > index 919a50e8b9d9..308f391c5ca1 100644
> > --- a/Documentation/userspace-api/media/drivers/camera-sensor.rst
> > +++ b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> > @@ -102,3 +102,31 @@ register programming sequences shall initialize the :ref:`V4L2_CID_HFLIP
> >  values programmed by the register sequences. The default values of these
> >  controls shall be 0 (disabled). Especially these controls shall not be inverted,
> >  independently of the sensor's mounting rotation.
> > +
> > +Embedded data
> > +-------------
> > +
> > +Many sensors, mostly raw sensors, support embedded data which is used to convey
> > +the sensor configuration for the captured frame back to the host. While CSI-2 is
> > +the most common bus used by such sensors, embedded data is not entirely limited
> > +to CSI-2 bus due to e.g. bridge devices.
> 
> I'm not quite sure what you mean with "embedded data is not entirely limited
> to CSI-2 bus due to e.g. bridge devices": do you just want to say: "embedded data
> can be available on other bus types as well."?

Bridge devices that transmit what they received over CSI-2 may result in
embedded data on parallel bus, for instance.

I'm fine with using the text you suggested, it's equally true as well.

> 
> > +
> > +Embedded data support should use an internal source pad and route to the
> 
> should or shall/must?


I'll change this to shall.

-- 
Regards,

Sakari Ailus

Re: [PATCH v6 09/28] media: Documentation: Document embedded data guidelines for camera sensors

[Sakari Ailus]

Moi,

On Thu, Oct 05, 2023 at 02:53:25PM +0300, Tomi Valkeinen wrote:
> On 05/10/2023 13:10, Hans Verkuil wrote:
> > On 03/10/2023 13:52, Sakari Ailus wrote:
> > > Document how embedded data support should be implemented for camera
> > > sensors, and when and how CCS embedded data format should be referenced.
> > > 
> > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > ---
> > >   .../media/drivers/camera-sensor.rst           | 28 +++++++++++++++++++
> > >   1 file changed, 28 insertions(+)
> > > 
> > > diff --git a/Documentation/userspace-api/media/drivers/camera-sensor.rst b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> > > index 919a50e8b9d9..308f391c5ca1 100644
> > > --- a/Documentation/userspace-api/media/drivers/camera-sensor.rst
> > > +++ b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> > > @@ -102,3 +102,31 @@ register programming sequences shall initialize the :ref:`V4L2_CID_HFLIP
> > >   values programmed by the register sequences. The default values of these
> > >   controls shall be 0 (disabled). Especially these controls shall not be inverted,
> > >   independently of the sensor's mounting rotation.
> > > +
> > > +Embedded data
> > > +-------------
> > > +
> > > +Many sensors, mostly raw sensors, support embedded data which is used to convey
> > > +the sensor configuration for the captured frame back to the host. While CSI-2 is
> > > +the most common bus used by such sensors, embedded data is not entirely limited
> > > +to CSI-2 bus due to e.g. bridge devices.
> > 
> > I'm not quite sure what you mean with "embedded data is not entirely limited
> > to CSI-2 bus due to e.g. bridge devices": do you just want to say: "embedded data
> > can be available on other bus types as well."?
> 
> Right, nothing CSI-2 specific here. I have a parallel sensor that sends
> embedded data as the last (or was it first?) few lines of the frame.
> 
> Also, is this about sensors only, and more specifically, about generating
> the embedded data? If yes, why mention bridge devices (and should the title
> be "Embedded data generation" or such)? If not, is it about metadata in
> general, in which case bridges come into play?
> 
> > > +
> > > +Embedded data support should use an internal source pad and route to the
> > 
> > should or shall/must?
> 
> Is the above necessary? I don't see anything wrong with, say, having a
> sensor represented with multiple subdevs, and one subdev is used for the
> embedded data generation.
> 
> Basically, anything that can be represented with internal pads can also be
> done with multiple subdevs, although I think you don't want to go the
> multiple subdev route unless absolutely necessary.

True. This is trying to convey to the user space developers what's possible
vs. suggesting driver developers what to do. I'd expect embedded data to
use internal source pads, anything else will need to be separately
agreed upon.

-- 
Terveisin,

Sakari Ailus

Re: [PATCH v6 09/28] media: Documentation: Document embedded data guidelines for camera sensors

[Sakari Ailus]

Hi Hans,

On Thu, Oct 05, 2023 at 12:14:05PM +0200, Hans Verkuil wrote:
> On 03/10/2023 13:52, Sakari Ailus wrote:
> > Document how embedded data support should be implemented for camera
> > sensors, and when and how CCS embedded data format should be referenced.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  .../media/drivers/camera-sensor.rst           | 28 +++++++++++++++++++
> >  1 file changed, 28 insertions(+)
> > 
> > diff --git a/Documentation/userspace-api/media/drivers/camera-sensor.rst b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> > index 919a50e8b9d9..308f391c5ca1 100644
> > --- a/Documentation/userspace-api/media/drivers/camera-sensor.rst
> > +++ b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> > @@ -102,3 +102,31 @@ register programming sequences shall initialize the :ref:`V4L2_CID_HFLIP
> >  values programmed by the register sequences. The default values of these
> >  controls shall be 0 (disabled). Especially these controls shall not be inverted,
> >  independently of the sensor's mounting rotation.
> > +
> > +Embedded data
> > +-------------
> > +
> > +Many sensors, mostly raw sensors, support embedded data which is used to convey
> > +the sensor configuration for the captured frame back to the host. While CSI-2 is
> > +the most common bus used by such sensors, embedded data is not entirely limited
> > +to CSI-2 bus due to e.g. bridge devices.
> > +
> > +Embedded data support should use an internal source pad and route to the
> 
> "internal source pad" -> "internal sink pad"?
> 
> > +external pad. If embedded data output can be disabled in hardware, it should be
> 
> "external pad" -> "external source pad"?
> 
> Or perhaps "a source pad of the entity"?

The documentation discusses "internal source pads" that are in fact
internal pads with the INTERNAL and SINK flags set. This is probably
present in other patches, too. The terms are indeed confusing, I'll see if
I could rephrase it.

Of the two options, I prefer the former.

-- 
Sakari Ailus

Re: [PATCH v6 13/28] media: v4l: subdev: Add a function to lock two sub-device states, use it

[Tomi Valkeinen]

On 03/10/2023 15:07, Sakari Ailus wrote:
> Add two new functions, v4l2_subdev_lock_states() and
> v4l2_subdev_unclock_states(), to acquire and release the state of two
> sub-devices. They differ from calling v4l2_subdev_{un,}lock_state() so
> that if the two states share the same lock, the lock is acquired only
> once.
> 
> Also use the new functions in v4l2_subdev_link_validate().
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>   drivers/media/v4l2-core/v4l2-subdev.c | 12 +++-----
>   include/media/v4l2-subdev.h           | 40 +++++++++++++++++++++++++++
>   2 files changed, 44 insertions(+), 8 deletions(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 854f9d4db923..df9a1ae65410 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -1377,17 +1377,13 @@ int v4l2_subdev_link_validate(struct media_link *link)
>   
>   	states_locked = sink_state && source_state;
>   
> -	if (states_locked) {
> -		v4l2_subdev_lock_state(sink_state);
> -		v4l2_subdev_lock_state(source_state);
> -	}
> +	if (states_locked)
> +		v4l2_subdev_lock_states(sink_state, source_state);
>   
>   	ret = v4l2_subdev_link_validate_locked(link, states_locked);
>   
> -	if (states_locked) {
> -		v4l2_subdev_unlock_state(sink_state);
> -		v4l2_subdev_unlock_state(source_state);
> -	}
> +	if (states_locked)
> +		v4l2_subdev_unlock_states(sink_state, source_state);
>   
>   	return ret;
>   }
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 7c34243ffed9..e49e8af2fb52 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -1418,6 +1418,46 @@ static inline void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state)
>   	mutex_unlock(state->lock);
>   }
>   
> +/**
> + * v4l2_subdev_lock_states - Lock two sub-device states
> + * @state1: One subdevice state
> + * @state2: The other subdevice state
> + *
> + * Locks the state of two sub-devices.
> + *
> + * The states must be unlocked with v4l2_subdev_unlock_states() after use.
> + *
> + * This differs from calling v4l2_subdev_lock_state() on both states so that if
> + * the states share the same lock, the lock is acquired only once (so no
> + * deadlock occurs). Note that it must be ensured the locks must always be
> + * acquired in the same order.

Either s/must/will/ or maybe "note that the locks must always be 
acquired...".

Maybe it should be stated that state1 and state2 have to be from 
subdevices that are connected via a media link and something like "the 
sink subdevice must always be state 1, and the source subdevice must 
always be state2".

  Tomi

> + */
> +static inline void v4l2_subdev_lock_states(struct v4l2_subdev_state *state1,
> +					   struct v4l2_subdev_state *state2)
> +{
> +	mutex_lock(state1->lock);
> +	if (state1->lock != state2->lock)
> +		mutex_lock(state2->lock);
> +}
> +
> +/**
> + * v4l2_subdev_unlock_states() - Unlock two sub-device states
> + * @state1: One subdevice state
> + * @state2: The other subdevice state
> + *
> + * Unlocks the state of two sub-devices.
> + *
> + * This differs from calling v4l2_subdev_unlock_state() on both states so that if
> + * the states share the same lock, the lock is released only once.
> + */
> +static inline void v4l2_subdev_unlock_states(struct v4l2_subdev_state *state1,
> +					     struct v4l2_subdev_state *state2)
> +{
> +	mutex_unlock(state1->lock);
> +	if (state1->lock != state2->lock)
> +		mutex_unlock(state2->lock);
> +}
> +
>   /**
>    * v4l2_subdev_get_unlocked_active_state() - Checks that the active subdev state
>    *					     is unlocked and returns it

Re: [PATCH v6 14/28] media: v4l: subdev: Move G_ROUTING handling below S_ROUTING

[Tomi Valkeinen]

On 03/10/2023 15:07, Sakari Ailus wrote:
> Move G_ROUTING IOCTL handling below that of S_ROUTING. G_ROUTING
> implementation will soon needed in handling S_ROUTING as well.

There's probably a "be" missing in the above text. Also, it's a bit 
unclear on why this is needed. You could say it'll allow switch-case 
fall-through.

  Tomi

> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>   drivers/media/v4l2-core/v4l2-subdev.c | 54 +++++++++++++--------------
>   1 file changed, 27 insertions(+), 27 deletions(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index df9a1ae65410..614ff0031831 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -887,33 +887,6 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>   	case VIDIOC_SUBDEV_QUERYSTD:
>   		return v4l2_subdev_call(sd, video, querystd, arg);
>   
> -	case VIDIOC_SUBDEV_G_ROUTING: {
> -		struct v4l2_subdev_routing *routing = arg;
> -		struct v4l2_subdev_krouting *krouting;
> -
> -		if (!v4l2_subdev_enable_streams_api)
> -			return -ENOIOCTLCMD;
> -
> -		if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS))
> -			return -ENOIOCTLCMD;
> -
> -		memset(routing->reserved, 0, sizeof(routing->reserved));
> -
> -		krouting = &state->routing;
> -
> -		if (routing->num_routes < krouting->num_routes) {
> -			routing->num_routes = krouting->num_routes;
> -			return -ENOSPC;
> -		}
> -
> -		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
> -		       krouting->routes,
> -		       krouting->num_routes * sizeof(*krouting->routes));
> -		routing->num_routes = krouting->num_routes;
> -
> -		return 0;
> -	}
> -
>   	case VIDIOC_SUBDEV_S_ROUTING: {
>   		struct v4l2_subdev_routing *routing = arg;
>   		struct v4l2_subdev_route *routes =
> @@ -962,6 +935,33 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>   					routing->which, &krouting);
>   	}
>   
> +	case VIDIOC_SUBDEV_G_ROUTING: {
> +		struct v4l2_subdev_routing *routing = arg;
> +		struct v4l2_subdev_krouting *krouting;
> +
> +		if (!v4l2_subdev_enable_streams_api)
> +			return -ENOIOCTLCMD;
> +
> +		if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS))
> +			return -ENOIOCTLCMD;
> +
> +		memset(routing->reserved, 0, sizeof(routing->reserved));
> +
> +		krouting = &state->routing;
> +
> +		if (routing->num_routes < krouting->num_routes) {
> +			routing->num_routes = krouting->num_routes;
> +			return -ENOSPC;
> +		}
> +
> +		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
> +		       krouting->routes,
> +		       krouting->num_routes * sizeof(*krouting->routes));
> +		routing->num_routes = krouting->num_routes;
> +
> +		return 0;
> +	}
> +
>   	case VIDIOC_SUBDEV_G_CLIENT_CAP: {
>   		struct v4l2_subdev_client_capability *client_cap = arg;
>   

Re: [PATCH v6 18/28] media: uapi: Allow a larger number of routes than there's room for

[Tomi Valkeinen]

On 03/10/2023 15:08, Sakari Ailus wrote:
> On VIDIOC_SUBDEV_[GS]_ROUTING, only return as many routes back to the user
> as there's room. Do not consider it an error if more routes existed.
> Simply inform the user there are more routes.

Inform how? And I agree with Hans here. How about return ENOSPC, but the 
kernel fills in num_routes to tell the userspace how many there actually 
are?

  Tomi

> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>   .../userspace-api/media/v4l/vidioc-subdev-g-routing.rst   | 4 ----
>   drivers/media/v4l2-core/v4l2-subdev.c                     | 8 ++------
>   2 files changed, 2 insertions(+), 10 deletions(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> index ced53ea5f23c..99d3c15fd759 100644
> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> @@ -145,10 +145,6 @@ On success 0 is returned, on error -1 and the ``errno`` variable is set
>   appropriately. The generic error codes are described at the
>   :ref:`Generic Error Codes <gen-errors>` chapter.
>   
> -ENOSPC
> -   The application provided ``num_routes`` is not big enough to contain
> -   all the available routes the subdevice exposes.
> -
>   EINVAL
>      The sink or source pad identifiers reference a non-existing pad, or reference
>      pads of different types (ie. the sink_pad identifiers refers to a source pad).
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 9a34e13dfd96..dd48e7e549fb 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -956,14 +956,10 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>   
>   		krouting = &state->routing;
>   
> -		if (routing->len_routes < krouting->num_routes) {
> -			routing->num_routes = krouting->num_routes;
> -			return -ENOSPC;
> -		}
> -
>   		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
>   		       krouting->routes,
> -		       krouting->num_routes * sizeof(*krouting->routes));
> +		       min(krouting->num_routes, routing->len_routes) *
> +		       sizeof(*krouting->routes));
>   		routing->num_routes = krouting->num_routes;
>   
>   		return 0;

Re: [PATCH v6 19/28] media: v4l: subdev: Add trivial set_routing support

[Tomi Valkeinen]

On 03/10/2023 15:08, Sakari Ailus wrote:
> Add trivial S_ROUTING IOCTL support for drivers where routing is static.
> Essentially this means returning the same information G_ROUTING call would
> have done.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>   drivers/media/v4l2-core/v4l2-subdev.c | 5 +++++
>   1 file changed, 5 insertions(+)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index dd48e7e549fb..7d7028de581a 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -894,6 +894,9 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>   		struct v4l2_subdev_krouting krouting = {};
>   		unsigned int i;
>   
> +		if (!v4l2_subdev_has_op(sd, pad, set_routing))
> +			goto do_vidioc_subdev_g_routing;
> +
>   		if (!v4l2_subdev_enable_streams_api)
>   			return -ENOIOCTLCMD;
>   
> @@ -939,6 +942,8 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>   					routing->which, &krouting);
>   		if (rval < 0)
>   			return rval;
> +do_vidioc_subdev_g_routing:
> +		;
>   	}
>   		fallthrough;
>   

As Hans mentioned in an earlier comment, the code would perhaps be more 
readable if the s_routing ioctl code would just handle the g_routing 
part itself. Or we could have a helper function, called by both 
s_routing and g_routing. It looks very confusing with these patches.

  Tomi

Re: [PATCH v6 00/28] Generic line based metadata support, internal pads

[Sakari Ailus]

Moi,

On Thu, Oct 05, 2023 at 02:17:07PM +0300, Tomi Valkeinen wrote:
> On 05/10/2023 12:04, Sakari Ailus wrote:
> > Hyvää huomenta!
> > 
> > On Thu, Oct 05, 2023 at 11:05:54AM +0300, Tomi Valkeinen wrote:
> > > Hi!
> > > 
> > > Thanks for working on this. I think this series is very important.
> > > 
> > > On 03/10/2023 14:52, Sakari Ailus wrote:
> > > > Hi folks,
> > > > 
> > > > Here are a few patches to add support generic, line based metadata as well
> > > > as internal pads. While the amount of code is not very large, to the
> > > > contrary it is quite small actually IMO, I presume what this is about and
> > > > why it is being proposed requires some explaining.
> > > > 
> > > > Metadata mbus codes and formats have existed for some time in V4L2. They
> > > > however have been only used by drivers that produce the data itself and
> > > > effectively this metadata has always been statistics of some sort (at
> > > > least when it comes to ISPs). What is different here is that we intend to
> > > > add support for metadata originating from camera sensors.
> > > > 
> > > > Camera sensors produce different kinds of metadata, embedded data (usually
> > > > register address--value pairs used to capture the frame, in a more or less
> > > > sensor specific format), histograms (in a very sensor specific format),
> > > > dark pixels etc. The number of these formats is probably going to be about
> > > > as large as image data formats if not larger, as the image data formats
> > > > are much better standardised but a smaller subset of them will be
> > > > supported by V4L2, at least initially but possibly much more in the long
> > > > run.
> > > > 
> > > > Having this many device specific formats would be a major problem for all
> > > > the other drivers along that pipeline (not to mention the users of those
> > > > drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2
> > > > receiver drivers that have DMA: the poor driver developer would not only
> > > > need to know camera sensor specific formats but to choose the specific
> > > > packing of that format suitable for the DMA used by the hardware. It is
> > > > unlikely many of these would ever get tested while being present on the
> > > > driver API. Also adding new sensors with new embedded data formats would
> > > > involve updating all bridge and CSI-2 receiver drivers. I don't expect
> > > > this to be a workable approach.
> > > > 
> > > > Instead what I'm proposing is to use specific metadata formats on the
> > > > sensor devices only, on internal pads (more about those soon) of the
> > > > sensors, only visible in the UAPI, and then generic mbus formats along the
> > > 
> > > What do you mean with "only visible in the UAPI"?
> > 
> > Other drivers won't bother with specific metadata formats: they are only
> > present on the internal pads while external pads have generic formats.
> > 
> > > 
> > > > pipeline and finally generic V4L2 metadata formats on the DMAs (specific
> > > > to bit depth and packing). This would unsnarl the two, defining what data
> > > > there is (specific mbus code) and how that is transported and packed
> > > > (generic mbus codes and V4L2 formats).
> > > > 
> > > > The user space would be required to "know" the path of that data from the
> > > > sensor's internal pad to the V4L2 video node. I do not see this as these
> > > > devices require at least some knowledge of the pipeline, i.e. hardware at
> > > > hand. Separating what the data means and how it is packed may even be
> > > > beneficial: it allows separating code that interprets the data (sensor
> > > > internal mbus code) from the code that accesses it (packing).
> > > > 
> > > > These formats are in practice line based, meaning that there may be
> > > > padding at the end of the line, depending on the bus as well as the DMA.
> > > > If non-line based formats are needed, it is always possible to set the
> > > > "height" field to 1.
> > > > 
> > > > The internal (source) pads are an alternative to source routes [1]. The
> > > > source routes were not universally liked and I do have to say I like
> > > > re-using existing interface concepts (pads and everything you can do with
> > > > pads, including access format, selections etc.) wherever it makes sense,
> > > > instead of duplicating functionality.
> > > > 
> > > > Effectively internal source pads behave mostly just like sink pads, but
> > > > they describe a flow of data that originates from a sub-device instead of
> > > > arriving to a sub-device. The SUBDEV_S_ROUTING IOCTLs are used to enable
> > > > and disable routes from internal source pads to sub-device's source pads.
> > > > The subdev format IOCTLs are usable, too, so one can find which subdev
> > > > format is available on given internal source pad.
> > > 
> > > I think the internal pads require a bit more praise, as they can be used for
> > > other things too. E.g. the ds90ub953 FPD-Link serializer has a test pattern
> > > generator, which can be modeled very nicely with internal pads. The internal
> > > pad represents the TPG, and the user can use routing to choose if the output
> > > of the device is sourced from the normal input or from the TPG. And one can
> > > set the format on the TPG pad, thus configuring the TPG.
> > 
> > Well, yes, indeed.
> > 
> > Could you review especially the documentation patches to ensure we're
> > aligned on this?
> 
> Sure.
> 
> > > 
> > > > This set depends on these patches:
> > > > 
> > > > <URL:https://lore.kernel.org/linux-media/20231002105557.28972-1-sakari.ailus@linux.intel.com/T/#t>
> > > 
> > > Hmm, it's a bit odd for a generic series to depend on a device specific
> > > series. That makes backporting these more difficult. Why do these depend on
> > > ov2740 and css patches?
> > 
> > Patchset-wise that is the dependency, individual patches may be backported
> > without backporting _all_ driver patches in the previous set. However, if
> > you need those drivers as well, then you'll need to backport these patches,
> > too.
> 
> Ok. I don't like the structure of these serieses that much.
> 
> I haven't reviewed the patches yet, so maybe there are reasons for the
> structure, but I'd rather have, in this order, preferably as separate
> serieses:
> 
> - cleanups/fixes not directly related to ccs, embedded data or internal pads
> - internal pads
> - embedded data
> - driver changes
> 
> That would help in backporting, but it'd also help with the reviews as the
> series would be more concentrated on a single topic.

It's also useful to use new APIs once they're added rather than (much)
later in the set. This is one of the reasons why the patches are arranged as
they are.

I'll see if this could be taken better into account and if further cleanups
could be merged earlier, I think there are at least a couple of such
patches.

-- 
Regards,

Sakari Ailus

Re: [PATCH v6 20/28] media: uapi: v4l: subdev: Enable streams API

[Tomi Valkeinen]

On 03/10/2023 15:08, Sakari Ailus wrote:
> Remove v4l2_subdev_enable_streams_api variable that was used to easily
> enable streams API for development, and conditions that use the variable.
> 
> This patch enables the streams API for V4L2 sub-device interface which
> allows transporting multiple streams on a single MC link.

I wouldn't go there yet. Just in this series you break the API.

  Tomi


> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>   drivers/media/v4l2-core/v4l2-subdev.c | 23 -----------------------
>   1 file changed, 23 deletions(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 7d7028de581a..5e24a638bdba 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -26,15 +26,6 @@
>   #include <media/v4l2-fh.h>
>   #include <media/v4l2-ioctl.h>
>   
> -#if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
> -/*
> - * The Streams API is an experimental feature. To use the Streams API, set
> - * 'v4l2_subdev_enable_streams_api' to 1 below.
> - */
> -
> -static bool v4l2_subdev_enable_streams_api;
> -#endif
> -
>   /*
>    * Maximum stream ID is 63 for now, as we use u64 bitmask to represent a set
>    * of streams.
> @@ -897,9 +888,6 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>   		if (!v4l2_subdev_has_op(sd, pad, set_routing))
>   			goto do_vidioc_subdev_g_routing;
>   
> -		if (!v4l2_subdev_enable_streams_api)
> -			return -ENOIOCTLCMD;
> -
>   		if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS))
>   			return -ENOIOCTLCMD;
>   
> @@ -951,9 +939,6 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>   		struct v4l2_subdev_routing *routing = arg;
>   		struct v4l2_subdev_krouting *krouting;
>   
> -		if (!v4l2_subdev_enable_streams_api)
> -			return -ENOIOCTLCMD;
> -
>   		if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS))
>   			return -ENOIOCTLCMD;
>   
> @@ -981,14 +966,6 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>   	case VIDIOC_SUBDEV_S_CLIENT_CAP: {
>   		struct v4l2_subdev_client_capability *client_cap = arg;
>   
> -		/*
> -		 * Clear V4L2_SUBDEV_CLIENT_CAP_STREAMS if streams API is not
> -		 * enabled. Remove this when streams API is no longer
> -		 * experimental.
> -		 */
> -		if (!v4l2_subdev_enable_streams_api)
> -			client_cap->capabilities &= ~V4L2_SUBDEV_CLIENT_CAP_STREAMS;
> -
>   		/* Filter out unsupported capabilities */
>   		client_cap->capabilities &= V4L2_SUBDEV_CLIENT_CAP_STREAMS;
>   

Re: [PATCH v6 10/28] media: Documentation: v4l: Document source routes

[Sakari Ailus]

Moi,

On Thu, Oct 05, 2023 at 03:37:01PM +0300, Tomi Valkeinen wrote:
> On 03/10/2023 15:07, Sakari Ailus wrote:
> > Document how internal pads are used on source routes. Use the IMX219
> > camera sensor as an example.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >   .../userspace-api/media/v4l/dev-subdev.rst    | 179 ++++++++++++++++++
> >   1 file changed, 179 insertions(+)
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > index a387e8a15b8d..fb73a95401c3 100644
> > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > @@ -553,6 +553,27 @@ A stream at a specific point in the media pipeline is identified by the
> >   sub-device and a (pad, stream) pair. For sub-devices that do not support
> >   multiplexed streams the 'stream' field is always 0.
> > +.. _v4l2-subdev-source-routes:
> > +
> > +Internal pads and source routes
> > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > +
> > +Cases where a single sub-device source pad is traversed by multiple streams one
> > +or more of which originate from within the sub-device itself are special as
> > +there is no external sink pad for such routes. In those cases, the sources of
> > +the internally generated streams are represented by internal sink pads, which
> > +are sink pads that have the :ref:`MEDIA_PAD_FL_INTERNAL <MEDIA-PAD-FL-INTERNAL>`
> > +pad flag set.
> > +
> > +Internal pads have all the properties of an external pad, including formats and
> > +selections. The format in this case is the source format of the stream. An
> > +internal pad always has a single stream only (0).
> > +
> > +*Source routes* are routes from an internal sink pad to an external source
> > +pad. In most cases source routes are not modifiable but they can be activated
> > +and deactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > +<v4l2-subdev-routing-flags>` flag, depending on driver capabilities.
> > +
> >   Interaction between routes, streams, formats and selections
> >   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > @@ -668,3 +689,161 @@ To configure this pipeline, the userspace must take the following steps:
> >      the configurations along the stream towards the receiver, using
> >      :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each
> >      stream endpoint in each sub-device.
> > +
> > +Internal pads setup example
> > +^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > +
> > +A simple example of a multiplexed stream setup might be as follows:
> > +
> > +- A CCS camera sensor source sub-device, with one sink pad (0), one source pad
> > +  (1), an internal sink pad (2) that represents the source of embedded
> 
> Wouldn't a simpler and better example be a sensor with a single sub-device,
> with two internal pads and one external pad? Now your example is a partial
> example, as the sensor subdevice has a external sink pad connected to
> something which is not covered in the example.
> 
> Or is part of the point here to show an example with a subdevice with both
> internal and external sink pads?

Ah. I forgot to change this part of the text --- I'll address this for v7.
There was supposed to be a single sub-device here.

> 
> > +  data. There are two routes, one from the sink pad to the source, and another
> > +  from the internal sink pad to the source pad. The embedded data stream needs
> > +  to be enabled by activating the related route. The configuration of the rest
> > +  of the CCS sub-devices is omitted from this example.
> > +
> > +- Multiplexer bridge (Bridge). The bridge has one sink pad, connected to the
> > +  sensor (pad 0), and one source pad (pad 1), which outputs two streams.
> 
> What does it multiplex if there's a single input and a single output?

The name is probably from the other example --- simply "bridge" is probably
a better name for this.

> 
> > +- Receiver in the SoC (Receiver). The receiver has a single sink pad (pad 0),
> > +  connected to the bridge, and two source pads (pads 1-2), going to the DMA
> > +  engine. The receiver demultiplexes the incoming streams to the source pads.
> > +
> > +- DMA Engines in the SoC (DMA Engine), one for each stream. Each DMA engine is
> > +  connected to a single source pad in the receiver.
> > +
> > +The sensor, the bridge and the receiver are modeled as V4L2 sub-devices,
> > +exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are
> > +modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
> > +
> > +To configure this pipeline, the userspace must take the following steps:
> > +
> > +1) Set up media links between entities: connect the sensors to the bridge,
> > +   bridge to the receiver, and the receiver to the DMA engines. This step does
> > +   not differ from normal non-multiplexed media controller setup.
> > +
> > +2) Configure routing
> > +
> > +.. flat-table:: Camera sensor
> > +    :header-rows: 1
> > +
> > +    * - Sink Pad/Stream
> > +      - Source Pad/Stream
> > +      - Routing Flags
> > +      - Comments
> > +    * - 0/0
> > +      - 1/0
> > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > +      - Pixel data stream from the sink pad
> > +    * - 2/0
> > +      - 1/1
> > +      - **V4L2_SUBDEV_ROUTE_FL_ACTIVE**
> 
> Why is this one bold?

Leftover from old CCS example. I'll remove it.

> 
> > +      - Metadata stream from the internal sink pad
> > +
> > +.. flat-table:: Bridge routing table
> > +    :header-rows: 1
> > +
> > +    * - Sink Pad/Stream
> > +      - Source Pad/Stream
> > +      - Routing Flags
> > +      - Comments
> > +    * - 0/0
> > +      - 1/0
> > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > +      - Pixel data stream from camera sensor
> > +    * - 0/1
> > +      - 1/1
> > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > +      - Metadata stream from camera sensor
> > +
> > +.. flat-table:: Receiver routing table
> > +    :header-rows:  1
> > +
> > +    * - Sink Pad/Stream
> > +      - Source Pad/Stream
> > +      - Routing Flags
> > +      - Comments
> > +    * - 0/0
> > +      - 1/0
> > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > +      - Pixel data stream from camera sensor
> > +    * - 0/1
> > +      - 2/0
> > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > +      - Metadata stream from camera sensor
> > +
> > +The options available in sensor's routing configuration are dictated by
> > +hardware capabilities: typically camera sensors always produce image data
> > +stream while the embedded data stream typically can be either enabled or
> > +disabled.
> > +
> > +3) Configure formats and selections
> > +
> > +   This example assumes that the formats are propagated from sink pad to the
> > +   source pad as-is. The tables contain fields of both struct v4l2_subdev_format
> > +   and struct v4l2_mbus_framefmt.
> > +
> > +.. flat-table:: Formats set on the sub-devices. Bold values are set, others are
> > +                static or propagated.
> > +    :header-rows: 1
> > +    :fill-cells:
> > +
> > +    * - Sub-device
> > +      - Pad/Stream
> > +      - Width
> > +      - Height
> > +      - Code
> > +    * - :rspan:`3` Camera sensor sub-device (IMX219)
> > +      - 1/0
> > +      - 3296
> > +      - 2480
> > +      - MEDIA_BUS_FMT_SRGGB10
> > +    * - 0/0
> 
> I think the 0/0 stream should be the first one in the table.

The order here is that of routes: this route is from pad/stream 1/0 to 0/0.

I could add this to the text above.

> 
> > +      - **3296**
> > +      - **2480**
> > +      - **MEDIA_BUS_FMT_SRGGB10**
> > +    * - 2/0
> > +      - 3296
> > +      - 2
> > +      - MEDIA_BUS_FMT_IMX219_EMBEDDED
> > +    * - 1/1
> > +      - 3296
> > +      - 2
> > +      - MEDIA_BUS_FMT_META_10
> > +    * - :rspan:`3` Bridge
> > +      - 0/0
> > +      - **3296**
> > +      - **2480**
> > +      - **MEDIA_BUS_FMT_SRGGB10**
> > +    * - 1/0
> > +      - 3296
> > +      - 2480
> > +      - MEDIA_BUS_FMT_SRGGB10
> > +    * - 0/1
> > +      - **3296**
> > +      - **2**
> > +      - **MEDIA_BUS_FMT_META_10**
> > +    * - 1/1
> > +      - 3296
> > +      - 2
> > +      - MEDIA_BUS_FMT_META_10
> > +    * - :rspan:`3` Receiver
> > +      - 0/0
> > +      - **3296**
> > +      - **2480**
> > +      - **MEDIA_BUS_FMT_SRGGB10**
> > +    * - 1/0
> > +      - 3296
> > +      - 2480
> > +      - MEDIA_BUS_FMT_SRGGB10
> > +    * - 0/1
> > +      - **3296**
> > +      - **2**
> > +      - **MEDIA_BUS_FMT_META_10**
> > +    * - 2/0
> > +      - 3296
> > +      - 2
> > +      - MEDIA_BUS_FMT_META_10
> > +
> > +The embedded data format does not need to be configured as the format is
> > +dictated by the pixel data format in this case.
> 

-- 
Regards,

Sakari Ailus

Re: [PATCH v6 10/28] media: Documentation: v4l: Document source routes

[Sakari Ailus]

Hi Hans,

On Thu, Oct 05, 2023 at 12:26:41PM +0200, Hans Verkuil wrote:
> On 03/10/2023 14:07, Sakari Ailus wrote:
> > Document how internal pads are used on source routes. Use the IMX219
> > camera sensor as an example.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  .../userspace-api/media/v4l/dev-subdev.rst    | 179 ++++++++++++++++++
> >  1 file changed, 179 insertions(+)
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > index a387e8a15b8d..fb73a95401c3 100644
> > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > @@ -553,6 +553,27 @@ A stream at a specific point in the media pipeline is identified by the
> >  sub-device and a (pad, stream) pair. For sub-devices that do not support
> >  multiplexed streams the 'stream' field is always 0.
> >  
> > +.. _v4l2-subdev-source-routes:
> > +
> > +Internal pads and source routes
> > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > +
> > +Cases where a single sub-device source pad is traversed by multiple streams one
> 
> Comma after "streams"
> 
> > +or more of which originate from within the sub-device itself are special as
> 
> and comma after "itself".

Yes.

> 
> > +there is no external sink pad for such routes. In those cases, the sources of
> > +the internally generated streams are represented by internal sink pads, which
> > +are sink pads that have the :ref:`MEDIA_PAD_FL_INTERNAL <MEDIA-PAD-FL-INTERNAL>`
> > +pad flag set.
> > +
> > +Internal pads have all the properties of an external pad, including formats and
> 
> Perhaps the glossary should have definitions for "internal pad" and "external pad"?
> Or define it somewhere in:
> 
> https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/mediactl/media-controller.html

I'll add them to the glossary.

> 
> > +selections. The format in this case is the source format of the stream. An
> > +internal pad always has a single stream only (0).
> > +
> > +*Source routes* are routes from an internal sink pad to an external source
> > +pad. In most cases source routes are not modifiable but they can be activated
> > +and deactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > +<v4l2-subdev-routing-flags>` flag, depending on driver capabilities.
> > +
> >  Interaction between routes, streams, formats and selections
> >  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> >  
> > @@ -668,3 +689,161 @@ To configure this pipeline, the userspace must take the following steps:
> >     the configurations along the stream towards the receiver, using
> >     :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each
> >     stream endpoint in each sub-device.
> > +
> > +Internal pads setup example
> > +^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > +
> > +A simple example of a multiplexed stream setup might be as follows:
> > +
> > +- A CCS camera sensor source sub-device, with one sink pad (0), one source pad
> > +  (1), an internal sink pad (2) that represents the source of embedded
> > +  data. There are two routes, one from the sink pad to the source, and another
> > +  from the internal sink pad to the source pad. The embedded data stream needs
> > +  to be enabled by activating the related route. The configuration of the rest
> > +  of the CCS sub-devices is omitted from this example.
> > +
> > +- Multiplexer bridge (Bridge). The bridge has one sink pad, connected to the
> > +  sensor (pad 0), and one source pad (pad 1), which outputs two streams.
> 
> I think this should mention "CSI" somewhere, since this transmits over the CSI bus.

I'll add CSI-2 in v7.

I'll also remove the bridge in v7 as this isn't a common configuration.

> 
> > +
> > +- Receiver in the SoC (Receiver). The receiver has a single sink pad (pad 0),
> > +  connected to the bridge, and two source pads (pads 1-2), going to the DMA
> > +  engine. The receiver demultiplexes the incoming streams to the source pads.
> 
> Ditto: this receives data from the CSI bus.

Yes.

> 
> > +
> > +- DMA Engines in the SoC (DMA Engine), one for each stream. Each DMA engine is
> > +  connected to a single source pad in the receiver.
> > +
> > +The sensor, the bridge and the receiver are modeled as V4L2 sub-devices,
> > +exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are
> > +modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
> > +
> > +To configure this pipeline, the userspace must take the following steps:
> > +
> > +1) Set up media links between entities: connect the sensors to the bridge,
> > +   bridge to the receiver, and the receiver to the DMA engines. This step does
> > +   not differ from normal non-multiplexed media controller setup.
> > +
> > +2) Configure routing
> > +
> > +.. flat-table:: Camera sensor
> > +    :header-rows: 1
> > +
> > +    * - Sink Pad/Stream
> > +      - Source Pad/Stream
> > +      - Routing Flags
> > +      - Comments
> > +    * - 0/0
> > +      - 1/0
> > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > +      - Pixel data stream from the sink pad
> > +    * - 2/0
> > +      - 1/1
> > +      - **V4L2_SUBDEV_ROUTE_FL_ACTIVE**
> > +      - Metadata stream from the internal sink pad
> > +
> > +.. flat-table:: Bridge routing table
> > +    :header-rows: 1
> > +
> > +    * - Sink Pad/Stream
> > +      - Source Pad/Stream
> > +      - Routing Flags
> > +      - Comments
> > +    * - 0/0
> > +      - 1/0
> > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > +      - Pixel data stream from camera sensor
> > +    * - 0/1
> > +      - 1/1
> > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > +      - Metadata stream from camera sensor
> > +
> > +.. flat-table:: Receiver routing table
> > +    :header-rows:  1
> > +
> > +    * - Sink Pad/Stream
> > +      - Source Pad/Stream
> > +      - Routing Flags
> > +      - Comments
> > +    * - 0/0
> > +      - 1/0
> > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > +      - Pixel data stream from camera sensor
> > +    * - 0/1
> > +      - 2/0
> > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > +      - Metadata stream from camera sensor
> > +
> > +The options available in sensor's routing configuration are dictated by
> > +hardware capabilities: typically camera sensors always produce image data
> 
> image data -> an image data

Yes.

> 
> > +stream while the embedded data stream typically can be either enabled or
> > +disabled.
> > +
> > +3) Configure formats and selections
> > +
> > +   This example assumes that the formats are propagated from sink pad to the
> > +   source pad as-is. The tables contain fields of both struct v4l2_subdev_format
> > +   and struct v4l2_mbus_framefmt.
> > +
> > +.. flat-table:: Formats set on the sub-devices. Bold values are set, others are
> > +                static or propagated.
> > +    :header-rows: 1
> > +    :fill-cells:
> > +
> > +    * - Sub-device
> > +      - Pad/Stream
> > +      - Width
> > +      - Height
> > +      - Code
> > +    * - :rspan:`3` Camera sensor sub-device (IMX219)
> > +      - 1/0
> > +      - 3296
> > +      - 2480
> > +      - MEDIA_BUS_FMT_SRGGB10
> > +    * - 0/0
> > +      - **3296**
> > +      - **2480**
> > +      - **MEDIA_BUS_FMT_SRGGB10**
> > +    * - 2/0
> > +      - 3296
> > +      - 2
> > +      - MEDIA_BUS_FMT_IMX219_EMBEDDED
> > +    * - 1/1
> > +      - 3296
> > +      - 2
> > +      - MEDIA_BUS_FMT_META_10
> > +    * - :rspan:`3` Bridge
> > +      - 0/0
> > +      - **3296**
> > +      - **2480**
> > +      - **MEDIA_BUS_FMT_SRGGB10**
> > +    * - 1/0
> > +      - 3296
> > +      - 2480
> > +      - MEDIA_BUS_FMT_SRGGB10
> > +    * - 0/1
> > +      - **3296**
> > +      - **2**
> > +      - **MEDIA_BUS_FMT_META_10**
> > +    * - 1/1
> > +      - 3296
> > +      - 2
> > +      - MEDIA_BUS_FMT_META_10
> > +    * - :rspan:`3` Receiver
> > +      - 0/0
> > +      - **3296**
> > +      - **2480**
> > +      - **MEDIA_BUS_FMT_SRGGB10**
> > +    * - 1/0
> > +      - 3296
> > +      - 2480
> > +      - MEDIA_BUS_FMT_SRGGB10
> > +    * - 0/1
> > +      - **3296**
> > +      - **2**
> > +      - **MEDIA_BUS_FMT_META_10**
> > +    * - 2/0
> > +      - 3296
> > +      - 2
> > +      - MEDIA_BUS_FMT_META_10
> > +
> > +The embedded data format does not need to be configured as the format is
> > +dictated by the pixel data format in this case.
> 
> "pixel data format"? That doesn't sound right for embedded data.

This is meant to convey that the configuration of embedded data stream is
dictated by the pixel data stream: I don't think it'd be useful to be able
to configure it the other way around. In hardware there's a dependency in
any case. Also, user space that isn't aware of the embedded data just does
configure the pixel data stream.

-- 
Regards,

Sakari Ailus

Re: [PATCH v6 01/28] media: mc: Add INTERNAL pad flag

[Sakari Ailus]

On Thu, Oct 05, 2023 at 10:22:42AM +0000, Sakari Ailus wrote:
> > Note that "media pipeline" isn't defined either. Should that be added?
> 
> I can add that, too...

Instead I'll remove it altogether: it's not needed here.

-- 
Sakari Ailus

Re: [PATCH v6 01/28] media: mc: Add INTERNAL pad flag

[Sakari Ailus]

Moi,

Thanks for the review.

On Thu, Oct 05, 2023 at 02:04:55PM +0300, Tomi Valkeinen wrote:
> On 03/10/2023 14:52, Sakari Ailus wrote:
> > Internal source pads will be used as routing endpoints in V4L2
> > [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
> 
> It would probably be good to explain here and in the comments/docs, that a
> sink pad is a source pad =). When you say above "internal source pad", it's
> actually MEDIA_PAD_FL_INTERNAL | MEDIA_PAD_FL_SINK. I think this will
> confuse people time and time again, so it's probably good to explain it in
> as many places as possible.

I'll add it here.

> 
> > Also prevent creating links to pads that have been flagged as internal and
> > initialising source pads with internal flag set.
> > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >   Documentation/userspace-api/media/glossary.rst         |  6 ++++++
> >   .../userspace-api/media/mediactl/media-types.rst       |  6 ++++++
> >   drivers/media/mc/mc-entity.c                           | 10 ++++++++--
> >   include/uapi/linux/media.h                             |  1 +
> >   4 files changed, 21 insertions(+), 2 deletions(-)
> > 
> > diff --git a/Documentation/userspace-api/media/glossary.rst b/Documentation/userspace-api/media/glossary.rst
> > index 96a360edbf3b..f7b99a4527c7 100644
> > --- a/Documentation/userspace-api/media/glossary.rst
> > +++ b/Documentation/userspace-api/media/glossary.rst
> > @@ -173,6 +173,12 @@ Glossary
> >   	An integrated circuit that integrates all components of a computer
> >   	or other electronic systems.
> > +_media-glossary-stream:
> > +    Stream
> > +	A distinct flow of data (image data or metadata) over a media pipeline
> > +	from source to sink. A source may be e.g. an image sensor and a sink
> > +	e.g. a memory buffer.
> > +
> >       V4L2 API
> >   	**V4L2 userspace API**
> > diff --git a/Documentation/userspace-api/media/mediactl/media-types.rst b/Documentation/userspace-api/media/mediactl/media-types.rst
> > index 0ffeece1e0c8..28941da27790 100644
> > --- a/Documentation/userspace-api/media/mediactl/media-types.rst
> > +++ b/Documentation/userspace-api/media/mediactl/media-types.rst
> > @@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
> >   .. _MEDIA-PAD-FL-SINK:
> >   .. _MEDIA-PAD-FL-SOURCE:
> >   .. _MEDIA-PAD-FL-MUST-CONNECT:
> > +.. _MEDIA-PAD-FL-INTERNAL:
> >   .. flat-table:: Media pad flags
> >       :header-rows:  0
> > @@ -382,6 +383,11 @@ Types and flags used to represent the media graph elements
> >   	  when this flag isn't set; the absence of the flag doesn't imply
> >   	  there is none.
> > +    *  -  ``MEDIA_PAD_FL_INTERNAL``
> > +       -  The internal flag indicates an internal pad that has no external
> > +	  connections. Such a pad shall not be connected with a link. The
> > +	  internal flag indicates that the :ref:``stream
> > +	  <media-glossary-stream>`` either starts or ends in the entity.
> >   One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
> >   must be set for every pad.
> > diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> > index 543a392f8635..f5f290781021 100644
> > --- a/drivers/media/mc/mc-entity.c
> > +++ b/drivers/media/mc/mc-entity.c
> > @@ -213,7 +213,9 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
> >   		iter->index = i++;
> >   		if (hweight32(iter->flags & (MEDIA_PAD_FL_SINK |
> > -					     MEDIA_PAD_FL_SOURCE)) != 1) {
> > +					     MEDIA_PAD_FL_SOURCE)) != 1 ||
> > +		    (iter->flags & MEDIA_PAD_FL_INTERNAL &&
> > +		     !(iter->flags & MEDIA_PAD_FL_SINK))) {
> >   			ret = -EINVAL;
> >   			break;
> 
> Would it make sense to have this "only internal-sinks supported"-check as a
> separate check, with an error print? It is a valid thing to do, we just want
> to disable it for the time being.

This would be a driver bug. We don't have such complaints on setting both
sink and source flags either, or similar things in the V4L2 framework. I'd
leave it as-is.

Maybe a WARN_ON() as we have in some cases below for creating links?

> 
> >   		}
> > @@ -1075,7 +1077,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
> >   	for (i = 0; i < entity->num_pads; i++) {
> >   		if ((entity->pads[i].flags &
> > -		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
> > +		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
> > +		      MEDIA_PAD_FL_INTERNAL)) != pad_type)
> >   			continue;
> >   		if (entity->pads[i].sig_type == sig_type)
> > @@ -1100,6 +1103,9 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
> >   		return -EINVAL;
> >   	if (WARN_ON(!(sink->pads[sink_pad].flags & MEDIA_PAD_FL_SINK)))
> >   		return -EINVAL;
> > +	if (WARN_ON(source->pads[source_pad].flags & MEDIA_PAD_FL_INTERNAL) ||
> > +	    WARN_ON(source->pads[sink_pad].flags & MEDIA_PAD_FL_INTERNAL))
> > +		return -EINVAL;
> >   	link = media_add_link(&source->links);
> >   	if (link == NULL)
> > diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
> > index 1c80b1d6bbaf..80cfd12a43fc 100644
> > --- a/include/uapi/linux/media.h
> > +++ b/include/uapi/linux/media.h
> > @@ -208,6 +208,7 @@ struct media_entity_desc {
> >   #define MEDIA_PAD_FL_SINK			(1U << 0)
> >   #define MEDIA_PAD_FL_SOURCE			(1U << 1)
> >   #define MEDIA_PAD_FL_MUST_CONNECT		(1U << 2)
> > +#define MEDIA_PAD_FL_INTERNAL			(1U << 3)
> >   struct media_pad_desc {
> >   	__u32 entity;		/* entity ID */
> 

-- 
Sakari Ailus

Re: [PATCH v6 03/28] media: uapi: Document which mbus format fields are valid for metadata

[Sakari Ailus]

On Thu, Oct 05, 2023 at 02:28:20PM +0300, Tomi Valkeinen wrote:
> On 03/10/2023 14:52, Sakari Ailus wrote:
> > Now that metadata mbus formats have been added, it is necessary to define
> > which fields in struct v4l2_mbus_format are applicable to them (not many).
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >   include/uapi/linux/v4l2-mediabus.h | 18 ++++++++++++------
> >   1 file changed, 12 insertions(+), 6 deletions(-)
> > 
> > diff --git a/include/uapi/linux/v4l2-mediabus.h b/include/uapi/linux/v4l2-mediabus.h
> > index 6b07b73473b5..3cadb3b58b85 100644
> > --- a/include/uapi/linux/v4l2-mediabus.h
> > +++ b/include/uapi/linux/v4l2-mediabus.h
> > @@ -19,12 +19,18 @@
> >    * @width:	image width
> >    * @height:	image height
> >    * @code:	data format code (from enum v4l2_mbus_pixelcode)
> > - * @field:	used interlacing type (from enum v4l2_field)
> > - * @colorspace:	colorspace of the data (from enum v4l2_colorspace)
> > - * @ycbcr_enc:	YCbCr encoding of the data (from enum v4l2_ycbcr_encoding)
> > - * @hsv_enc:	HSV encoding of the data (from enum v4l2_hsv_encoding)
> > - * @quantization: quantization of the data (from enum v4l2_quantization)
> > - * @xfer_func:  transfer function of the data (from enum v4l2_xfer_func)
> > + * @field:	used interlacing type (from enum v4l2_field), not applicable
> > + *		to metadata mbus codes
> 
> This one does say "zero", like the others do. Is that on purpose? Annoyingly
> V4L2_FIELD_NONE is 1, not 0, but perhaps zeroing this would still be best?

I'll document this being zero as the others.

> 
> > + * @colorspace:	colorspace of the data (from enum v4l2_colorspace), zero on
> > + *		metadata mbus codes
> > + * @ycbcr_enc:	YCbCr encoding of the data (from enum v4l2_ycbcr_encoding), zero
> > + *		on metadata mbus codes
> > + * @hsv_enc:	HSV encoding of the data (from enum v4l2_hsv_encoding), zero on
> > + *		metadata mbus codes
> > + * @quantization: quantization of the data (from enum v4l2_quantization), zero
> > + *		on metadata mbus codes
> > + * @xfer_func:  transfer function of the data (from enum v4l2_xfer_func), zero
> > + *		on metadata mbus codes
> >    * @flags:	flags (V4L2_MBUS_FRAMEFMT_*)
> >    * @reserved:  reserved bytes that can be later used
> >    */
> 

-- 
Sakari Ailus

Re: [PATCH v6 11/28] media: Documentation: Document S_ROUTING behaviour

[Sakari Ailus]

Moi,

On Thu, Oct 05, 2023 at 03:59:05PM +0300, Tomi Valkeinen wrote:
> On 03/10/2023 15:07, Sakari Ailus wrote:
> > Document S_ROUTING behaviour for different devices.
> > 
> > Generally in devices that produce streams the streams are static and some
> 
> I'm not sure what "static" means here. "Generally in devices that produce
> streams there is a fixed amount of streams..."?

Perhaps this Wordnet definition fits:

     3: showing little if any change; "a static population" [syn:
         {static}, {stable}, {unchanging}]

Similarly, we have static links in MC.

> 
> > can be enabled and disabled, whereas in devices that just transport them
> > or write them to memory, more configurability is allowed. Document this.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >   .../userspace-api/media/v4l/dev-subdev.rst    | 21 +++++++++++++++++++
> >   1 file changed, 21 insertions(+)
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > index fb73a95401c3..83993775237f 100644
> > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > @@ -593,6 +593,27 @@ Any configurations of a stream within a pad, such as format or selections,
> >   are independent of similar configurations on other streams. This is
> >   subject to change in the future.
> > +Device types and routing setup
> > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > +
> > +Different kinds of sub-devices have differing behaviour for route inactivation,
> 
> Would "activation" convey the same, but be a bit clearer?

I'd think so.

> 
> > +depending on the hardware. Devices generating the streams may often allow
> 
> Maybe drop the "often".

Seems fine.

> 
> > +enabling and disabling some of these or the configuration is fixed. If these
> 
> "some of these" -> "some of the routes".

Yes.

> 
> > +routes can be disabled, not declaring these routes (or declaring them without
> 
> Here also, I think "the routes" is more readable than repeating "these
> routes".

Works for me.

> 
> > +``VIDIOC_SUBDEV_STREAM_FL_ACTIVE`` flag set) in ``VIDIOC_SUBDEV_S_ROUTING`` will
> 
> Why is the flag sentence in parenthesis? Aren't both options of the same
> value?

It's a long sentence. This is an alternative, indeed, and I think it's
easier to read this way.

> 
> > +disable the routes while the sub-device driver retain the streams and their
> 
> What does this mean? That even if the user disables a route, the driver must
> keep the configuration that was set earlier related to that route?

Yes. As the routes remain, so does the sub-device state.

Would you instead prefer to reset it to (some) defaults? I think driver
implementation will need some code in that case.

> 
> > +configuration. The ``VIDIOC_SUBDEV_S_ROUTING`` will still return such routes
> > +back to the user in the routes array, with the ``V4L2_SUBDEV_STREAM_FL_ACTIVE``
> > +flag unset.
> 
> So a generating device should always return all its routes with both
> G_ROUTING and S_ROUTING, right? But with disabled routes not having

Correct.

> VIDIOC_SUBDEV_STREAM_FL_ACTIVE. The text doesn't mention G_ROUTING at all.

G_ROUTING is sort of trivial in this sense --- it just returns the routes
to the user, and this is documented in the IOCTL documentation.

> 
> > +Devices transporting the streams almost always have more configurability with
> > +respect to routing. Typically any route between the sub-device's sink and source
> > +pads is possible, and multiple routes (usually up to certain limited number) may
> > +be active simultaneously. For such devices, no routes are created by the driver
> > +and user-created routes are fully replaced when ``VIDIOC_SUBDEV_S_ROUTING`` is
> > +called on the sub-device. Such newly created routes have the device's default
> > +configuration for format and selection rectangles.
> > +
> 
> I think this paragraph is ok. But could this whole section be restructured a
> bit, as the previous paragraph gets quite confusing. Maybe:
> 
> First paragraph to explain the two different kinds of devices, and perhaps a
> mention that a route is considered disabled if either it does not exist in
> the routing table or if VIDIOC_SUBDEV_STREAM_FL_ACTIVE is not set.

If there's no route, the route isn't disabled. Or did you mean the routes
array for S_ROUTING IOCTL?

> 
> Then a paragraph for generating devices, and then a paragraph for
> transporting devices.

I'll see what I can do.

-- 
Terveisin,

Sakari Ailus

Re: [PATCH v6 15/28] media: v4l: subdev: Copy argument back to user also for S_ROUTING

[Sakari Ailus]

Hi Hans,

On Thu, Oct 05, 2023 at 12:37:41PM +0200, Hans Verkuil wrote:
> On 03/10/2023 14:08, Sakari Ailus wrote:
> > As the user needs to know what went wrong for S_ROUTING, copy array
> > arguments back to the user.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  drivers/media/v4l2-core/v4l2-ioctl.c | 3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> > 
> > diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> > index fb453b7d0c91..6921a72566df 100644
> > --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> > +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> > @@ -3419,7 +3419,8 @@ video_usercopy(struct file *file, unsigned int orig_cmd, unsigned long arg,
> >  	 * in case of failure, but it is not defined here as part of the
> >  	 * 'v4l2_ioctls' array, insert an ad-hoc check to address that.
> >  	 */
> > -	if (err < 0 && !always_copy && cmd != VIDIOC_SUBDEV_G_ROUTING)
> > +	if (err < 0 && !always_copy && cmd != VIDIOC_SUBDEV_G_ROUTING &&
> > +	    cmd != VIDIOC_SUBDEV_S_ROUTING)
> >  		goto out;
> >  
> >  	if (has_array_args) {
> 
> This is ugly.
> 
> How about this:
> 
> 	if (cmd == VIDIOC_SUBDEV_G_ROUTING || cmd == VIDIOC_SUBDEV_S_ROUTING)
> 		always_copy = true;
> 
> 	if (err < 0 && !always_copy)
> 		goto out;
> 
> The first 'if' can also be a 'switch', I have no preference.
> 
> This cmd check can also be done earlier in the function, right after
> the call to video_get_user(). It might be a better place.

[GS]_ROUTING are already handled here (plus the related comment is right
above) so I prefer this location, i.e. exact lines above you suggested.
It's indeed cleaner.

-- 
Regards,

Sakari Ailus

Re: [PATCH v6 12/28] media: v4l: subdev: Add helpers for format, crop and compose pointers

[Sakari Ailus]

Moi,

On Thu, Oct 05, 2023 at 04:12:00PM +0300, Tomi Valkeinen wrote:
> On 03/10/2023 15:07, Sakari Ailus wrote:
> > Add a helper for obtaining format, crop and compose pointers. These are
> > convenient for drivers, independently of the driver uses streams or not.
> 
> If we go with these, should we deprecate
> v4l2_subdev_state_get_stream_format() and v4l2_subdev_get_pad_format()?
> 
> Having three different ways to get the fmt seems a bit excessive.
> 
> Should we add 'num_pads' to the state? That would remove the need pass the
> subdevice to these functions.

Good question. This would make it easier to refactor the code later,
drivers would still need to begin calling v4l2_subdev_init_finalize(). But
it's one step forward I think.

I'll add this earlier in the set.

-- 
Terveisin,

Sakari Ailus

Re: [PATCH v6 17/28] media: v4l: subdev: Return routes set using S_ROUTING

[Sakari Ailus]

Hi Hans,

On Thu, Oct 05, 2023 at 01:05:17PM +0200, Hans Verkuil wrote:
> On 03/10/2023 14:08, Sakari Ailus wrote:
> > Return the routes set using S_ROUTING back to the user. Also reflect this
> > in documentation.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  .../userspace-api/media/v4l/vidioc-subdev-g-routing.rst      | 5 +++--
> >  drivers/media/v4l2-core/v4l2-subdev.c                        | 5 ++++-
> >  2 files changed, 7 insertions(+), 3 deletions(-)
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > index 9a9765ddc316..ced53ea5f23c 100644
> > --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > @@ -43,8 +43,9 @@ The routing configuration determines the flows of data inside an entity.
> >  Drivers report their current routing tables using the
> >  ``VIDIOC_SUBDEV_G_ROUTING`` ioctl and application may enable or disable routes
> >  with the ``VIDIOC_SUBDEV_S_ROUTING`` ioctl, by adding or removing routes and
> > -setting or clearing flags of the  ``flags`` field of a
> > -struct :c:type:`v4l2_subdev_route`.
> > +setting or clearing flags of the ``flags`` field of a struct
> > +:c:type:`v4l2_subdev_route`. Similarly to ``VIDIOC_SUBDEV_G_ROUTING``, also
> > +``VIDIOC_SUBDEV_S_ROUTING`` returns the routes back to the user.
> >  
> >  All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is
> >  called. This means that the userspace must reconfigure all streams after calling
> > diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> > index bd1e8205913c..9a34e13dfd96 100644
> > --- a/drivers/media/v4l2-core/v4l2-subdev.c
> > +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> > @@ -935,9 +935,12 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
> >  		krouting.len_routes = routing->len_routes;
> >  		krouting.routes = routes;
> >  
> > -		return v4l2_subdev_call(sd, pad, set_routing, state,
> > +		rval = v4l2_subdev_call(sd, pad, set_routing, state,
> >  					routing->which, &krouting);
> > +		if (rval < 0)
> > +			return rval;
> >  	}
> > +		fallthrough;
> 
> Admittedly, it is a bit hard to see from just this patch, but I think
> the actual code that VIDIOC_SUBDEV_S_ROUTING has to do here is just
> a very few lines from VIDIOC_SUBDEV_G_ROUTING. I think it is better
> to explicitly have them here, then doing a fallthrough.
> 
> Most of what is in VIDIOC_SUBDEV_G_ROUTING are just a bunch of checks
> that are already done in VIDIOC_SUBDEV_S_ROUTING.

Ack, I'll change this for v7.

-- 
Regards,

Sakari Ailus

Re: [PATCH v6 18/28] media: uapi: Allow a larger number of routes than there's room for

[Sakari Ailus]

Hi Hans,

On Thu, Oct 05, 2023 at 01:08:25PM +0200, Hans Verkuil wrote:
> On 03/10/2023 14:08, Sakari Ailus wrote:
> > On VIDIOC_SUBDEV_[GS]_ROUTING, only return as many routes back to the user
> > as there's room. Do not consider it an error if more routes existed.
> > Simply inform the user there are more routes.
> 
> I'm not convinced by this change. Can userspace do anything useful with only
> a partially filled routes array?
> 
> Just silently allowing bad data is asking for problems, IMHO.

The idea is that the driver may return more routes than the user sets but
the user may be interested in just one route, the one it enabled. Should
the user space be required to obtain streams even if they are not of
interest?

-- 
Regards,

Sakari Ailus

Re: [PATCH v6 14/28] media: v4l: subdev: Move G_ROUTING handling below S_ROUTING

[Sakari Ailus]

Moi,

On Mon, Oct 09, 2023 at 01:36:46PM +0300, Tomi Valkeinen wrote:
> On 03/10/2023 15:07, Sakari Ailus wrote:
> > Move G_ROUTING IOCTL handling below that of S_ROUTING. G_ROUTING
> > implementation will soon needed in handling S_ROUTING as well.
> 
> There's probably a "be" missing in the above text. Also, it's a bit unclear
> on why this is needed. You could say it'll allow switch-case fall-through.

I'll drop this patch as it seems it won't be needed after all: [GS]_ROUTING
will remain separate.

-- 
Sakari Ailus

Re: [PATCH v6 18/28] media: uapi: Allow a larger number of routes than there's room for

[Laurent Pinchart]

On Mon, Oct 09, 2023 at 01:56:21PM +0300, Tomi Valkeinen wrote:
> On 03/10/2023 15:08, Sakari Ailus wrote:
> > On VIDIOC_SUBDEV_[GS]_ROUTING, only return as many routes back to the user
> > as there's room. Do not consider it an error if more routes existed.
> > Simply inform the user there are more routes.
> 
> Inform how? And I agree with Hans here. How about return ENOSPC, but the 
> kernel fills in num_routes to tell the userspace how many there actually 
> are?

For VIDIOC_SUBDEV_G_ROUTING I have no objection. For
VIDIOC_SUBDEV_S_ROUTING, however, I would prefer the ioctl to succeed if
the routes can be applied but there's not enough space to return them
all to the application. The application may not have an interest in
getting the applied routes back.

> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >   .../userspace-api/media/v4l/vidioc-subdev-g-routing.rst   | 4 ----
> >   drivers/media/v4l2-core/v4l2-subdev.c                     | 8 ++------
> >   2 files changed, 2 insertions(+), 10 deletions(-)
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > index ced53ea5f23c..99d3c15fd759 100644
> > --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > @@ -145,10 +145,6 @@ On success 0 is returned, on error -1 and the ``errno`` variable is set
> >   appropriately. The generic error codes are described at the
> >   :ref:`Generic Error Codes <gen-errors>` chapter.
> >   
> > -ENOSPC
> > -   The application provided ``num_routes`` is not big enough to contain
> > -   all the available routes the subdevice exposes.
> > -
> >   EINVAL
> >      The sink or source pad identifiers reference a non-existing pad, or reference
> >      pads of different types (ie. the sink_pad identifiers refers to a source pad).
> > diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> > index 9a34e13dfd96..dd48e7e549fb 100644
> > --- a/drivers/media/v4l2-core/v4l2-subdev.c
> > +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> > @@ -956,14 +956,10 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
> >   
> >   		krouting = &state->routing;
> >   
> > -		if (routing->len_routes < krouting->num_routes) {
> > -			routing->num_routes = krouting->num_routes;
> > -			return -ENOSPC;
> > -		}
> > -
> >   		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
> >   		       krouting->routes,
> > -		       krouting->num_routes * sizeof(*krouting->routes));
> > +		       min(krouting->num_routes, routing->len_routes) *
> > +		       sizeof(*krouting->routes));
> >   		routing->num_routes = krouting->num_routes;
> >   
> >   		return 0;

-- 
Regards,

Laurent Pinchart

Re: [PATCH v6 13/28] media: v4l: subdev: Add a function to lock two sub-device states, use it

[Sakari Ailus]

Moi,

On Mon, Oct 09, 2023 at 01:34:00PM +0300, Tomi Valkeinen wrote:
> On 03/10/2023 15:07, Sakari Ailus wrote:
> > Add two new functions, v4l2_subdev_lock_states() and
> > v4l2_subdev_unclock_states(), to acquire and release the state of two
> > sub-devices. They differ from calling v4l2_subdev_{un,}lock_state() so
> > that if the two states share the same lock, the lock is acquired only
> > once.
> > 
> > Also use the new functions in v4l2_subdev_link_validate().
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >   drivers/media/v4l2-core/v4l2-subdev.c | 12 +++-----
> >   include/media/v4l2-subdev.h           | 40 +++++++++++++++++++++++++++
> >   2 files changed, 44 insertions(+), 8 deletions(-)
> > 
> > diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> > index 854f9d4db923..df9a1ae65410 100644
> > --- a/drivers/media/v4l2-core/v4l2-subdev.c
> > +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> > @@ -1377,17 +1377,13 @@ int v4l2_subdev_link_validate(struct media_link *link)
> >   	states_locked = sink_state && source_state;
> > -	if (states_locked) {
> > -		v4l2_subdev_lock_state(sink_state);
> > -		v4l2_subdev_lock_state(source_state);
> > -	}
> > +	if (states_locked)
> > +		v4l2_subdev_lock_states(sink_state, source_state);
> >   	ret = v4l2_subdev_link_validate_locked(link, states_locked);
> > -	if (states_locked) {
> > -		v4l2_subdev_unlock_state(sink_state);
> > -		v4l2_subdev_unlock_state(source_state);
> > -	}
> > +	if (states_locked)
> > +		v4l2_subdev_unlock_states(sink_state, source_state);
> >   	return ret;
> >   }
> > diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> > index 7c34243ffed9..e49e8af2fb52 100644
> > --- a/include/media/v4l2-subdev.h
> > +++ b/include/media/v4l2-subdev.h
> > @@ -1418,6 +1418,46 @@ static inline void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state)
> >   	mutex_unlock(state->lock);
> >   }
> > +/**
> > + * v4l2_subdev_lock_states - Lock two sub-device states
> > + * @state1: One subdevice state
> > + * @state2: The other subdevice state
> > + *
> > + * Locks the state of two sub-devices.
> > + *
> > + * The states must be unlocked with v4l2_subdev_unlock_states() after use.
> > + *
> > + * This differs from calling v4l2_subdev_lock_state() on both states so that if
> > + * the states share the same lock, the lock is acquired only once (so no
> > + * deadlock occurs). Note that it must be ensured the locks must always be
> > + * acquired in the same order.
> 
> Either s/must/will/ or maybe "note that the locks must always be
> acquired...".

How about:

The caller is responsible for ensuring the locks will always be acquired in
the same order.

> 
> Maybe it should be stated that state1 and state2 have to be from subdevices
> that are connected via a media link and something like "the sink subdevice
> must always be state 1, and the source subdevice must always be state2".

That's less generic than the text in the patch. But in practice I presume
this would be the case --- this is intended to be used in link validation.

-- 
Regards,

Sakari Ailus

Re: [PATCH v6 20/28] media: uapi: v4l: subdev: Enable streams API

[Sakari Ailus]

Moi,

On Mon, Oct 09, 2023 at 03:52:27PM +0300, Tomi Valkeinen wrote:
> On 03/10/2023 15:08, Sakari Ailus wrote:
> > Remove v4l2_subdev_enable_streams_api variable that was used to easily
> > enable streams API for development, and conditions that use the variable.
> > 
> > This patch enables the streams API for V4L2 sub-device interface which
> > allows transporting multiple streams on a single MC link.
> 
> I wouldn't go there yet. Just in this series you break the API.

That was perfectly fine as it was done before this patch. ;-)

But more seriously, this patch does not necessarily need to be merged with
the rest of the set. There is also a possibility of merging it early in 6.7
without this patch, as the UAPI will remain disabled.

-- 
Terveisin,

Sakari Ailus

Re: [PATCH v6 01/28] media: mc: Add INTERNAL pad flag

[Sakari Ailus]

Hi Hans,

On Thu, Oct 05, 2023 at 01:16:27PM +0200, Hans Verkuil wrote:
> On 05/10/2023 12:22, Sakari Ailus wrote:
> > Hi Hans,
> > 
> > Thanks for the review!
> > 
> > On Thu, Oct 05, 2023 at 11:52:08AM +0200, Hans Verkuil wrote:
> >> On 03/10/2023 13:52, Sakari Ailus wrote:
> >>> Internal source pads will be used as routing endpoints in V4L2
> >>> [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
> >>>
> >>> Also prevent creating links to pads that have been flagged as internal and
> >>> initialising source pads with internal flag set.
> >>>
> >>> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> >>> ---
> >>>  Documentation/userspace-api/media/glossary.rst         |  6 ++++++
> >>>  .../userspace-api/media/mediactl/media-types.rst       |  6 ++++++
> >>>  drivers/media/mc/mc-entity.c                           | 10 ++++++++--
> >>>  include/uapi/linux/media.h                             |  1 +
> >>>  4 files changed, 21 insertions(+), 2 deletions(-)
> >>>
> >>> diff --git a/Documentation/userspace-api/media/glossary.rst b/Documentation/userspace-api/media/glossary.rst
> >>> index 96a360edbf3b..f7b99a4527c7 100644
> >>> --- a/Documentation/userspace-api/media/glossary.rst
> >>> +++ b/Documentation/userspace-api/media/glossary.rst
> >>> @@ -173,6 +173,12 @@ Glossary
> >>>  	An integrated circuit that integrates all components of a computer
> >>>  	or other electronic systems.
> >>>  
> >>> +_media-glossary-stream:
> >>> +    Stream
> >>> +	A distinct flow of data (image data or metadata) over a media pipeline
> >>> +	from source to sink. A source may be e.g. an image sensor and a sink
> >>> +	e.g. a memory buffer.
> >>
> >> Hmm, I think this is a bit confusing. I think it would be better to replace
> >> "from source to sink" with "from the initial source to the final sink".
> > 
> > Seems fine to me. I'll use that in v7.
> > 
> >>
> >> The original text doesn't make it clear that there can be many hops in
> >> between.
> >>
> >> So also: "A source" -> "An initial source", and "a sink" -> "a final sink".
> >>
> >> Note that "media pipeline" isn't defined either. Should that be added?
> > 
> > I can add that, too...
> > 
> >>
> >> Finally, I think this should be a separate patch as it has nothing to do with
> >> adding the INTERNAL pad flag.
> > 
> > I agree, will split in v7.
> > 
> >>
> >>> +
> >>>      V4L2 API
> >>>  	**V4L2 userspace API**
> >>>  
> >>> diff --git a/Documentation/userspace-api/media/mediactl/media-types.rst b/Documentation/userspace-api/media/mediactl/media-types.rst
> >>> index 0ffeece1e0c8..28941da27790 100644
> >>> --- a/Documentation/userspace-api/media/mediactl/media-types.rst
> >>> +++ b/Documentation/userspace-api/media/mediactl/media-types.rst
> >>> @@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
> >>>  .. _MEDIA-PAD-FL-SINK:
> >>>  .. _MEDIA-PAD-FL-SOURCE:
> >>>  .. _MEDIA-PAD-FL-MUST-CONNECT:
> >>> +.. _MEDIA-PAD-FL-INTERNAL:
> >>>  
> >>>  .. flat-table:: Media pad flags
> >>>      :header-rows:  0
> >>> @@ -382,6 +383,11 @@ Types and flags used to represent the media graph elements
> >>>  	  when this flag isn't set; the absence of the flag doesn't imply
> >>>  	  there is none.
> >>>  
> >>> +    *  -  ``MEDIA_PAD_FL_INTERNAL``
> >>> +       -  The internal flag indicates an internal pad that has no external
> >>> +	  connections. Such a pad shall not be connected with a link. The
> >>> +	  internal flag indicates that the :ref:``stream
> >>> +	  <media-glossary-stream>`` either starts or ends in the entity.
> >>
> >> This suggests that INTERNAL can be used for both sinks and sources, but...
> > 
> > Right. The INTERNAL flag in UAPI shouldn't be limited to sources, but at
> > the same time we don't have use for it in sinks. I'd prefer to leave this
> > open in the future. We could of course say that explicitly.
> 
> I think it is better to mention that it is for streams that start in the
> entity only, but that in the future it might be extended to support streams
> that end in the entity.
> 
> When we add support for that, we need to update the documentation as well,
> at minimum with one or more examples of how that would be used.

I'll use this instead of the original sentence:

	  The internal flag may currently be present only in a source pad only
	  where it indicates that the :ref:``stream <media-glossary-stream>``
	  originates from within the entity.

-- 
Regards,

Sakari Ailus

Re: [PATCH v6 02/28] media: uapi: Add generic serial metadata mbus formats

[Sakari Ailus]

Hi Laurent,

On Fri, Oct 27, 2023 at 05:47:42PM +0300, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.

Thanks for the review!

> 
> On Tue, Oct 03, 2023 at 02:52:30PM +0300, Sakari Ailus wrote:
> > Add generic serial metadata mbus formats. These formats describe data
> > width and packing but not the content itself. The reason for specifying
> > such formats is that the formats as such are fairly device specific but
> > they are still handled by CSI-2 receiver drivers that should not be aware
> > of device specific formats. What makes generic metadata formats possible
> > is that these formats are parsed by software only, after capturing the
> > data to system memory.
> > 
> > Also add a definition for "Data unit" to cover what is essentially a pixel
> > but is not image data.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  .../userspace-api/media/glossary.rst          |   8 +
> >  .../media/v4l/subdev-formats.rst              | 258 ++++++++++++++++++
> >  include/uapi/linux/media-bus-format.h         |   9 +
> >  3 files changed, 275 insertions(+)
> > 
> > diff --git a/Documentation/userspace-api/media/glossary.rst b/Documentation/userspace-api/media/glossary.rst
> > index f7b99a4527c7..65217b8a44cc 100644
> > --- a/Documentation/userspace-api/media/glossary.rst
> > +++ b/Documentation/userspace-api/media/glossary.rst
> > @@ -25,6 +25,14 @@ Glossary
> >  
> >  	See :ref:`cec`.
> >  
> > +.. _media-glossary-data-unit:
> > +
> > +    Data unit
> > +
> > +	Unit of data transported by a bus. On parallel buses, this is called a
> > +	sample while on serial buses the data unit is logical. If the data unit
> > +	is image data, it may also be called a pixel.
> 
> I don't think this is correct. There are parallel formats that transmit
> multiple pixels per sample (e.g. UYYVYY8_0_5X24), or use multiple
> samples to transmit on pixel (e.g. YUYV8_2X8).

The text needs to be adjusted for parallel buses. How about:

	Unit of data transported on a bus. On parallel buses, a data unit
	consists of one or more samples that together form a single logical
	unit of data while on serial buses a data unit is purely logical.
	If a data unit is image data, it may also be called a pixel.

> > diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
> > index a03c543cb072..9ee031397372 100644
> > --- a/include/uapi/linux/media-bus-format.h
> > +++ b/include/uapi/linux/media-bus-format.h
> > @@ -173,4 +173,13 @@
> >   */
> >  #define MEDIA_BUS_FMT_METADATA_FIXED		0x7001
> >  
> > +/* Generic line based metadata formats for serial buses. Next is 0x8008. */
> > +#define MEDIA_BUS_FMT_META_8			0x8001
> > +#define MEDIA_BUS_FMT_META_10			0x8002
> > +#define MEDIA_BUS_FMT_META_12			0x8003
> > +#define MEDIA_BUS_FMT_META_14			0x8004
> > +#define MEDIA_BUS_FMT_META_16			0x8005
> > +#define MEDIA_BUS_FMT_META_20			0x8006
> > +#define MEDIA_BUS_FMT_META_24			0x8007
> 
> One thing I'd like to do at some point is to fix the historical mistake
> we make when adding raw image formats that encode the CFA pattern in the
> format itself. While this isn't on-topic for this series, I will then
> need to add RAW_{8,10,12,...} media bus codes, which will essentially
> describe the same format as the above formats. Could we already unify
> them, and add CSI-2 RAW formats instead of metadata-specific formats
> here ?

Am I right if my understanding of your view is RAW_* formats would only
include raw image formats of various colour components, on different pixel
orders, and non-raw image formats would have similar arrangements?

I'd say still metadata formats should be separate so you can distinguish
the type of the data (non-image data).

Ultimately the question is what we want to specify and where, but there
appear to be at least three things:

- Pixel depth
- Type of format (raw, yuv, metadata...)
- Colour components
- Pixel order for raw formats or order of transmission for non-raw formats

We've used the mbus code to describe all three. At least my hunch is now
that we'd use mbus code and format to denote the two first at least.

-- 
Regards,

Sakari Ailus

Re: [PATCH v6 18/28] media: uapi: Allow a larger number of routes than there's room for

[Sakari Ailus]

Hi Hans,

On Fri, Oct 27, 2023 at 03:13:55PM +0200, Hans Verkuil wrote:
> On 25/10/2023 23:07, Laurent Pinchart wrote:
> > On Mon, Oct 09, 2023 at 01:56:21PM +0300, Tomi Valkeinen wrote:
> >> On 03/10/2023 15:08, Sakari Ailus wrote:
> >>> On VIDIOC_SUBDEV_[GS]_ROUTING, only return as many routes back to the user
> >>> as there's room. Do not consider it an error if more routes existed.
> >>> Simply inform the user there are more routes.
> >>
> >> Inform how? And I agree with Hans here. How about return ENOSPC, but the 
> >> kernel fills in num_routes to tell the userspace how many there actually 
> >> are?
> > 
> > For VIDIOC_SUBDEV_G_ROUTING I have no objection. For
> > VIDIOC_SUBDEV_S_ROUTING, however, I would prefer the ioctl to succeed if
> > the routes can be applied but there's not enough space to return them
> > all to the application. The application may not have an interest in
> > getting the applied routes back.
> 
> For S_ROUTING, do we still want to update num_routes in that case? Even
> though we return 0 since we could actually set the routes.

num_routes should be updated in this case, otherwise the caller won't know
there are more routes. The caller can then decide what to do about it, to
get the routes using G_ROUTING for instance: the length of the routing
table is now known.

> 
> I think it depends a bit on the naming of these fields in v7.

I recall there was a comment on the naming now, possibly from you, but I
can't find the comments at the moment. :-I I'm open to renaming them but I
don't think the current naming is bad either.

> 
> How likely is it that an application would run into this anyway? I suspect a
> typical app will get the routes first, then modify it.

Laurent can probably comment on this from libcamera perspective, but I
presume that most users of routes are somehow specific to the device. I
would expect source sub-devices that produce the streams are typical places
where you may have dependencies between streams, but they do exist
elsewhere (but there are very few cases).

> 
> It's worth giving this a try, but it depends a bit on how easy it is to
> document it. If you need to jump through hoops to try to explain it to an
> end user, then perhaps this is overly complicated.

The documentation is changed in patch "media: v4l: subdev: Add len_routes
field to struct v4l2_subdev_routing", two patches prior to this. The
changes are split across several patches in order to avoid fewer difficult
to review patches.

I'll change the prefix of this patch to "v4l: subdev:" as well.

-- 
Regards,

Sakari Ailus

Re: [PATCH v6 16/28] media: v4l: subdev: Add len_routes field to struct v4l2_subdev_routing

[Sakari Ailus]

Hi Hans,

Thank you for the review. I somehow missed replying to this earlier, before
posting v7.

On Thu, Oct 05, 2023 at 01:00:15PM +0200, Hans Verkuil wrote:
> On 03/10/2023 14:08, Sakari Ailus wrote:
> > The len_routes field is used to tell the size of the routes array in
> > struct v4l2_subdev_routing. This way the number of routes returned from
> > S_ROUTING IOCTL may be larger than the number of routes provided, in case
> > there are more routes returned by the driver.
> > 
> > Note that this changes the (now-disabled) UAPI, users must be updated.
> 
> With "now-disabled" you mean "still disabled", right?
> 
> So:
> 
> "Note that this uAPI is still disabled in the code, so this change can
> safely be done. Anyone who manually patched the code to enable this uAPI
> must update their code."

I can add that.

> 
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  .../media/v4l/vidioc-subdev-g-routing.rst     | 31 ++++++++++++-------
> >  drivers/media/v4l2-core/v4l2-ioctl.c          |  4 +--
> >  drivers/media/v4l2-core/v4l2-subdev.c         |  6 +++-
> >  include/media/v4l2-subdev.h                   |  2 ++
> >  include/uapi/linux/v4l2-subdev.h              |  8 +++--
> >  5 files changed, 34 insertions(+), 17 deletions(-)
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > index 72677a280cd6..9a9765ddc316 100644
> > --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > @@ -46,20 +46,26 @@ with the ``VIDIOC_SUBDEV_S_ROUTING`` ioctl, by adding or removing routes and
> >  setting or clearing flags of the  ``flags`` field of a
> >  struct :c:type:`v4l2_subdev_route`.
> >  
> > -All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called. This
> > -means that the userspace must reconfigure all streams after calling the ioctl
> > -with e.g. ``VIDIOC_SUBDEV_S_FMT``.
> > +All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is
> > +called. This means that the userspace must reconfigure all streams after calling
> > +the ioctl with e.g. ``VIDIOC_SUBDEV_S_FMT``.
> >  
> >  Only subdevices which have both sink and source pads can support routing.
> >  
> > -When inspecting routes through ``VIDIOC_SUBDEV_G_ROUTING`` and the application
> > -provided ``num_routes`` is not big enough to contain all the available routes
> > -the subdevice exposes, drivers return the ENOSPC error code and adjust the
> > -value of the ``num_routes`` field. Application should then reserve enough memory
> > -for all the route entries and call ``VIDIOC_SUBDEV_G_ROUTING`` again.
> > +The ``num_routes`` field is used to denote the number of routes set (set by user
> > +space on ``VIDIOC_SUBDEV_S_ROUTING`` argument) on the routing table as well as
> > +the number of routes returned back from both IOCTLs. The ``len_routes``
> > +signifies the number of routes that can fit into the ``routes`` array. The
> > +userspace shall set ``len_routes`` for both IOCTLs and ``num_routes`` for
> > +``VIDIOC_SUBDEV_S_ROUTING``.
> >  
> > -On a successful ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the
> > -``num_routes`` field to reflect the actual number of routes returned.
> > +On a ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the ``num_routes``
> > +field to reflect the actual number of routes known by the driver. ``num_routes``
> > +larger than ``len_routes`` in both IOCTLs. In this ``len_routes`` were returned
> > +back to the userspace. This is not an error.
> 
> This paragraph is garbled.

Oops. This intended to read:

On a ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the ``num_routes``
field to reflect the actual number of routes known by the driver.
``num_routes`` larger than ``len_routes`` may be returned by both IOCTLs to
indicate there are more routes than fits to the ``routes`` array. In this
case first ``len_routes`` were returned back to the userspace in the
``routes`` array. This is not considered as an error.

> 
> > +
> > +Also ``VIDIOC_SUBDEV_S_ROUTING`` may return more route than the user provided in
> > +``num_routes`` field due to e.g. hardware properties.
> >  
> >  .. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
> >  
> > @@ -74,6 +80,9 @@ On a successful ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the
> >        - ``which``
> >        - Format to modified, from enum
> >          :ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
> > +    * - __u32
> > +      - ``len_routes``
> > +      - The length of the array (as in memory reserved for the array)
> 
> So is this in bytes or in number of route entries?
> 
> I think 'len_routes' is a terribly confusing name.
> 
> How about 'max_num_routes'? Or 'max_route_elems/entries'?
> 
> >      * - struct :c:type:`v4l2_subdev_route`
> >        - ``routes[]``
> >        - Array of struct :c:type:`v4l2_subdev_route` entries
> > @@ -81,7 +90,7 @@ On a successful ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the
> >        - ``num_routes``
> >        - Number of entries of the routes array
> 
> This is now a very confusing field description.
> 
> How about:
> 
> 'max_num_routes': Maximum number of entries that can fit in the routes array
> 'num_routes': Actual number of entries stored in the routes array.
> 
> >      * - __u32
> > -      - ``reserved``\ [5]
> > +      - ``reserved``\ [11]
> >        - Reserved for future extensions. Applications and drivers must set
> >  	the array to zero.
> >  
> > diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> > index 6921a72566df..1e3da9d64958 100644
> > --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> > +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> > @@ -3155,13 +3155,13 @@ static int check_array_args(unsigned int cmd, void *parg, size_t *array_size,
> >  	case VIDIOC_SUBDEV_S_ROUTING: {
> >  		struct v4l2_subdev_routing *routing = parg;
> >  
> > -		if (routing->num_routes > 256)
> > +		if (routing->len_routes > 256)
> >  			return -E2BIG;
> >  
> >  		*user_ptr = u64_to_user_ptr(routing->routes);
> >  		*kernel_ptr = (void **)&routing->routes;
> >  		*array_size = sizeof(struct v4l2_subdev_route)
> > -			    * routing->num_routes;
> > +			    * routing->len_routes;
> >  		ret = 1;
> >  		break;
> >  	}
> > diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> > index 614ff0031831..bd1e8205913c 100644
> > --- a/drivers/media/v4l2-core/v4l2-subdev.c
> > +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> > @@ -903,6 +903,9 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
> >  		if (routing->which != V4L2_SUBDEV_FORMAT_TRY && ro_subdev)
> >  			return -EPERM;
> >  
> > +		if (routing->num_routes > routing->len_routes)
> > +			return -EINVAL;
0> > +
> >  		memset(routing->reserved, 0, sizeof(routing->reserved));
> >  
> >  		for (i = 0; i < routing->num_routes; ++i) {
> > @@ -929,6 +932,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
> >  		}
> >  
> >  		krouting.num_routes = routing->num_routes;
> > +		krouting.len_routes = routing->len_routes;
> >  		krouting.routes = routes;
> >  
> >  		return v4l2_subdev_call(sd, pad, set_routing, state,
> > @@ -949,7 +953,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
> >  
> >  		krouting = &state->routing;
> >  
> > -		if (routing->num_routes < krouting->num_routes) {
> > +		if (routing->len_routes < krouting->num_routes) {
> >  			routing->num_routes = krouting->num_routes;
> >  			return -ENOSPC;
> >  		}
> > diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> > index e49e8af2fb52..baaa81a9497e 100644
> > --- a/include/media/v4l2-subdev.h
> > +++ b/include/media/v4l2-subdev.h
> > @@ -741,12 +741,14 @@ struct v4l2_subdev_stream_configs {
> >  /**
> >   * struct v4l2_subdev_krouting - subdev routing table
> >   *
> > + * @len_routes: length of routes array, in routes
> >   * @num_routes: number of routes
> >   * @routes: &struct v4l2_subdev_route
> >   *
> >   * This structure contains the routing table for a subdev.
> >   */
> >  struct v4l2_subdev_krouting {
> > +	unsigned int len_routes;
> >  	unsigned int num_routes;
> >  	struct v4l2_subdev_route *routes;
> >  };
> > diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
> > index 4a195b68f28f..b57fb89caa9e 100644
> > --- a/include/uapi/linux/v4l2-subdev.h
> > +++ b/include/uapi/linux/v4l2-subdev.h
> > @@ -222,15 +222,17 @@ struct v4l2_subdev_route {
> >   * struct v4l2_subdev_routing - Subdev routing information
> >   *
> >   * @which: configuration type (from enum v4l2_subdev_format_whence)
> > - * @num_routes: the total number of routes in the routes array
> > + * @len_routes: the length of the routes array, in routes
> >   * @routes: pointer to the routes array
> > + * @num_routes: the total number of routes in the routes array

I'll change this to reflect there may be more routes than fits to the
array.

> >   * @reserved: drivers and applications must zero this array
> >   */
> >  struct v4l2_subdev_routing {
> >  	__u32 which;
> > -	__u32 num_routes;
> > +	__u32 len_routes;
> >  	__u64 routes;
> > -	__u32 reserved[6];
> > +	__u32 num_routes;
> > +	__u32 reserved[11];
> >  };
> >  
> >  /*
> 
> I'm going with a NACK for the 'len_routes' name, it's a really, really
> bad name.
> 
> So:
> 
> Nacked-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>

That's bit harsh judgement due to a differing opinion on a field name. I
think len_routes is fine as it's the length of the routes array, not the
number of entries in it but I'm totally fine changing it.

max_num_routes isn't great either as num_routes can effectively be larger
than max_num_routes.

How about "size_routes", for instance? Or "max_elems_routes"?

-- 
Regards,

Sakari Ailus

Re: [PATCH v6 16/28] media: v4l: subdev: Add len_routes field to struct v4l2_subdev_routing

[Hans Verkuil]

On 28/11/2023 14:30, Sakari Ailus wrote:
> Hi Hans,
> 
> Thank you for the review. I somehow missed replying to this earlier, before
> posting v7.
> 
> On Thu, Oct 05, 2023 at 01:00:15PM +0200, Hans Verkuil wrote:
>> On 03/10/2023 14:08, Sakari Ailus wrote:
>>> The len_routes field is used to tell the size of the routes array in
>>> struct v4l2_subdev_routing. This way the number of routes returned from
>>> S_ROUTING IOCTL may be larger than the number of routes provided, in case
>>> there are more routes returned by the driver.
>>>
>>> Note that this changes the (now-disabled) UAPI, users must be updated.
>>
>> With "now-disabled" you mean "still disabled", right?
>>
>> So:
>>
>> "Note that this uAPI is still disabled in the code, so this change can
>> safely be done. Anyone who manually patched the code to enable this uAPI
>> must update their code."
> 
> I can add that.
> 
>>
>>>
>>> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
>>> ---
>>>  .../media/v4l/vidioc-subdev-g-routing.rst     | 31 ++++++++++++-------
>>>  drivers/media/v4l2-core/v4l2-ioctl.c          |  4 +--
>>>  drivers/media/v4l2-core/v4l2-subdev.c         |  6 +++-
>>>  include/media/v4l2-subdev.h                   |  2 ++
>>>  include/uapi/linux/v4l2-subdev.h              |  8 +++--
>>>  5 files changed, 34 insertions(+), 17 deletions(-)
>>>
>>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
>>> index 72677a280cd6..9a9765ddc316 100644
>>> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
>>> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
>>> @@ -46,20 +46,26 @@ with the ``VIDIOC_SUBDEV_S_ROUTING`` ioctl, by adding or removing routes and
>>>  setting or clearing flags of the  ``flags`` field of a
>>>  struct :c:type:`v4l2_subdev_route`.
>>>  
>>> -All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called. This
>>> -means that the userspace must reconfigure all streams after calling the ioctl
>>> -with e.g. ``VIDIOC_SUBDEV_S_FMT``.
>>> +All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is
>>> +called. This means that the userspace must reconfigure all streams after calling
>>> +the ioctl with e.g. ``VIDIOC_SUBDEV_S_FMT``.
>>>  
>>>  Only subdevices which have both sink and source pads can support routing.
>>>  
>>> -When inspecting routes through ``VIDIOC_SUBDEV_G_ROUTING`` and the application
>>> -provided ``num_routes`` is not big enough to contain all the available routes
>>> -the subdevice exposes, drivers return the ENOSPC error code and adjust the
>>> -value of the ``num_routes`` field. Application should then reserve enough memory
>>> -for all the route entries and call ``VIDIOC_SUBDEV_G_ROUTING`` again.
>>> +The ``num_routes`` field is used to denote the number of routes set (set by user
>>> +space on ``VIDIOC_SUBDEV_S_ROUTING`` argument) on the routing table as well as
>>> +the number of routes returned back from both IOCTLs. The ``len_routes``
>>> +signifies the number of routes that can fit into the ``routes`` array. The
>>> +userspace shall set ``len_routes`` for both IOCTLs and ``num_routes`` for
>>> +``VIDIOC_SUBDEV_S_ROUTING``.
>>>  
>>> -On a successful ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the
>>> -``num_routes`` field to reflect the actual number of routes returned.
>>> +On a ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the ``num_routes``
>>> +field to reflect the actual number of routes known by the driver. ``num_routes``
>>> +larger than ``len_routes`` in both IOCTLs. In this ``len_routes`` were returned
>>> +back to the userspace. This is not an error.
>>
>> This paragraph is garbled.
> 
> Oops. This intended to read:
> 
> On a ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the ``num_routes``
> field to reflect the actual number of routes known by the driver.
> ``num_routes`` larger than ``len_routes`` may be returned by both IOCTLs to
> indicate there are more routes than fits to the ``routes`` array. In this
> case first ``len_routes`` were returned back to the userspace in the
> ``routes`` array. This is not considered as an error.
> 
>>
>>> +
>>> +Also ``VIDIOC_SUBDEV_S_ROUTING`` may return more route than the user provided in
>>> +``num_routes`` field due to e.g. hardware properties.
>>>  
>>>  .. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
>>>  
>>> @@ -74,6 +80,9 @@ On a successful ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the
>>>        - ``which``
>>>        - Format to modified, from enum
>>>          :ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
>>> +    * - __u32
>>> +      - ``len_routes``
>>> +      - The length of the array (as in memory reserved for the array)
>>
>> So is this in bytes or in number of route entries?
>>
>> I think 'len_routes' is a terribly confusing name.
>>
>> How about 'max_num_routes'? Or 'max_route_elems/entries'?
>>
>>>      * - struct :c:type:`v4l2_subdev_route`
>>>        - ``routes[]``
>>>        - Array of struct :c:type:`v4l2_subdev_route` entries
>>> @@ -81,7 +90,7 @@ On a successful ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the
>>>        - ``num_routes``
>>>        - Number of entries of the routes array
>>
>> This is now a very confusing field description.
>>
>> How about:
>>
>> 'max_num_routes': Maximum number of entries that can fit in the routes array
>> 'num_routes': Actual number of entries stored in the routes array.
>>
>>>      * - __u32
>>> -      - ``reserved``\ [5]
>>> +      - ``reserved``\ [11]
>>>        - Reserved for future extensions. Applications and drivers must set
>>>  	the array to zero.
>>>  
>>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
>>> index 6921a72566df..1e3da9d64958 100644
>>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
>>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
>>> @@ -3155,13 +3155,13 @@ static int check_array_args(unsigned int cmd, void *parg, size_t *array_size,
>>>  	case VIDIOC_SUBDEV_S_ROUTING: {
>>>  		struct v4l2_subdev_routing *routing = parg;
>>>  
>>> -		if (routing->num_routes > 256)
>>> +		if (routing->len_routes > 256)
>>>  			return -E2BIG;
>>>  
>>>  		*user_ptr = u64_to_user_ptr(routing->routes);
>>>  		*kernel_ptr = (void **)&routing->routes;
>>>  		*array_size = sizeof(struct v4l2_subdev_route)
>>> -			    * routing->num_routes;
>>> +			    * routing->len_routes;
>>>  		ret = 1;
>>>  		break;
>>>  	}
>>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>>> index 614ff0031831..bd1e8205913c 100644
>>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>>> @@ -903,6 +903,9 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>>>  		if (routing->which != V4L2_SUBDEV_FORMAT_TRY && ro_subdev)
>>>  			return -EPERM;
>>>  
>>> +		if (routing->num_routes > routing->len_routes)
>>> +			return -EINVAL;
> 0> > +
>>>  		memset(routing->reserved, 0, sizeof(routing->reserved));
>>>  
>>>  		for (i = 0; i < routing->num_routes; ++i) {
>>> @@ -929,6 +932,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>>>  		}
>>>  
>>>  		krouting.num_routes = routing->num_routes;
>>> +		krouting.len_routes = routing->len_routes;
>>>  		krouting.routes = routes;
>>>  
>>>  		return v4l2_subdev_call(sd, pad, set_routing, state,
>>> @@ -949,7 +953,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>>>  
>>>  		krouting = &state->routing;
>>>  
>>> -		if (routing->num_routes < krouting->num_routes) {
>>> +		if (routing->len_routes < krouting->num_routes) {
>>>  			routing->num_routes = krouting->num_routes;
>>>  			return -ENOSPC;
>>>  		}
>>> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
>>> index e49e8af2fb52..baaa81a9497e 100644
>>> --- a/include/media/v4l2-subdev.h
>>> +++ b/include/media/v4l2-subdev.h
>>> @@ -741,12 +741,14 @@ struct v4l2_subdev_stream_configs {
>>>  /**
>>>   * struct v4l2_subdev_krouting - subdev routing table
>>>   *
>>> + * @len_routes: length of routes array, in routes
>>>   * @num_routes: number of routes
>>>   * @routes: &struct v4l2_subdev_route
>>>   *
>>>   * This structure contains the routing table for a subdev.
>>>   */
>>>  struct v4l2_subdev_krouting {
>>> +	unsigned int len_routes;
>>>  	unsigned int num_routes;
>>>  	struct v4l2_subdev_route *routes;
>>>  };
>>> diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
>>> index 4a195b68f28f..b57fb89caa9e 100644
>>> --- a/include/uapi/linux/v4l2-subdev.h
>>> +++ b/include/uapi/linux/v4l2-subdev.h
>>> @@ -222,15 +222,17 @@ struct v4l2_subdev_route {
>>>   * struct v4l2_subdev_routing - Subdev routing information
>>>   *
>>>   * @which: configuration type (from enum v4l2_subdev_format_whence)
>>> - * @num_routes: the total number of routes in the routes array
>>> + * @len_routes: the length of the routes array, in routes
>>>   * @routes: pointer to the routes array
>>> + * @num_routes: the total number of routes in the routes array
> 
> I'll change this to reflect there may be more routes than fits to the
> array.
> 
>>>   * @reserved: drivers and applications must zero this array
>>>   */
>>>  struct v4l2_subdev_routing {
>>>  	__u32 which;
>>> -	__u32 num_routes;
>>> +	__u32 len_routes;
>>>  	__u64 routes;
>>> -	__u32 reserved[6];
>>> +	__u32 num_routes;
>>> +	__u32 reserved[11];
>>>  };
>>>  
>>>  /*
>>
>> I'm going with a NACK for the 'len_routes' name, it's a really, really
>> bad name.
>>
>> So:
>>
>> Nacked-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
> 
> That's bit harsh judgement due to a differing opinion on a field name. I
> think len_routes is fine as it's the length of the routes array, not the
> number of entries in it but I'm totally fine changing it.
> 
> max_num_routes isn't great either as num_routes can effectively be larger
> than max_num_routes.
> 
> How about "size_routes", for instance? Or "max_elems_routes"?
> 

I think we should keep the meaning of num_routes as exactly that: the
number of routes that can be stored in the routes array. This is consistent
with the struct media_v2_topology use of the num_ fields.

What you want is to be able to tell the driver how many of the routes
in the routes array should be used for setting routes.

How about adding a field "set_routes" instead? It is ignored by G_ROUTING,
and S_ROUTING will set the first 'set_routes' elements in the 'routes'
array. I'm not sure if set_routes should be set to 0 upon return: does
the ordering of the routes change? Or are the first 'set_routes' elements
the same as what the user put in originally?

If set_routes > num_routes, then return -EINVAL. If set_routes is 0, would
that clear all routing?

With a naming scheme like this it can also be used if we ever implement a
S_TOPOLOGY ioctl.

Regards,

	Hans