[PATCH v8 00/38] Generic line based metadata support, internal pads

[PATCH v8 00/38] Generic line based metadata support, internal pads

[Sakari Ailus]

Hi folks,

Here are a few patches to add support generic, line based metadata as well
as internal pads. While the amount of code is not very large, to the
contrary it is quite small actually IMO, I presume what this is about and
why it is being proposed requires some explaining.

Metadata mbus codes and formats have existed for some time in V4L2. They
however have been only used by drivers that produce the data itself and
effectively this metadata has always been statistics of some sort (at
least when it comes to ISPs). What is different here is that we intend to
add support for metadata originating from camera sensors.

Camera sensors produce different kinds of metadata, embedded data (usually
register address--value pairs used to capture the frame, in a more or less
sensor specific format), histograms (in a very sensor specific format),
dark pixels etc. The number of these formats is probably going to be about
as large as image data formats if not larger, as the image data formats
are much better standardised but a smaller subset of them will be
supported by V4L2, at least initially but possibly much more in the long
run.

Having this many device specific formats would be a major problem for all
the other drivers along that pipeline (not to mention the users of those
drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2
receiver drivers that have DMA: the poor driver developer would not only
need to know camera sensor specific formats but to choose the specific
packing of that format suitable for the DMA used by the hardware. It is
unlikely many of these would ever get tested while being present on the
driver API. Also adding new sensors with new embedded data formats would
involve updating all bridge and CSI-2 receiver drivers. I don't expect
this to be a workable approach.

Instead what I'm proposing is to use specific metadata formats on the
sensor devices only, on internal pads (more about those soon) of the
sensors, only visible in the UAPI, and then generic mbus formats along the
pipeline and finally generic V4L2 metadata formats on the DMAs (specific
to bit depth and packing). This would unsnarl the two, defining what data
there is (specific mbus code) and how that is transported and packed
(generic mbus codes and V4L2 formats).

The user space would be required to "know" the path of that data from the
sensor's internal pad to the V4L2 video node. I do not see this as these
devices require at least some knowledge of the pipeline, i.e. hardware at
hand. Separating what the data means and how it is packed may even be
beneficial: it allows separating code that interprets the data (sensor
internal mbus code) from the code that accesses it (packing).

These formats are in practice line based, meaning that there may be
padding at the end of the line, depending on the bus as well as the DMA.
If non-line based formats are needed, it is always possible to set the
"height" field to 1.

The internal (source) pads are an alternative to source routes [1]. The
source routes were not universally liked and I do have to say I like
re-using existing interface concepts (pads and everything you can do with
pads, including access format, selections etc.) wherever it makes sense,
instead of duplicating functionality.

Effectively internal source pads behave mostly just like sink pads, but
they describe a flow of data that originates from a sub-device instead of
arriving to a sub-device. The SUBDEV_S_ROUTING IOCTLs are used to enable
and disable routes from internal source pads to sub-device's source pads.
The subdev format IOCTLs are usable, too, so one can find which subdev
format is available on given internal source pad.

I've also pushed these here and I'll keep updating the branch, I've also
included untested OV2740 patches:

<URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=metadata>

Questions and comments are most welcome.

Preliminary media-ctl and yavta patches can be found here:

<URL:https://git.retiisi.eu/?p=~sailus/yavta.git;a=shortlog;h=refs/heads/metadata>
<URL:https://git.retiisi.eu/?p=~sailus/v4l-utils.git;a=shortlog;h=refs/heads/metadata>

I have used IMX219 as an example on routing in a sensor driver in this
version.

The patches are on my master branch
<URL:https://git.linuxtv.org/sailus/media_tree.git/log/>.

[1] <URL:https://lore.kernel.org/linux-media/20220831141357.1396081-20-tomi.valkeinen@ideasonboard.com/>

since v7:

- Add embedded data support for the ov2740 driver.

- Add three patches on top, to add an IMMUTABLE flag to source streams
  when they cannot be disabled.

- Improved documentation of len_routes and num_routes arguments of
  [GS]_ROUTING.

- Remove one inclusion of twice-included media/v4l2-fwnode.h in
  drivers/media/i2c/ccs/ccs-core.c .

- Add missing forward declaration of ccs_internal_ops in
  drivers/media/i2c/ccs/ccs-core.c .

since v6:

- Improve embedded data UAPI documentation on camera sensors.

- Improve wording of stream glossary entry.

- Improve internal pad flag documentation.

- Fix definition of "data unit" and remove an extra "only" in INTERNAL pad
  flag description (1st patch).

- Use IMX219 driver in examples consistently.

- Remove the CSI-2 to parallel bridge from the example to simplify the
  example.

- Minor rewording of some parts of the routing examples.

- Rebase on unified sub-device state information access functions:
  <URL:https://lore.kernel.org/linux-media/20231027095913.1010187-1-sakari.ailus@linux.intel.com/T/#t>

- In CCS driver, do not maintain current active configuration in driver's
  device context struct (apart from mbus codes). Rely on sub-device state
  locking and clean up the code. (Multiple patches towards the end of the
  set.)

- Arrange the CCS patches early in the set towards the end of the set.

- Move the patch enabling streams API to the end of the set.

- Rework IOCTL argument copying condition for [GS]_ROUTING).

- Handle copying back routes in S_ROUTING, do not rely on G_ROUTING
  IOCTL implementation.

- Rebase on metadata preparation patchset v6:
  <URL:https://lore.kernel.org/linux-media/20231106121805.1266696-1-sakari.ailus@linux.intel.com/T/#t>.

since v5:

- Rebase on new set of preparation patches.

- Switch CCS driver from s_stream to enable_streams/disable_streams. Keep
  streaming state information --- the sensor remains in streaming state if
  any of the streams is enabled.

- Fix setting mbus code on embedded data in get_frame_desc() op in the CCS
  driver.

since v4:

- Add a patch to acquire two sub-device states that may use the same lock.

- Add a patch for CCS driver to remove ccs_get_crop_compose() helper.

- Add a patch for CCS driver moving acquiring and releasing the mutex to
  the s_stream callback.

- Add a patch for CCS driver to rely on sub-device state locking using a
  single driver-provided lock.

- Fixed calculating minimum number of routes in copying the routes
  (thanks, Laurent).

- Moved a label in S_ROUTING handling to make Clang happy (hopefully).

- Fixed setting emb_data_ctrl register for CCS embedded data support.

- Rebase on Laurent's cleanup patches.

- Wrap a few long lines.

- Write in embedded data documentation sensor drivers generally don't
  allow configuring it.

since v3:

- Separate preparation patches from this set.

- Add a definition for "Data unit", a pixel that is not image data and use
  it instead in format documentation.

- Fix more numbered lists in dev-subdev.rst.

- Remove a redundant definition for V4L2_META_FMT_GENERIC_CSI2_2_24 ---
  V4L2_META_FMT_GENERIC_CSI2_12 can be used instead.

- Use "X" instead of "p" to denote padding in format documentation.

- Use IMX219 in examples instead of CCS.

- Document that the generic V4L2 CSI-2 metadata formats use padding
  defined in CSI-2 spec and packing defined in CCS spec.

- Add patches to align [GS]_ROUTING behaviour with V4L2. This means mainly
  returning configured routes as part of S_ROUTING as well. "len_routes"
  field is added to denote the length of the array and having more routes
  than fits in the array is no longer an error. Also added more reserved
  fields.

- Added trivial support for S_ROUTING (via G_ROUTING implementation) for
  use in drivers with static-only routes.

- Added helper functions to obtain mbus format as well as crop and compose
  rectangles that are streams-independent.

- Added a patch to define generic CSI-2 long packet types.

- Removed MEDIA_BUS_FMT_IS_META() macro. It didn't seem useful in the end.

- Use a single CCS embedded data format. The bit depth can be selected
  using the meta stream on the source pad.

- Fix mbus code numbers (there were holes due to removed redundant
  formats).

- Fix generic mbus code documentation (byte was being used instead of
  bit).

- Fix spelling of "length".

- Added a patch to remove v4l2_subdev_enable_streams_api that disables
  streams API. This should be merged once libcamera support for streams
  works nicely.

- Don't use strings in printing frame descriptor flags.

- Warn on string truncation in printing frame descriptor.

since v2:

- Add a better example, with formats.

- Add CCS static data media bus codes.

- Added an example demonstrating the use of internal pads. --- Is the
  level of detail enough for the purpose?

- Improved documentation.

- Added a macro to tell whether a format is a metadata format.
  (Documentation could be added.)

- A small ReST syntax fix in the same section.

- Drop leftovers of a patch checking for the INTERNAL_SOURCE flag.

since v1:

- Make the new pad flag just "INTERNAL", requiring either SINK or SOURCE
  pad flag to accompany it. Removed the union in struct v4l2_subdev_route.

- Add the term "stream" to MC glossary.

- Improved and fixed documentation (according to comments).

- Note these formats are little endian.

- Remove 1X8 from the names of the mbus codes. These formats have generally
  8 bits per pixel.

- Fix mbus code numbering (had holes in RFC).

- Add new metadata fields to debug prints.

- Fix a minor documentation build issue.

Sakari Ailus (38):
  media: mc: Add INTERNAL pad flag
  media: Documentation: Add "stream" into glossary
  media: uapi: Add generic serial metadata mbus formats
  media: uapi: Document which mbus format fields are valid for metadata
  media: uapi: Add generic 8-bit metadata format definitions
  media: v4l: Support line-based metadata capture
  media: Documentation: Additional streams generally don't harm capture
  media: Documentation: Document embedded data guidelines for camera
    sensors
  media: Documentation: v4l: Document internal source pads
  media: Documentation: Document S_ROUTING behaviour
  media: v4l: subdev: Add a function to lock two sub-device states, use
    it
  media: v4l: subdev: Move G_ROUTING handling below S_ROUTING
  media: v4l: subdev: Copy argument back to user also for S_ROUTING
  media: v4l: subdev: Add len_routes field to struct v4l2_subdev_routing
  media: v4l: subdev: Return routes set using S_ROUTING
  media: v4l: subdev: Allow a larger number of routes than there's room
    for
  media: v4l: subdev: Add trivial set_routing support
  media: ccs: No need to set streaming to false in power off
  media: ccs: Use {enable,disable}_streams operations
  media: ccs: Track streaming state
  media: ccs: Move ccs_validate_csi_data_format up
  media: ccs: Support frame descriptors
  media: uapi: ccs: Add media bus code for MIPI CCS embedded data
  media: ccs: Add support for embedded data stream
  media: ccs: Remove ccs_get_crop_compose helper
  media: ccs: Rely on sub-device state locking
  media: ccs: Compute binning configuration from sub-device state
  media: ccs: Compute scaling configuration from sub-device state
  media: ccs: Remove which parameter from ccs_propagate
  media: Documentation: ccs: Document routing
  media: uapi: v4l: subdev: Enable streams API
  media: uapi: Add media bus code for ov2740 embedded data
  media: ov2740: Switch to {enable,disable}_streams
  media: ov2740: Track streaming state
  media: ov2740: Add support for embedded data
  media: v4l: Add V4L2_SUBDEV_ROUTE_FL_IMMUTABLE sub-device routing flag
  media: ccs: Add IMMUTABLE route flag
  media: ov2740: Add IMMUTABLE route flag

 .../media/drivers/camera-sensor.rst           |   29 +
 .../userspace-api/media/drivers/ccs.rst       |   34 +-
 .../userspace-api/media/glossary.rst          |   15 +
 .../media/mediactl/media-types.rst            |    8 +
 .../userspace-api/media/v4l/dev-meta.rst      |   15 +
 .../userspace-api/media/v4l/dev-subdev.rst    |  178 ++-
 .../userspace-api/media/v4l/meta-formats.rst  |    1 +
 .../media/v4l/metafmt-generic.rst             |  304 +++++
 .../media/v4l/subdev-formats.rst              |  354 ++++++
 .../media/v4l/vidioc-enum-fmt.rst             |    7 +
 .../media/v4l/vidioc-subdev-g-routing.rst     |   47 +-
 .../media/videodev2.h.rst.exceptions          |    1 +
 drivers/media/i2c/ccs/ccs-core.c              | 1033 +++++++++++------
 drivers/media/i2c/ccs/ccs-quirk.h             |    7 +
 drivers/media/i2c/ccs/ccs.h                   |   27 +-
 drivers/media/i2c/ov2740.c                    |  234 +++-
 drivers/media/mc/mc-entity.c                  |   10 +-
 drivers/media/v4l2-core/v4l2-ioctl.c          |   25 +-
 drivers/media/v4l2-core/v4l2-subdev.c         |  116 +-
 include/media/v4l2-subdev.h                   |   42 +
 include/uapi/linux/media-bus-format.h         |   13 +
 include/uapi/linux/media.h                    |    1 +
 include/uapi/linux/v4l2-mediabus.h            |   18 +-
 include/uapi/linux/v4l2-subdev.h              |   14 +-
 include/uapi/linux/videodev2.h                |   18 +
 25 files changed, 2062 insertions(+), 489 deletions(-)
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst


base-commit: 10ca30599cad9d577bb3f4003c74ed2000a7693e
-- 
2.39.2

[PATCH v8 01/38] media: mc: Add INTERNAL pad flag

[Sakari Ailus]

Internal source pads will be used as routing endpoints in V4L2
[GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
Internal source pads are pads that have both SINK and INTERNAL flags set.

Also prevent creating links to pads that have been flagged as internal and
initialising SOURCE pads with INTERNAL flag set.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../userspace-api/media/mediactl/media-types.rst       |  8 ++++++++
 drivers/media/mc/mc-entity.c                           | 10 ++++++++--
 include/uapi/linux/media.h                             |  1 +
 3 files changed, 17 insertions(+), 2 deletions(-)

diff --git a/Documentation/userspace-api/media/mediactl/media-types.rst b/Documentation/userspace-api/media/mediactl/media-types.rst
index 6332e8395263..f55ef055bcf8 100644
--- a/Documentation/userspace-api/media/mediactl/media-types.rst
+++ b/Documentation/userspace-api/media/mediactl/media-types.rst
@@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
 .. _MEDIA-PAD-FL-SINK:
 .. _MEDIA-PAD-FL-SOURCE:
 .. _MEDIA-PAD-FL-MUST-CONNECT:
+.. _MEDIA-PAD-FL-INTERNAL:
 
 .. flat-table:: Media pad flags
     :header-rows:  0
@@ -381,6 +382,13 @@ Types and flags used to represent the media graph elements
 	  enabled links even when this flag isn't set; the absence of the flag
 	  doesn't imply there is none.
 
+    *  -  ``MEDIA_PAD_FL_INTERNAL``
+       -  The internal flag indicates an internal pad that has no external
+	  connections. Such a pad shall not be connected with a link.
+
+	  The internal flag may currently be present only in a source pad where
+	  it indicates that the :ref:``stream <media-glossary-stream>``
+	  originates from within the entity.
 
 One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
 must be set for every pad.
diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 0e28b9a7936e..1973e9e1013e 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -213,7 +213,9 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 		iter->index = i++;
 
 		if (hweight32(iter->flags & (MEDIA_PAD_FL_SINK |
-					     MEDIA_PAD_FL_SOURCE)) != 1) {
+					     MEDIA_PAD_FL_SOURCE)) != 1 ||
+		    (iter->flags & MEDIA_PAD_FL_INTERNAL &&
+		     !(iter->flags & MEDIA_PAD_FL_SINK))) {
 			ret = -EINVAL;
 			break;
 		}
@@ -1112,7 +1114,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
 
 	for (i = 0; i < entity->num_pads; i++) {
 		if ((entity->pads[i].flags &
-		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
+		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
+		      MEDIA_PAD_FL_INTERNAL)) != pad_type)
 			continue;
 
 		if (entity->pads[i].sig_type == sig_type)
@@ -1142,6 +1145,9 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
 		return -EINVAL;
 	if (WARN_ON(!(sink->pads[sink_pad].flags & MEDIA_PAD_FL_SINK)))
 		return -EINVAL;
+	if (WARN_ON(source->pads[source_pad].flags & MEDIA_PAD_FL_INTERNAL) ||
+	    WARN_ON(sink->pads[sink_pad].flags & MEDIA_PAD_FL_INTERNAL))
+		return -EINVAL;
 
 	link = media_add_link(&source->links);
 	if (link == NULL)
diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
index 1c80b1d6bbaf..80cfd12a43fc 100644
--- a/include/uapi/linux/media.h
+++ b/include/uapi/linux/media.h
@@ -208,6 +208,7 @@ struct media_entity_desc {
 #define MEDIA_PAD_FL_SINK			(1U << 0)
 #define MEDIA_PAD_FL_SOURCE			(1U << 1)
 #define MEDIA_PAD_FL_MUST_CONNECT		(1U << 2)
+#define MEDIA_PAD_FL_INTERNAL			(1U << 3)
 
 struct media_pad_desc {
 	__u32 entity;		/* entity ID */
-- 
2.39.2

[PATCH v8 02/38] media: Documentation: Add "stream" into glossary

[Sakari Ailus]

Add term "stream" to the glossary of the Media subsystem documentation.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 Documentation/userspace-api/media/glossary.rst | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/Documentation/userspace-api/media/glossary.rst b/Documentation/userspace-api/media/glossary.rst
index 96a360edbf3b..ef0ab601b5bf 100644
--- a/Documentation/userspace-api/media/glossary.rst
+++ b/Documentation/userspace-api/media/glossary.rst
@@ -173,6 +173,12 @@ Glossary
 	An integrated circuit that integrates all components of a computer
 	or other electronic systems.
 
+_media-glossary-stream:
+    Stream
+	A distinct flow of data (image data or metadata) from an initial source
+	to a final sink. The initial source may be e.g. an image sensor and the
+	final sink e.g. a memory buffer.
+
     V4L2 API
 	**V4L2 userspace API**
 
-- 
2.39.2

[PATCH v8 03/38] media: uapi: Add generic serial metadata mbus formats

[Sakari Ailus]

Add generic serial metadata mbus formats. These formats describe data
width and packing but not the content itself. The reason for specifying
such formats is that the formats as such are fairly device specific but
they are still handled by CSI-2 receiver drivers that should not be aware
of device specific formats. What makes generic metadata formats possible
is that these formats are parsed by software only, after capturing the
data to system memory.

Also add a definition for "Data unit" to cover what is essentially a pixel
but is not image data.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../userspace-api/media/glossary.rst          |   9 +
 .../media/v4l/subdev-formats.rst              | 258 ++++++++++++++++++
 include/uapi/linux/media-bus-format.h         |   9 +
 3 files changed, 276 insertions(+)

diff --git a/Documentation/userspace-api/media/glossary.rst b/Documentation/userspace-api/media/glossary.rst
index ef0ab601b5bf..7078141894c5 100644
--- a/Documentation/userspace-api/media/glossary.rst
+++ b/Documentation/userspace-api/media/glossary.rst
@@ -25,6 +25,15 @@ Glossary
 
 	See :ref:`cec`.
 
+.. _media-glossary-data-unit:
+
+    Data unit
+
+	Unit of data transported by a bus. On parallel buses, the data unit
+	consists of one or more related samples while on serial buses the data
+	unit is logical. If the data unit is image data, it may also be called a
+	pixel.
+
     Device Driver
 	Part of the Linux Kernel that implements support for a hardware
 	component.
diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
index eb3cd20b0cf2..cbd475f7cae9 100644
--- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
+++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
@@ -8306,3 +8306,261 @@ The following table lists the existing metadata formats.
 	both sides of the link and the bus format is a fixed
 	metadata format that is not configurable from userspace.
 	Width and height will be set to 0 for this format.
+
+Generic Serial Metadata Formats
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Generic serial metadata formats are used on serial buses where the actual data
+content is more or less device specific but the data is transmitted and received
+by multiple devices that do not process the data in any way, simply writing
+it to system memory for processing in software at the end of the pipeline.
+
+The more specific variant describing the actual data is used on the internal
+source pad of the originating sub-device.
+
+"b" in an array cell signifies a byte of data, followed by the number of the bit
+and finally the bit number in subscript. "X" indicates a padding bit.
+
+.. _media-bus-format-generic-meta:
+
+.. cssclass: longtable
+
+.. flat-table:: Generic Serial Metadata Formats
+    :header-rows:  2
+    :stub-columns: 0
+
+    * - Identifier
+      - Code
+      -
+      - :cspan:`23` Data organization within bus ``Data unit
+	<media-glossary-data-unit>``
+    * -
+      -
+      - Bit
+      - 23
+      - 22
+      - 21
+      - 20
+      - 19
+      - 18
+      - 17
+      - 16
+      - 15
+      - 14
+      - 13
+      - 12
+      - 11
+      - 10
+      - 9
+      - 8
+      - 7
+      - 6
+      - 5
+      - 4
+      - 3
+      - 2
+      - 1
+      - 0
+    * .. _MEDIA-BUS-FMT-META-8:
+
+      - MEDIA_BUS_FMT_META_8
+      - 0x8001
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+    * .. _MEDIA-BUS-FMT-META-10:
+
+      - MEDIA_BUS_FMT_META_10
+      - 0x8002
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - X
+      - X
+    * .. _MEDIA-BUS-FMT-META-12:
+
+      - MEDIA_BUS_FMT_META_12
+      - 0x8003
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - X
+      - X
+      - X
+      - X
+    * .. _MEDIA-BUS-FMT-META-14:
+
+      - MEDIA_BUS_FMT_META_14
+      - 0x8004
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+    * .. _MEDIA-BUS-FMT-META-16:
+
+      - MEDIA_BUS_FMT_META_16
+      - 0x8005
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+    * .. _MEDIA-BUS-FMT-META-20:
+
+      - MEDIA_BUS_FMT_META_20
+      - 0x8006
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+    * .. _MEDIA-BUS-FMT-META-24:
+
+      - MEDIA_BUS_FMT_META_24
+      - 0x8007
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
+      - X
diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
index f05f747e444d..d4c1d991014b 100644
--- a/include/uapi/linux/media-bus-format.h
+++ b/include/uapi/linux/media-bus-format.h
@@ -174,4 +174,13 @@
  */
 #define MEDIA_BUS_FMT_METADATA_FIXED		0x7001
 
+/* Generic line based metadata formats for serial buses. Next is 0x8008. */
+#define MEDIA_BUS_FMT_META_8			0x8001
+#define MEDIA_BUS_FMT_META_10			0x8002
+#define MEDIA_BUS_FMT_META_12			0x8003
+#define MEDIA_BUS_FMT_META_14			0x8004
+#define MEDIA_BUS_FMT_META_16			0x8005
+#define MEDIA_BUS_FMT_META_20			0x8006
+#define MEDIA_BUS_FMT_META_24			0x8007
+
 #endif /* __LINUX_MEDIA_BUS_FORMAT_H */
-- 
2.39.2

[PATCH v8 04/38] media: uapi: Document which mbus format fields are valid for metadata

[Sakari Ailus]

Now that metadata mbus formats have been added, it is necessary to define
which fields in struct v4l2_mbus_format are applicable to them (not many).

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 include/uapi/linux/v4l2-mediabus.h | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)

diff --git a/include/uapi/linux/v4l2-mediabus.h b/include/uapi/linux/v4l2-mediabus.h
index 6b07b73473b5..de1d6161bf62 100644
--- a/include/uapi/linux/v4l2-mediabus.h
+++ b/include/uapi/linux/v4l2-mediabus.h
@@ -19,12 +19,18 @@
  * @width:	image width
  * @height:	image height
  * @code:	data format code (from enum v4l2_mbus_pixelcode)
- * @field:	used interlacing type (from enum v4l2_field)
- * @colorspace:	colorspace of the data (from enum v4l2_colorspace)
- * @ycbcr_enc:	YCbCr encoding of the data (from enum v4l2_ycbcr_encoding)
- * @hsv_enc:	HSV encoding of the data (from enum v4l2_hsv_encoding)
- * @quantization: quantization of the data (from enum v4l2_quantization)
- * @xfer_func:  transfer function of the data (from enum v4l2_xfer_func)
+ * @field:	used interlacing type (from enum v4l2_field), zero on metadata
+ *		mbus codes
+ * @colorspace:	colorspace of the data (from enum v4l2_colorspace), zero on
+ *		metadata mbus codes
+ * @ycbcr_enc:	YCbCr encoding of the data (from enum v4l2_ycbcr_encoding), zero
+ *		on metadata mbus codes
+ * @hsv_enc:	HSV encoding of the data (from enum v4l2_hsv_encoding), zero on
+ *		metadata mbus codes
+ * @quantization: quantization of the data (from enum v4l2_quantization), zero
+ *		on metadata mbus codes
+ * @xfer_func:  transfer function of the data (from enum v4l2_xfer_func), zero
+ *		on metadata mbus codes
  * @flags:	flags (V4L2_MBUS_FRAMEFMT_*)
  * @reserved:  reserved bytes that can be later used
  */
-- 
2.39.2

[PATCH v8 05/38] media: uapi: Add generic 8-bit metadata format definitions

[Sakari Ailus]

Generic 8-bit metadata formats define the in-memory data layout but not
the format of the data itself. The reasoning for having such formats is to
allow CSI-2 receiver drivers to receive and DMA drivers to write the data
to memory without knowing a large number of device specific formats.

These formats may be used only in conjunction of a Media controller
pipeline where the internal pad of the source sub-device defines the
specific format of the data (using an mbus code).

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../userspace-api/media/v4l/dev-subdev.rst    |   2 +
 .../userspace-api/media/v4l/meta-formats.rst  |   1 +
 .../media/v4l/metafmt-generic.rst             | 304 ++++++++++++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c          |   7 +
 include/uapi/linux/videodev2.h                |   8 +
 5 files changed, 322 insertions(+)
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst

diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index 43988516acdd..f375b820ab68 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -506,6 +506,8 @@ source pads.
 
     subdev-formats
 
+.. _subdev-routing:
+
 Streams, multiplexed media pads and internal routing
 ----------------------------------------------------
 
diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
index 0bb61fc5bc00..919f595576b9 100644
--- a/Documentation/userspace-api/media/v4l/meta-formats.rst
+++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
@@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only.
     metafmt-vsp1-hgo
     metafmt-vsp1-hgt
     metafmt-vivid
+    metafmt-generic
diff --git a/Documentation/userspace-api/media/v4l/metafmt-generic.rst b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
new file mode 100644
index 000000000000..2ebab1f895e0
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
@@ -0,0 +1,304 @@
+.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
+
+**************************************************************************************************************************************************************************************************************************************************************************************************************************
+V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O')
+**************************************************************************************************************************************************************************************************************************************************************************************************************************
+
+
+Generic line-based metadata formats
+
+
+Description
+===========
+
+These generic line-based metadata formats define the memory layout of the data
+without defining the format or meaning of the metadata itself. These formats may
+only be used with a Media controller pipeline where the more specific format is
+defined in an :ref:`internal source pad <MEDIA-PAD-FL-INTERNAL>` of the source
+sub-device. See also :ref:`source routes <subdev-routing>`.
+
+.. _v4l2-meta-fmt-generic-8:
+
+V4L2_META_FMT_GENERIC_8
+-----------------------
+
+The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format.
+
+This format is also used on CSI-2 for both 8 bits per ``Data unit
+<media-glossary-data-unit>`` as well as for 16 bits per Data unit when two bytes
+of metadata are packed into one 16-bit Data unit.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_8.**
+Each cell is one byte. "M" denotes a byte of metadata.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+    * - start + 4:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+
+.. _v4l2-meta-fmt-generic-csi2-10:
+
+V4L2_META_FMT_GENERIC_CSI2_10
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
+for each 8 bits of data. Every four bytes of metadata is followed by a single
+byte of padding. The way the data is packed follows the MIPI CSI-2 specification
+and the padding is defined in the MIPI CCS specification.
+
+This format is also used in conjunction with 20 bits per ``Data unit
+<media-glossary-data-unit>`` formats that pack two bytes of metadata into one
+Data unit.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
+Each cell is one byte. "M" denotes a byte of metadata and "X" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - X
+    * - start + 5:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - X
+
+.. _v4l2-meta-fmt-generic-csi2-12:
+
+V4L2_META_FMT_GENERIC_CSI2_12
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_12 contains packed 8-bit generic metadata, 12 bits
+for each 8 bits of data. Every four bytes of metadata is followed by a single
+byte of padding. The way the data is packed follows the MIPI CSI-2 specification
+and the padding is defined in the MIPI CCS specification.
+
+This format is also used in conjunction with 24 bits per ``Data unit
+<media-glossary-data-unit>`` formats that pack two bytes of metadata into one
+Data unit.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_12.**
+Each cell is one byte. "M" denotes a byte of metadata and "X" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - X
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - X
+    * - start + 6:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - X
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - X
+
+.. _v4l2-meta-fmt-generic-csi2-14:
+
+V4L2_META_FMT_GENERIC_CSI2_14
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_14 contains packed 8-bit generic metadata, 14 bits
+for each 8 bits of data. Every four bytes of metadata is followed by three bytes
+of padding. The way the data is packed follows the MIPI CSI-2 specification and
+the padding is defined in the MIPI CCS specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_14.**
+Each cell is one byte. "M" denotes a byte of metadata and "X" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - X
+      - X
+      - X
+    * - start + 7:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - X
+      - X
+      - X
+
+.. _v4l2-meta-fmt-generic-csi2-16:
+
+V4L2_META_FMT_GENERIC_CSI2_16
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_16 contains packed 8-bit generic metadata, 16 bits
+for each 8 bits of data. Every byte of metadata is followed by one byte of
+padding. The way the data is packed follows the MIPI CSI-2 specification and the
+padding is defined in the MIPI CCS specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_16.**
+Each cell is one byte. "M" denotes a byte of metadata and "X" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - X
+      - M\ :sub:`10`
+      - X
+      - M\ :sub:`20`
+      - X
+      - M\ :sub:`30`
+      - X
+    * - start + 8:
+      - M\ :sub:`01`
+      - X
+      - M\ :sub:`11`
+      - X
+      - M\ :sub:`21`
+      - X
+      - M\ :sub:`31`
+      - X
+
+.. _v4l2-meta-fmt-generic-csi2-20:
+
+V4L2_META_FMT_GENERIC_CSI2_20
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_20 contains packed 8-bit generic metadata, 20 bits
+for each 8 bits of data. Every byte of metadata is followed by alternating one
+and two bytes of padding. The way the data is packed follows the MIPI CSI-2
+specification and the padding is defined in the MIPI CCS specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_20.**
+Each cell is one byte. "M" denotes a byte of metadata and "X" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - X
+      - M\ :sub:`10`
+      - X
+      - X
+      - M\ :sub:`20`
+      - X
+      - M\ :sub:`30`
+      - X
+      - X
+    * - start + 10:
+      - M\ :sub:`01`
+      - X
+      - M\ :sub:`11`
+      - X
+      - X
+      - M\ :sub:`21`
+      - X
+      - M\ :sub:`31`
+      - X
+      - X
+
+.. _v4l2-meta-fmt-generic-csi2-24:
+
+V4L2_META_FMT_GENERIC_CSI2_24
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_24 contains packed 8-bit generic metadata, 24 bits
+for each 8 bits of data. Every byte of metadata is followed by two bytes of
+padding. The way the data is packed follows the MIPI CSI-2 specification and the
+padding is defined in the MIPI CCS specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_24.**
+Each cell is one byte. "M" denotes a byte of metadata and "X" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - X
+      - X
+      - M\ :sub:`10`
+      - X
+      - X
+      - M\ :sub:`20`
+      - X
+      - X
+      - M\ :sub:`30`
+      - X
+      - X
+    * - start + 12:
+      - M\ :sub:`01`
+      - X
+      - X
+      - M\ :sub:`11`
+      - X
+      - X
+      - M\ :sub:`21`
+      - X
+      - X
+      - M\ :sub:`31`
+      - X
+      - X
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index 33076af4dfdb..7cb6063f7056 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -1452,6 +1452,13 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
 	case V4L2_PIX_FMT_Y210:		descr = "10-bit YUYV Packed"; break;
 	case V4L2_PIX_FMT_Y212:		descr = "12-bit YUYV Packed"; break;
 	case V4L2_PIX_FMT_Y216:		descr = "16-bit YUYV Packed"; break;
+	case V4L2_META_FMT_GENERIC_8:	descr = "8-bit Generic Metadata"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_10:	descr = "8b Generic Meta, 10b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_12:	descr = "8b Generic Meta, 12b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_14:	descr = "8b Generic Meta, 14b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_16:	descr = "8b Generic Meta, 16b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_20:	descr = "8b Generic Meta, 20b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_24:	descr = "8b Generic Meta, 24b CSI-2"; break;
 
 	default:
 		/* Compressed formats */
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index a8015e5e7fa4..6a4f8ae30186 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -839,6 +839,14 @@ struct v4l2_pix_format {
 #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
 #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
 
+#define V4L2_META_FMT_GENERIC_8		v4l2_fourcc('M', 'E', 'T', '8') /* Generic 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_10	v4l2_fourcc('M', 'C', '1', 'A') /* 10-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_12	v4l2_fourcc('M', 'C', '1', 'C') /* 12-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_14	v4l2_fourcc('M', 'C', '1', 'E') /* 14-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_16	v4l2_fourcc('M', 'C', '1', 'G') /* 16-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_20	v4l2_fourcc('M', 'C', '1', 'K') /* 20-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_24	v4l2_fourcc('M', 'C', '1', 'O') /* 24-bit CSI-2 packed 8-bit metadata */
+
 /* priv field value to indicates that subsequent fields are valid. */
 #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
 
-- 
2.39.2

[PATCH v8 06/38] media: v4l: Support line-based metadata capture

[Sakari Ailus]

many camera sensors, among other devices, transmit embedded data and image
data for each CSI-2 frame. This embedded data typically contains register
configuration of the sensor that has been used to capture the image data
of the same frame.

The embedded data is received by the CSI-2 receiver and has the same
properties as the image data, including that it is line based: it has
width, height and bytesperline (stride).

Add these fields to struct v4l2_meta_format and document them.

Also add V4L2_FMT_FLAG_META_LINE_BASED to tell a given format is
line-based i.e. these fields of struct v4l2_meta_format are valid for it.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../userspace-api/media/v4l/dev-meta.rst          | 15 +++++++++++++++
 .../userspace-api/media/v4l/vidioc-enum-fmt.rst   |  7 +++++++
 .../media/videodev2.h.rst.exceptions              |  1 +
 drivers/media/v4l2-core/v4l2-ioctl.c              |  5 +++--
 include/uapi/linux/videodev2.h                    | 10 ++++++++++
 5 files changed, 36 insertions(+), 2 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/dev-meta.rst b/Documentation/userspace-api/media/v4l/dev-meta.rst
index 0e7e1ee1471a..4dfd79e0a705 100644
--- a/Documentation/userspace-api/media/v4l/dev-meta.rst
+++ b/Documentation/userspace-api/media/v4l/dev-meta.rst
@@ -65,3 +65,18 @@ to 0.
       - ``buffersize``
       - Maximum buffer size in bytes required for data. The value is set by the
         driver.
+    * - __u32
+      - ``width``
+      - Width of a line of metadata in Data units. Valid when
+	:c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set,
+	otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
+    * - __u32
+      - ``height``
+      - Number of rows of metadata. Valid when :c:type`v4l2_fmtdesc` flag
+	``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
+	:c:func:`VIDIOC_ENUM_FMT`.
+    * - __u32
+      - ``bytesperline``
+      - Offset in bytes between the beginning of two consecutive lines. Valid
+	when :c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is
+	set, otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
diff --git a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
index 000c154b0f98..a79abf4428c8 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
@@ -227,6 +227,13 @@ the ``mbus_code`` field is handled differently:
 	The application can ask to configure the quantization of the capture
 	device when calling the :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl with
 	:ref:`V4L2_PIX_FMT_FLAG_SET_CSC <v4l2-pix-fmt-flag-set-csc>` set.
+    * - ``V4L2_FMT_FLAG_META_LINE_BASED``
+      - 0x0200
+      - The metadata format is line-based. In this case the ``width``,
+	``height`` and ``bytesperline`` fields of :c:type:`v4l2_meta_format` are
+	valid. The buffer consists of ``height`` lines, each having ``width``
+	Data units of data and offset (in bytes) between the beginning of each
+	two consecutive lines is ``bytesperline``.
 
 Return Value
 ============
diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b/Documentation/userspace-api/media/videodev2.h.rst.exceptions
index 3e58aac4ef0b..bdc628e8c1d6 100644
--- a/Documentation/userspace-api/media/videodev2.h.rst.exceptions
+++ b/Documentation/userspace-api/media/videodev2.h.rst.exceptions
@@ -215,6 +215,7 @@ replace define V4L2_FMT_FLAG_CSC_XFER_FUNC fmtdesc-flags
 replace define V4L2_FMT_FLAG_CSC_YCBCR_ENC fmtdesc-flags
 replace define V4L2_FMT_FLAG_CSC_HSV_ENC fmtdesc-flags
 replace define V4L2_FMT_FLAG_CSC_QUANTIZATION fmtdesc-flags
+replace define V4L2_FMT_FLAG_META_LINE_BASED fmtdesc-flags
 
 # V4L2 timecode types
 replace define V4L2_TC_TYPE_24FPS timecode-type
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index 7cb6063f7056..d125d23e4e61 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -343,8 +343,9 @@ static void v4l_print_format(const void *arg, bool write_only)
 	case V4L2_BUF_TYPE_META_OUTPUT:
 		meta = &p->fmt.meta;
 		pixelformat = meta->dataformat;
-		pr_cont(", dataformat=%p4cc, buffersize=%u\n",
-			&pixelformat, meta->buffersize);
+		pr_cont(", dataformat=%p4cc, buffersize=%u, width=%u, height=%u, bytesperline=%u\n",
+			&pixelformat, meta->buffersize, meta->width,
+			meta->height, meta->bytesperline);
 		break;
 	}
 }
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 6a4f8ae30186..88fdf190a437 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -877,6 +877,7 @@ struct v4l2_fmtdesc {
 #define V4L2_FMT_FLAG_CSC_YCBCR_ENC		0x0080
 #define V4L2_FMT_FLAG_CSC_HSV_ENC		V4L2_FMT_FLAG_CSC_YCBCR_ENC
 #define V4L2_FMT_FLAG_CSC_QUANTIZATION		0x0100
+#define V4L2_FMT_FLAG_META_LINE_BASED		0x0200
 
 	/* Frame Size and frame rate enumeration */
 /*
@@ -2423,10 +2424,19 @@ struct v4l2_sdr_format {
  * struct v4l2_meta_format - metadata format definition
  * @dataformat:		little endian four character code (fourcc)
  * @buffersize:		maximum size in bytes required for data
+ * @width:		number of data units of data per line (valid for line
+ *			based formats only, see format documentation)
+ * @height:		number of lines of data per buffer (valid for line based
+ *			formats only)
+ * @bytesperline:	offset between the beginnings of two adjacent lines in
+ *			bytes (valid for line based formats only)
  */
 struct v4l2_meta_format {
 	__u32				dataformat;
 	__u32				buffersize;
+	__u32				width;
+	__u32				height;
+	__u32				bytesperline;
 } __attribute__ ((packed));
 
 /**
-- 
2.39.2

[PATCH v8 07/38] media: Documentation: Additional streams generally don't harm capture

[Sakari Ailus]

Having extra streams on the source end of the link that cannot be captured
by the sink sub-device generally are not an issue, at least not on CSI-2
bus. Still document that there may be hardware specific limitations. For
example on parallel bus this might not work on all cases.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 Documentation/userspace-api/media/v4l/dev-subdev.rst | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index f375b820ab68..a387e8a15b8d 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -529,9 +529,9 @@ the its sink pad and allows to route them individually to one of its source
 pads.
 
 Subdevice drivers that support multiplexed streams are compatible with
-non-multiplexed subdev drivers, but, of course, require a routing configuration
-where the link between those two types of drivers contains only a single
-stream.
+non-multiplexed subdev drivers. However, if the driver at the sink end of a link
+does not support streams, then only the stream 0 on source end may be
+captured. There may be additional hardware specific limitations.
 
 Understanding streams
 ^^^^^^^^^^^^^^^^^^^^^
-- 
2.39.2

[PATCH v8 08/38] media: Documentation: Document embedded data guidelines for camera sensors

[Sakari Ailus]

Document how embedded data support should be implemented for camera
sensors, and when and how CCS embedded data format should be referenced.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../media/drivers/camera-sensor.rst           | 29 +++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/Documentation/userspace-api/media/drivers/camera-sensor.rst b/Documentation/userspace-api/media/drivers/camera-sensor.rst
index 919a50e8b9d9..92545538b855 100644
--- a/Documentation/userspace-api/media/drivers/camera-sensor.rst
+++ b/Documentation/userspace-api/media/drivers/camera-sensor.rst
@@ -102,3 +102,32 @@ register programming sequences shall initialize the :ref:`V4L2_CID_HFLIP
 values programmed by the register sequences. The default values of these
 controls shall be 0 (disabled). Especially these controls shall not be inverted,
 independently of the sensor's mounting rotation.
+
+Embedded data
+-------------
+
+Many sensors, mostly raw sensors, support embedded data which is used to convey
+the sensor configuration for the captured frame back to the host. While CSI-2 is
+the most common bus used by such sensors, embedded data can be available on
+other bus types as well.
+
+Embedded data support shall use an internal source pad (a pad that has
+``MEDIA_PAD_FL_SINK <MEDIA-PAD-FL-SINK>`` and ``MEDIA_PAD_FL_INTERNAL
+<MEDIA-PAD-FL-INTERNAL>`` flags set) and route to the external pad. If embedded
+data output can be disabled in hardware, it should be possible to disable the
+embedded data route via ``VIDIOC_SUBDEV_S_ROUTING`` IOCTL.
+
+In general, changing the embedded data format from the driver-configured values
+is not supported. The height of the metadata is hardware specific and the width
+is that (or less of that) of the image width, as configured on the pixel data
+stream.
+
+CCS and non-CCS embedded data
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Embedded data which is compliant with CCS definitions shall use ``CCS embedded
+data format <MEDIA-BUS-FMT-CCS-EMBEDDED>``. Device specific embedded data which
+is compliant up to MIPI CCS embedded data levels 1 or 2 only shall refer to CCS
+embedded data formats and document the level of conformance. The rest of the
+device specific embedded data format shall be documented in the context of the
+data format itself.
-- 
2.39.2

[PATCH v8 09/38] media: Documentation: v4l: Document internal source pads

[Sakari Ailus]

Document internal source pads, pads that have both SINK and INTERNAL flags
set. Use the IMX219 camera sensor as an example.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../userspace-api/media/v4l/dev-subdev.rst    | 145 ++++++++++++++++++
 1 file changed, 145 insertions(+)

diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index a387e8a15b8d..1808f40f63e3 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -553,6 +553,27 @@ A stream at a specific point in the media pipeline is identified by the
 sub-device and a (pad, stream) pair. For sub-devices that do not support
 multiplexed streams the 'stream' field is always 0.
 
+.. _v4l2-subdev-internal-source-pads:
+
+Internal source pads and routing
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Cases where a single sub-device source pad is traversed by multiple streams, one
+or more of which originate from within the sub-device itself, are special as
+there is no external sink pad for such routes. In those cases, the sources of
+the internally generated streams are represented by internal source pads, which
+are sink pads that have the :ref:`MEDIA_PAD_FL_INTERNAL <MEDIA-PAD-FL-INTERNAL>`
+pad flag set.
+
+Internal pads have all the properties of an external pad, including formats and
+selections. The format in this case is the source format of the stream. An
+internal pad always has a single stream only (0).
+
+Routes from an internal source pad to an external source pad are typically not
+modifiable but they can be activated and deactivated using the
+:ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE <v4l2-subdev-routing-flags>` flag, depending
+on driver capabilities.
+
 Interaction between routes, streams, formats and selections
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -668,3 +689,127 @@ To configure this pipeline, the userspace must take the following steps:
    the configurations along the stream towards the receiver, using
    :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each
    stream endpoint in each sub-device.
+
+Internal pads setup example
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A simple example of a multiplexed stream setup might be as follows:
+
+- An IMX219 camera sensor source sub-device, with one sink pad (0), one source pad
+  (1), an internal sink pad (2) that represents the source of embedded
+  data. There are two routes, one from the sink pad to the source, and another
+  from the internal sink pad to the source pad. Both streams are always active,
+  i.e. there is no need to separately enable the embedded data stream. The
+  sensor uses the CSI-2 bus.
+
+- A CSI-2 receiver in the SoC (Receiver). The receiver has a single sink pad
+  (pad 0), connected to the bridge, and two source pads (pads 1-2), going to the
+  DMA engine. The receiver demultiplexes the incoming streams to the source
+  pads.
+
+- DMA Engines in the SoC (DMA Engine), one for each stream. Each DMA engine is
+  connected to a single source pad in the receiver.
+
+The sensor, the bridge and the receiver are modeled as V4L2 sub-devices,
+exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are
+modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
+
+To configure this pipeline, the userspace must take the following steps:
+
+1) Set up media links between entities: connect the sensors to the bridge,
+   bridge to the receiver, and the receiver to the DMA engines. This step does
+   not differ from normal non-multiplexed media controller setup.
+
+2) Configure routing
+
+.. flat-table:: Camera sensor. There are no configurable routes.
+    :header-rows: 1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0
+      - 1/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from the sink pad
+    * - 2/0
+      - 1/1
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from the internal sink pad
+
+.. flat-table:: Receiver routing table. Typically both routes need to be
+		explicitly set.
+    :header-rows:  1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0
+      - 1/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from camera sensor
+    * - 0/1
+      - 2/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from camera sensor
+
+The options available in sensor's routing configuration are dictated by hardware
+capabilities: typically camera sensors always produce an image data stream while
+it may be possible to enable and disable the embedded data stream.
+
+3) Configure formats and selections
+
+   This example assumes that the formats are propagated from sink pad to the
+   source pad as-is. The tables contain fields of both struct v4l2_subdev_format
+   and struct v4l2_mbus_framefmt.
+
+.. flat-table:: Formats set on the sub-devices. Bold values are set, others are
+                static or propagated. The order is aligned with configured
+                routes.
+    :header-rows: 1
+    :fill-cells:
+
+    * - Sub-device
+      - Pad/Stream
+      - Width
+      - Height
+      - Code
+    * - :rspan:`3` Camera sensor sub-device (IMX219)
+      - 1/0
+      - 3296
+      - 2480
+      - MEDIA_BUS_FMT_SRGGB10
+    * - 0/0
+      - **3296**
+      - **2480**
+      - **MEDIA_BUS_FMT_SRGGB10**
+    * - 2/0
+      - 3296
+      - 2
+      - MEDIA_BUS_FMT_IMX219_EMBEDDED
+    * - 1/1
+      - 3296
+      - 2
+      - MEDIA_BUS_FMT_META_10
+    * - :rspan:`3` Receiver
+      - 0/0
+      - **3296**
+      - **2480**
+      - **MEDIA_BUS_FMT_SRGGB10**
+    * - 1/0
+      - 3296
+      - 2480
+      - MEDIA_BUS_FMT_SRGGB10
+    * - 0/1
+      - **3296**
+      - **2**
+      - **MEDIA_BUS_FMT_META_10**
+    * - 2/0
+      - 3296
+      - 2
+      - MEDIA_BUS_FMT_META_10
+
+The embedded data format does not need to be configured as the format is
+dictated by the pixel data format in this case.
-- 
2.39.2

[PATCH v8 10/38] media: Documentation: Document S_ROUTING behaviour

[Sakari Ailus]

Document S_ROUTING behaviour for different devices.

Generally in devices that produce streams the streams are static and some
can be enabled and disabled, whereas in devices that just transport them
or write them to memory, more configurability is allowed. Document this.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../userspace-api/media/v4l/dev-subdev.rst    | 24 +++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index 1808f40f63e3..08495cc6f4a6 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -593,6 +593,30 @@ Any configurations of a stream within a pad, such as format or selections,
 are independent of similar configurations on other streams. This is
 subject to change in the future.
 
+Device types and routing setup
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Different kinds of sub-devices have differing behaviour for route activation,
+depending on the hardware. In all cases, however, only routes that have the
+``V4L2_SUBDEV_STREAM_FL_ACTIVE`` flag set are active.
+
+Devices generating the streams may allow enabling and disabling some of the
+routes or the configuration is fixed. If the routes can be disabled, not
+declaring the routes (or declaring them without
+``VIDIOC_SUBDEV_STREAM_FL_ACTIVE`` flag set) in ``VIDIOC_SUBDEV_S_ROUTING`` will
+disable the routes while the sub-device driver retains the streams and their
+configuration. The ``VIDIOC_SUBDEV_S_ROUTING`` will still return such routes
+back to the user in the routes array, with the ``V4L2_SUBDEV_STREAM_FL_ACTIVE``
+flag unset.
+
+Devices transporting the streams almost always have more configurability with
+respect to routing. Typically any route between the sub-device's sink and source
+pads is possible, and multiple routes (usually up to certain limited number) may
+be active simultaneously. For such devices, no routes are created by the driver
+and user-created routes are fully replaced when ``VIDIOC_SUBDEV_S_ROUTING`` is
+called on the sub-device. Such newly created routes have the device's default
+configuration for format and selection rectangles.
+
 Configuring streams
 ^^^^^^^^^^^^^^^^^^^
 
-- 
2.39.2

[PATCH v8 11/38] media: v4l: subdev: Add a function to lock two sub-device states, use it

[Sakari Ailus]

Add two new functions, v4l2_subdev_lock_states() and
v4l2_subdev_unclock_states(), to acquire and release the state of two
sub-devices. They differ from calling v4l2_subdev_{un,}lock_state() so
that if the two states share the same lock, the lock is acquired only
once.

Also use the new functions in v4l2_subdev_link_validate().

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 12 +++-----
 include/media/v4l2-subdev.h           | 40 +++++++++++++++++++++++++++
 2 files changed, 44 insertions(+), 8 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 4c6198c48dd6..6c34df19ea2f 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1400,17 +1400,13 @@ int v4l2_subdev_link_validate(struct media_link *link)
 
 	states_locked = sink_state && source_state;
 
-	if (states_locked) {
-		v4l2_subdev_lock_state(sink_state);
-		v4l2_subdev_lock_state(source_state);
-	}
+	if (states_locked)
+		v4l2_subdev_lock_states(sink_state, source_state);
 
 	ret = v4l2_subdev_link_validate_locked(link, states_locked);
 
-	if (states_locked) {
-		v4l2_subdev_unlock_state(sink_state);
-		v4l2_subdev_unlock_state(source_state);
-	}
+	if (states_locked)
+		v4l2_subdev_unlock_states(sink_state, source_state);
 
 	return ret;
 }
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index a9e6b8146279..9cce48365975 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1724,6 +1724,46 @@ static inline void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state)
 	mutex_unlock(state->lock);
 }
 
+/**
+ * v4l2_subdev_lock_states - Lock two sub-device states
+ * @state1: One subdevice state
+ * @state2: The other subdevice state
+ *
+ * Locks the state of two sub-devices.
+ *
+ * The states must be unlocked with v4l2_subdev_unlock_states() after use.
+ *
+ * This differs from calling v4l2_subdev_lock_state() on both states so that if
+ * the states share the same lock, the lock is acquired only once (so no
+ * deadlock occurs). The caller is responsible for ensuring the locks will
+ * always be acquired in the same order.
+ */
+static inline void v4l2_subdev_lock_states(struct v4l2_subdev_state *state1,
+					   struct v4l2_subdev_state *state2)
+{
+	mutex_lock(state1->lock);
+	if (state1->lock != state2->lock)
+		mutex_lock(state2->lock);
+}
+
+/**
+ * v4l2_subdev_unlock_states() - Unlock two sub-device states
+ * @state1: One subdevice state
+ * @state2: The other subdevice state
+ *
+ * Unlocks the state of two sub-devices.
+ *
+ * This differs from calling v4l2_subdev_unlock_state() on both states so that
+ * if the states share the same lock, the lock is released only once.
+ */
+static inline void v4l2_subdev_unlock_states(struct v4l2_subdev_state *state1,
+					     struct v4l2_subdev_state *state2)
+{
+	mutex_unlock(state1->lock);
+	if (state1->lock != state2->lock)
+		mutex_unlock(state2->lock);
+}
+
 /**
  * v4l2_subdev_get_unlocked_active_state() - Checks that the active subdev state
  *					     is unlocked and returns it
-- 
2.39.2

[PATCH v8 12/38] media: v4l: subdev: Move G_ROUTING handling below S_ROUTING

[Sakari Ailus]

Move G_ROUTING IOCTL handling below that of S_ROUTING. G_ROUTING
implementation will soon needed in handling S_ROUTING as well.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 54 +++++++++++++--------------
 1 file changed, 27 insertions(+), 27 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 6c34df19ea2f..ec8de3f872b9 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -909,33 +909,6 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 	case VIDIOC_SUBDEV_QUERYSTD:
 		return v4l2_subdev_call(sd, video, querystd, arg);
 
-	case VIDIOC_SUBDEV_G_ROUTING: {
-		struct v4l2_subdev_routing *routing = arg;
-		struct v4l2_subdev_krouting *krouting;
-
-		if (!v4l2_subdev_enable_streams_api)
-			return -ENOIOCTLCMD;
-
-		if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS))
-			return -ENOIOCTLCMD;
-
-		memset(routing->reserved, 0, sizeof(routing->reserved));
-
-		krouting = &state->routing;
-
-		if (routing->num_routes < krouting->num_routes) {
-			routing->num_routes = krouting->num_routes;
-			return -ENOSPC;
-		}
-
-		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
-		       krouting->routes,
-		       krouting->num_routes * sizeof(*krouting->routes));
-		routing->num_routes = krouting->num_routes;
-
-		return 0;
-	}
-
 	case VIDIOC_SUBDEV_S_ROUTING: {
 		struct v4l2_subdev_routing *routing = arg;
 		struct v4l2_subdev_route *routes =
@@ -984,6 +957,33 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 					routing->which, &krouting);
 	}
 
+	case VIDIOC_SUBDEV_G_ROUTING: {
+		struct v4l2_subdev_routing *routing = arg;
+		struct v4l2_subdev_krouting *krouting;
+
+		if (!v4l2_subdev_enable_streams_api)
+			return -ENOIOCTLCMD;
+
+		if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS))
+			return -ENOIOCTLCMD;
+
+		memset(routing->reserved, 0, sizeof(routing->reserved));
+
+		krouting = &state->routing;
+
+		if (routing->num_routes < krouting->num_routes) {
+			routing->num_routes = krouting->num_routes;
+			return -ENOSPC;
+		}
+
+		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
+		       krouting->routes,
+		       krouting->num_routes * sizeof(*krouting->routes));
+		routing->num_routes = krouting->num_routes;
+
+		return 0;
+	}
+
 	case VIDIOC_SUBDEV_G_CLIENT_CAP: {
 		struct v4l2_subdev_client_capability *client_cap = arg;
 
-- 
2.39.2

[PATCH v8 13/38] media: v4l: subdev: Copy argument back to user also for S_ROUTING

[Sakari Ailus]

As the user needs to know what went wrong for S_ROUTING, copy array
arguments back to the user.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/v4l2-core/v4l2-ioctl.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index d125d23e4e61..95bd56145d38 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -3415,11 +3415,14 @@ video_usercopy(struct file *file, unsigned int orig_cmd, unsigned long arg,
 	 * FIXME: subdev IOCTLS are partially handled here and partially in
 	 * v4l2-subdev.c and the 'always_copy' flag can only be set for IOCTLS
 	 * defined here as part of the 'v4l2_ioctls' array. As
-	 * VIDIOC_SUBDEV_G_ROUTING needs to return results to applications even
-	 * in case of failure, but it is not defined here as part of the
+	 * VIDIOC_SUBDEV_[GS]_ROUTING needs to return results to applications
+	 * even in case of failure, but it is not defined here as part of the
 	 * 'v4l2_ioctls' array, insert an ad-hoc check to address that.
 	 */
-	if (err < 0 && !always_copy && cmd != VIDIOC_SUBDEV_G_ROUTING)
+	if (cmd == VIDIOC_SUBDEV_G_ROUTING || cmd == VIDIOC_SUBDEV_S_ROUTING)
+		always_copy = true;
+
+	if (err < 0 && !always_copy)
 		goto out;
 
 	if (has_array_args) {
-- 
2.39.2

[PATCH v8 14/38] media: v4l: subdev: Add len_routes field to struct v4l2_subdev_routing

[Sakari Ailus]

The len_routes field is used to tell the size of the routes array in
struct v4l2_subdev_routing. This way the number of routes returned from
S_ROUTING IOCTL may be larger than the number of routes provided, in case
there are more routes returned by the driver.

Note that this uAPI is still disabled in the code, so this change can
safely be done. Anyone who manually patched the code to enable this uAPI
must update their code.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../media/v4l/vidioc-subdev-g-routing.rst     | 33 ++++++++++++-------
 drivers/media/v4l2-core/v4l2-ioctl.c          |  4 +--
 drivers/media/v4l2-core/v4l2-subdev.c         |  6 +++-
 include/media/v4l2-subdev.h                   |  2 ++
 include/uapi/linux/v4l2-subdev.h              |  9 +++--
 5 files changed, 37 insertions(+), 17 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
index 26b5004bfe6d..8f9aa83275c5 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
@@ -46,20 +46,28 @@ with the ``VIDIOC_SUBDEV_S_ROUTING`` ioctl, by adding or removing routes and
 setting or clearing flags of the  ``flags`` field of a
 struct :c:type:`v4l2_subdev_route`.
 
-All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called. This
-means that the userspace must reconfigure all streams after calling the ioctl
-with e.g. ``VIDIOC_SUBDEV_S_FMT``.
+All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is
+called. This means that the userspace must reconfigure all streams after calling
+the ioctl with e.g. ``VIDIOC_SUBDEV_S_FMT``.
 
 Only subdevices which have both sink and source pads can support routing.
 
-When inspecting routes through ``VIDIOC_SUBDEV_G_ROUTING`` and the application
-provided ``num_routes`` is not big enough to contain all the available routes
-the subdevice exposes, drivers return the ENOSPC error code and adjust the
-value of the ``num_routes`` field. Application should then reserve enough memory
-for all the route entries and call ``VIDIOC_SUBDEV_G_ROUTING`` again.
+The ``num_routes`` field is used to denote the number of routes set (set by user
+space on ``VIDIOC_SUBDEV_S_ROUTING`` argument) on the routing table as well as
+the number of routes returned back from both IOCTLs. ``len_routes`` signifies
+the number of routes that can fit into the ``routes`` array. The userspace shall
+set ``len_routes`` for both IOCTLs and ``num_routes`` for
+``VIDIOC_SUBDEV_S_ROUTING``.
 
-On a successful ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the
-``num_routes`` field to reflect the actual number of routes returned.
+On a ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the ``num_routes``
+field to reflect the actual number of routes known by the driver.
+``num_routes`` larger than ``len_routes`` may be returned by both IOCTLs to
+indicate there are more routes than fits to the ``routes`` array. In this
+case first ``len_routes`` were returned back to the userspace in the
+``routes`` array. This is not considered as an error.
+
+Also ``VIDIOC_SUBDEV_S_ROUTING`` may return more route than the user provided in
+``num_routes`` field due to e.g. hardware properties.
 
 .. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
 
@@ -74,6 +82,9 @@ On a successful ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the
       - ``which``
       - Routing table to be accessed, from enum
         :ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
+    * - __u32
+      - ``len_routes``
+      - The length of the array (as in memory reserved for the array)
     * - struct :c:type:`v4l2_subdev_route`
       - ``routes[]``
       - Array of struct :c:type:`v4l2_subdev_route` entries
@@ -81,7 +92,7 @@ On a successful ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the
       - ``num_routes``
       - Number of entries of the routes array
     * - __u32
-      - ``reserved``\ [5]
+      - ``reserved``\ [11]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index 95bd56145d38..018e7efc21ca 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -3155,13 +3155,13 @@ static int check_array_args(unsigned int cmd, void *parg, size_t *array_size,
 	case VIDIOC_SUBDEV_S_ROUTING: {
 		struct v4l2_subdev_routing *routing = parg;
 
-		if (routing->num_routes > 256)
+		if (routing->len_routes > 256)
 			return -E2BIG;
 
 		*user_ptr = u64_to_user_ptr(routing->routes);
 		*kernel_ptr = (void **)&routing->routes;
 		*array_size = sizeof(struct v4l2_subdev_route)
-			    * routing->num_routes;
+			    * routing->len_routes;
 		ret = 1;
 		break;
 	}
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index ec8de3f872b9..486a5c3016de 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -925,6 +925,9 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 		if (routing->which != V4L2_SUBDEV_FORMAT_TRY && ro_subdev)
 			return -EPERM;
 
+		if (routing->num_routes > routing->len_routes)
+			return -EINVAL;
+
 		memset(routing->reserved, 0, sizeof(routing->reserved));
 
 		for (i = 0; i < routing->num_routes; ++i) {
@@ -951,6 +954,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 		}
 
 		krouting.num_routes = routing->num_routes;
+		krouting.len_routes = routing->len_routes;
 		krouting.routes = routes;
 
 		return v4l2_subdev_call(sd, pad, set_routing, state,
@@ -971,7 +975,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 
 		krouting = &state->routing;
 
-		if (routing->num_routes < krouting->num_routes) {
+		if (routing->len_routes < krouting->num_routes) {
 			routing->num_routes = krouting->num_routes;
 			return -ENOSPC;
 		}
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 9cce48365975..1df6b963a1c9 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -728,12 +728,14 @@ struct v4l2_subdev_stream_configs {
 /**
  * struct v4l2_subdev_krouting - subdev routing table
  *
+ * @len_routes: length of routes array, in routes
  * @num_routes: number of routes
  * @routes: &struct v4l2_subdev_route
  *
  * This structure contains the routing table for a subdev.
  */
 struct v4l2_subdev_krouting {
+	unsigned int len_routes;
 	unsigned int num_routes;
 	struct v4l2_subdev_route *routes;
 };
diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
index 7048c51581c6..ca543982460c 100644
--- a/include/uapi/linux/v4l2-subdev.h
+++ b/include/uapi/linux/v4l2-subdev.h
@@ -224,15 +224,18 @@ struct v4l2_subdev_route {
  * struct v4l2_subdev_routing - Subdev routing information
  *
  * @which: configuration type (from enum v4l2_subdev_format_whence)
- * @num_routes: the total number of routes in the routes array
+ * @len_routes: the length of the routes array, in routes
  * @routes: pointer to the routes array
+ * @num_routes: the total number of routes, possibly more than fits in the
+ *		routes array
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_routing {
 	__u32 which;
-	__u32 num_routes;
+	__u32 len_routes;
 	__u64 routes;
-	__u32 reserved[6];
+	__u32 num_routes;
+	__u32 reserved[11];
 };
 
 /*
-- 
2.39.2

[PATCH v8 15/38] media: v4l: subdev: Return routes set using S_ROUTING

[Sakari Ailus]

Return the routes set using S_ROUTING back to the user. Also reflect this
in documentation.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../media/v4l/vidioc-subdev-g-routing.rst            |  5 +++--
 drivers/media/v4l2-core/v4l2-subdev.c                | 12 +++++++++++-
 2 files changed, 14 insertions(+), 3 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
index 8f9aa83275c5..6eb6a59570dc 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
@@ -43,8 +43,9 @@ The routing configuration determines the flows of data inside an entity.
 Drivers report their current routing tables using the
 ``VIDIOC_SUBDEV_G_ROUTING`` ioctl and application may enable or disable routes
 with the ``VIDIOC_SUBDEV_S_ROUTING`` ioctl, by adding or removing routes and
-setting or clearing flags of the  ``flags`` field of a
-struct :c:type:`v4l2_subdev_route`.
+setting or clearing flags of the ``flags`` field of a struct
+:c:type:`v4l2_subdev_route`. Similarly to ``VIDIOC_SUBDEV_G_ROUTING``, also
+``VIDIOC_SUBDEV_S_ROUTING`` returns the routes back to the user.
 
 All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is
 called. This means that the userspace must reconfigure all streams after calling
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 486a5c3016de..a357ce318192 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -957,8 +957,18 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 		krouting.len_routes = routing->len_routes;
 		krouting.routes = routes;
 
-		return v4l2_subdev_call(sd, pad, set_routing, state,
+		rval = v4l2_subdev_call(sd, pad, set_routing, state,
 					routing->which, &krouting);
+		if (rval < 0)
+			return rval;
+
+		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
+		       state->routing.routes,
+		       state->routing.num_routes *
+		       sizeof(*state->routing.routes));
+		routing->num_routes = state->routing.num_routes;
+
+		return 0;
 	}
 
 	case VIDIOC_SUBDEV_G_ROUTING: {
-- 
2.39.2

[PATCH v8 16/38] media: v4l: subdev: Allow a larger number of routes than there's room for

[Sakari Ailus]

On VIDIOC_SUBDEV_[GS]_ROUTING, only return as many routes back to the user
as there's room. Do not consider it an error if more routes existed.
Simply inform the user there are more routes.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../media/v4l/vidioc-subdev-g-routing.rst              |  4 ----
 drivers/media/v4l2-core/v4l2-subdev.c                  | 10 +++-------
 2 files changed, 3 insertions(+), 11 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
index 6eb6a59570dc..08b8d17cef3f 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
@@ -147,10 +147,6 @@ On success 0 is returned, on error -1 and the ``errno`` variable is set
 appropriately. The generic error codes are described at the
 :ref:`Generic Error Codes <gen-errors>` chapter.
 
-ENOSPC
-   The application provided ``num_routes`` is not big enough to contain
-   all the available routes the subdevice exposes.
-
 EINVAL
    The sink or source pad identifiers reference a non-existing pad or reference
    pads of different types (ie. the sink_pad identifiers refers to a source
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index a357ce318192..a6107e440ef0 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -964,7 +964,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 
 		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
 		       state->routing.routes,
-		       state->routing.num_routes *
+		       min(state->routing.num_routes, routing->len_routes)  *
 		       sizeof(*state->routing.routes));
 		routing->num_routes = state->routing.num_routes;
 
@@ -985,14 +985,10 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 
 		krouting = &state->routing;
 
-		if (routing->len_routes < krouting->num_routes) {
-			routing->num_routes = krouting->num_routes;
-			return -ENOSPC;
-		}
-
 		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
 		       krouting->routes,
-		       krouting->num_routes * sizeof(*krouting->routes));
+		       min(krouting->num_routes, routing->len_routes) *
+		       sizeof(*krouting->routes));
 		routing->num_routes = krouting->num_routes;
 
 		return 0;
-- 
2.39.2

[PATCH v8 17/38] media: v4l: subdev: Add trivial set_routing support

[Sakari Ailus]

Add trivial S_ROUTING IOCTL support for drivers where routing is static.
Essentially this means returning the same information G_ROUTING call would
have done.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index a6107e440ef0..c8c435df92c8 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -930,6 +930,20 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 
 		memset(routing->reserved, 0, sizeof(routing->reserved));
 
+		/*
+		 * If the driver doesn't support setting routing, just return
+		 * the routing table here.
+		 */
+		if (!v4l2_subdev_has_op(sd, pad, set_routing)) {
+			memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
+			       state->routing.routes,
+			       min(state->routing.num_routes, routing->len_routes) *
+			       sizeof(*state->routing.routes));
+			routing->num_routes = state->routing.num_routes;
+
+			return 0;
+		}
+
 		for (i = 0; i < routing->num_routes; ++i) {
 			const struct v4l2_subdev_route *route = &routes[i];
 			const struct media_pad *pads = sd->entity.pads;
-- 
2.39.2

[PATCH v8 18/38] media: ccs: No need to set streaming to false in power off

[Sakari Ailus]

Streaming will have been stopped by the sensor is powered off, and so
sensor->streaming is also false already. Do not set it as part of the
runtime suspend callback.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ccs/ccs-core.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
index e21287d50c15..671540a8ab6a 100644
--- a/drivers/media/i2c/ccs/ccs-core.c
+++ b/drivers/media/i2c/ccs/ccs-core.c
@@ -1710,7 +1710,6 @@ static int ccs_power_off(struct device *dev)
 	usleep_range(5000, 5000);
 	regulator_bulk_disable(ARRAY_SIZE(ccs_regulators),
 			       sensor->regulators);
-	sensor->streaming = false;
 
 	return 0;
 }
-- 
2.39.2

[PATCH v8 19/38] media: ccs: Use {enable,disable}_streams operations

[Sakari Ailus]

Switch from s_stream() video op to enable_streams() and disable_streams()
pad operations. They are preferred and required for streams support.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ccs/ccs-core.c | 187 +++++++++++++++----------------
 1 file changed, 88 insertions(+), 99 deletions(-)

diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
index 671540a8ab6a..9e70946653e9 100644
--- a/drivers/media/i2c/ccs/ccs-core.c
+++ b/drivers/media/i2c/ccs/ccs-core.c
@@ -1715,22 +1715,66 @@ static int ccs_power_off(struct device *dev)
 }
 
 /* -----------------------------------------------------------------------------
- * Video stream management
+ * V4L2 subdev video operations
  */
 
-static int ccs_start_streaming(struct ccs_sensor *sensor)
+static int ccs_pm_get_init(struct ccs_sensor *sensor)
+{
+	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
+	int rval;
+
+	/*
+	 * It can't use pm_runtime_resume_and_get() here, as the driver
+	 * relies at the returned value to detect if the device was already
+	 * active or not.
+	 */
+	rval = pm_runtime_get_sync(&client->dev);
+	if (rval < 0)
+		goto error;
+
+	/* Device was already active, so don't set controls */
+	if (rval == 1 && !sensor->handler_setup_needed)
+		return 0;
+
+	sensor->handler_setup_needed = false;
+
+	/* Restore V4L2 controls to the previously suspended device */
+	rval = __v4l2_ctrl_handler_setup(&sensor->pixel_array->ctrl_handler);
+	if (rval)
+		goto error;
+
+	rval = __v4l2_ctrl_handler_setup(&sensor->src->ctrl_handler);
+	if (rval)
+		goto error;
+
+	/* Keep PM runtime usage_count incremented on success */
+	return 0;
+error:
+	pm_runtime_put(&client->dev);
+	return rval;
+}
+
+static int ccs_enable_streams(struct v4l2_subdev *subdev,
+			      struct v4l2_subdev_state *state, u32 pad,
+			      u64 streams_mask)
 {
+	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
 	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
 	unsigned int binning_mode;
 	int rval;
 
-	mutex_lock(&sensor->mutex);
+	if (pad != CCS_PAD_SRC)
+		return -EINVAL;
+
+	rval = ccs_pm_get_init(sensor);
+	if (rval)
+		return rval;
 
 	rval = ccs_write(sensor, CSI_DATA_FORMAT,
 			 (sensor->csi_format->width << 8) |
 			 sensor->csi_format->compressed);
 	if (rval)
-		goto out;
+		goto err_pm_put;
 
 	/* Binning configuration */
 	if (sensor->binning_horizontal == 1 &&
@@ -1743,38 +1787,38 @@ static int ccs_start_streaming(struct ccs_sensor *sensor)
 
 		rval = ccs_write(sensor, BINNING_TYPE, binning_type);
 		if (rval < 0)
-			goto out;
+			goto err_pm_put;
 
 		binning_mode = 1;
 	}
 	rval = ccs_write(sensor, BINNING_MODE, binning_mode);
 	if (rval < 0)
-		goto out;
+		goto err_pm_put;
 
 	/* Set up PLL */
 	rval = ccs_pll_configure(sensor);
 	if (rval)
-		goto out;
+		goto err_pm_put;
 
 	/* Analog crop start coordinates */
 	rval = ccs_write(sensor, X_ADDR_START, sensor->pa_src.left);
 	if (rval < 0)
-		goto out;
+		goto err_pm_put;
 
 	rval = ccs_write(sensor, Y_ADDR_START, sensor->pa_src.top);
 	if (rval < 0)
-		goto out;
+		goto err_pm_put;
 
 	/* Analog crop end coordinates */
 	rval = ccs_write(sensor, X_ADDR_END,
 			 sensor->pa_src.left + sensor->pa_src.width - 1);
 	if (rval < 0)
-		goto out;
+		goto err_pm_put;
 
 	rval = ccs_write(sensor, Y_ADDR_END,
 			 sensor->pa_src.top + sensor->pa_src.height - 1);
 	if (rval < 0)
-		goto out;
+		goto err_pm_put;
 
 	/*
 	 * Output from pixel array, including blanking, is set using
@@ -1787,22 +1831,22 @@ static int ccs_start_streaming(struct ccs_sensor *sensor)
 		rval = ccs_write(sensor, DIGITAL_CROP_X_OFFSET,
 				 sensor->scaler_sink.left);
 		if (rval < 0)
-			goto out;
+			goto err_pm_put;
 
 		rval = ccs_write(sensor, DIGITAL_CROP_Y_OFFSET,
 				 sensor->scaler_sink.top);
 		if (rval < 0)
-			goto out;
+			goto err_pm_put;
 
 		rval = ccs_write(sensor, DIGITAL_CROP_IMAGE_WIDTH,
 				 sensor->scaler_sink.width);
 		if (rval < 0)
-			goto out;
+			goto err_pm_put;
 
 		rval = ccs_write(sensor, DIGITAL_CROP_IMAGE_HEIGHT,
 				 sensor->scaler_sink.height);
 		if (rval < 0)
-			goto out;
+			goto err_pm_put;
 	}
 
 	/* Scaling */
@@ -1810,20 +1854,20 @@ static int ccs_start_streaming(struct ccs_sensor *sensor)
 	    != CCS_SCALING_CAPABILITY_NONE) {
 		rval = ccs_write(sensor, SCALING_MODE, sensor->scaling_mode);
 		if (rval < 0)
-			goto out;
+			goto err_pm_put;
 
 		rval = ccs_write(sensor, SCALE_M, sensor->scale_m);
 		if (rval < 0)
-			goto out;
+			goto err_pm_put;
 	}
 
 	/* Output size from sensor */
 	rval = ccs_write(sensor, X_OUTPUT_SIZE, sensor->src_src.width);
 	if (rval < 0)
-		goto out;
+		goto err_pm_put;
 	rval = ccs_write(sensor, Y_OUTPUT_SIZE, sensor->src_src.height);
 	if (rval < 0)
-		goto out;
+		goto err_pm_put;
 
 	if (CCS_LIM(sensor, FLASH_MODE_CAPABILITY) &
 	    (CCS_FLASH_MODE_CAPABILITY_SINGLE_STROBE |
@@ -1832,111 +1876,52 @@ static int ccs_start_streaming(struct ccs_sensor *sensor)
 	    sensor->hwcfg.strobe_setup->trigger != 0) {
 		rval = ccs_setup_flash_strobe(sensor);
 		if (rval)
-			goto out;
+			goto err_pm_put;
 	}
 
 	rval = ccs_call_quirk(sensor, pre_streamon);
 	if (rval) {
 		dev_err(&client->dev, "pre_streamon quirks failed\n");
-		goto out;
+		goto err_pm_put;
 	}
 
 	rval = ccs_write(sensor, MODE_SELECT, CCS_MODE_SELECT_STREAMING);
 
-out:
-	mutex_unlock(&sensor->mutex);
-
-	return rval;
-}
-
-static int ccs_stop_streaming(struct ccs_sensor *sensor)
-{
-	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
-	int rval;
-
-	mutex_lock(&sensor->mutex);
-	rval = ccs_write(sensor, MODE_SELECT, CCS_MODE_SELECT_SOFTWARE_STANDBY);
-	if (rval)
-		goto out;
-
-	rval = ccs_call_quirk(sensor, post_streamoff);
-	if (rval)
-		dev_err(&client->dev, "post_streamoff quirks failed\n");
-
-out:
-	mutex_unlock(&sensor->mutex);
-	return rval;
-}
-
-/* -----------------------------------------------------------------------------
- * V4L2 subdev video operations
- */
-
-static int ccs_pm_get_init(struct ccs_sensor *sensor)
-{
-	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
-	int rval;
-
-	/*
-	 * It can't use pm_runtime_resume_and_get() here, as the driver
-	 * relies at the returned value to detect if the device was already
-	 * active or not.
-	 */
-	rval = pm_runtime_get_sync(&client->dev);
-	if (rval < 0)
-		goto error;
-
-	/* Device was already active, so don't set controls */
-	if (rval == 1 && !sensor->handler_setup_needed)
-		return 0;
-
-	sensor->handler_setup_needed = false;
+	sensor->streaming = true;
 
-	/* Restore V4L2 controls to the previously suspended device */
-	rval = v4l2_ctrl_handler_setup(&sensor->pixel_array->ctrl_handler);
-	if (rval)
-		goto error;
+	return 0;
 
-	rval = v4l2_ctrl_handler_setup(&sensor->src->ctrl_handler);
-	if (rval)
-		goto error;
+err_pm_put:
+	pm_runtime_mark_last_busy(&client->dev);
+	pm_runtime_put_autosuspend(&client->dev);
 
-	/* Keep PM runtime usage_count incremented on success */
-	return 0;
-error:
-	pm_runtime_put(&client->dev);
 	return rval;
 }
 
-static int ccs_set_stream(struct v4l2_subdev *subdev, int enable)
+static int ccs_disable_streams(struct v4l2_subdev *subdev,
+			       struct v4l2_subdev_state *state, u32 pad,
+			       u64 streams_mask)
 {
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
 	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
 	int rval;
 
-	if (!enable) {
-		ccs_stop_streaming(sensor);
-		sensor->streaming = false;
-		pm_runtime_mark_last_busy(&client->dev);
-		pm_runtime_put_autosuspend(&client->dev);
-
-		return 0;
-	}
+	if (pad != CCS_PAD_SRC)
+		return -EINVAL;
 
-	rval = ccs_pm_get_init(sensor);
+	rval = ccs_write(sensor, MODE_SELECT, CCS_MODE_SELECT_SOFTWARE_STANDBY);
 	if (rval)
 		return rval;
 
-	sensor->streaming = true;
+	rval = ccs_call_quirk(sensor, post_streamoff);
+	if (rval)
+		dev_err(&client->dev, "post_streamoff quirks failed\n");
 
-	rval = ccs_start_streaming(sensor);
-	if (rval < 0) {
-		sensor->streaming = false;
-		pm_runtime_mark_last_busy(&client->dev);
-		pm_runtime_put_autosuspend(&client->dev);
-	}
+	sensor->streaming = false;
+	pm_runtime_mark_last_busy(&client->dev);
+	pm_runtime_put_autosuspend(&client->dev);
 
-	return rval;
+	return 0;
 }
 
 static int ccs_pre_streamon(struct v4l2_subdev *subdev, u32 flags)
@@ -1962,7 +1947,9 @@ static int ccs_pre_streamon(struct v4l2_subdev *subdev, u32 flags)
 		}
 	}
 
+	mutex_lock(&sensor->mutex);
 	rval = ccs_pm_get_init(sensor);
+	mutex_unlock(&sensor->mutex);
 	if (rval)
 		return rval;
 
@@ -3046,7 +3033,7 @@ static int ccs_init_state(struct v4l2_subdev *sd,
 }
 
 static const struct v4l2_subdev_video_ops ccs_video_ops = {
-	.s_stream = ccs_set_stream,
+	.s_stream = v4l2_subdev_s_stream_helper,
 	.pre_streamon = ccs_pre_streamon,
 	.post_streamoff = ccs_post_streamoff,
 };
@@ -3057,6 +3044,8 @@ static const struct v4l2_subdev_pad_ops ccs_pad_ops = {
 	.set_fmt = ccs_set_format,
 	.get_selection = ccs_get_selection,
 	.set_selection = ccs_set_selection,
+	.enable_streams = ccs_enable_streams,
+	.disable_streams = ccs_disable_streams,
 };
 
 static const struct v4l2_subdev_sensor_ops ccs_sensor_ops = {
-- 
2.39.2

[PATCH v8 20/38] media: ccs: Track streaming state

[Sakari Ailus]

With enable_streams and disable_streams, the driver for a device where
streams are not independently started and stopped needs to maintain state
information on streams that have been requested to be started. Do that
now.

In the future, a helper function in the framework is a desirable way to do
this instead.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ccs/ccs-core.c | 13 ++++++++++---
 drivers/media/i2c/ccs/ccs.h      |  2 +-
 2 files changed, 11 insertions(+), 4 deletions(-)

diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
index 9e70946653e9..0cd8ee957655 100644
--- a/drivers/media/i2c/ccs/ccs-core.c
+++ b/drivers/media/i2c/ccs/ccs-core.c
@@ -1766,6 +1766,11 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
 	if (pad != CCS_PAD_SRC)
 		return -EINVAL;
 
+	if (sensor->streaming) {
+		sensor->streaming |= streams_mask;
+		return 0;
+	}
+
 	rval = ccs_pm_get_init(sensor);
 	if (rval)
 		return rval;
@@ -1887,7 +1892,7 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
 
 	rval = ccs_write(sensor, MODE_SELECT, CCS_MODE_SELECT_STREAMING);
 
-	sensor->streaming = true;
+	sensor->streaming |= streams_mask;
 
 	return 0;
 
@@ -1909,6 +1914,10 @@ static int ccs_disable_streams(struct v4l2_subdev *subdev,
 	if (pad != CCS_PAD_SRC)
 		return -EINVAL;
 
+	sensor->streaming &= ~streams_mask;
+	if (sensor->streaming)
+		return 0;
+
 	rval = ccs_write(sensor, MODE_SELECT, CCS_MODE_SELECT_SOFTWARE_STANDBY);
 	if (rval)
 		return rval;
@@ -1917,7 +1926,6 @@ static int ccs_disable_streams(struct v4l2_subdev *subdev,
 	if (rval)
 		dev_err(&client->dev, "post_streamoff quirks failed\n");
 
-	sensor->streaming = false;
 	pm_runtime_mark_last_busy(&client->dev);
 	pm_runtime_put_autosuspend(&client->dev);
 
@@ -3525,7 +3533,6 @@ static int ccs_probe(struct i2c_client *client)
 		goto out_cleanup;
 	}
 
-	sensor->streaming = false;
 	sensor->dev_init_done = true;
 	sensor->handler_setup_needed = true;
 
diff --git a/drivers/media/i2c/ccs/ccs.h b/drivers/media/i2c/ccs/ccs.h
index 096573845a10..4725e6eca8d0 100644
--- a/drivers/media/i2c/ccs/ccs.h
+++ b/drivers/media/i2c/ccs/ccs.h
@@ -236,7 +236,7 @@ struct ccs_sensor {
 	u16 image_start; /* image data start line */
 	u16 visible_pixel_start; /* start pixel of the visible image */
 
-	bool streaming;
+	u8 streaming;
 	bool dev_init_done;
 	bool handler_setup_needed;
 	u8 compressed_min_bpp;
-- 
2.39.2

[PATCH v8 21/38] media: ccs: Move ccs_validate_csi_data_format up

[Sakari Ailus]

ccs_validate_csi_data_format() will soon be needed elsewhere, above its
current location. Move it up.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ccs/ccs-core.c | 28 ++++++++++++++--------------
 1 file changed, 14 insertions(+), 14 deletions(-)

diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
index 0cd8ee957655..0efbc63534bc 100644
--- a/drivers/media/i2c/ccs/ccs-core.c
+++ b/drivers/media/i2c/ccs/ccs-core.c
@@ -1979,6 +1979,20 @@ static int ccs_post_streamoff(struct v4l2_subdev *subdev)
 	return pm_runtime_put(&client->dev);
 }
 
+static const struct ccs_csi_data_format
+*ccs_validate_csi_data_format(struct ccs_sensor *sensor, u32 code)
+{
+	unsigned int i;
+
+	for (i = 0; i < ARRAY_SIZE(ccs_csi_data_formats); i++) {
+		if (sensor->mbus_frame_fmts & (1 << i) &&
+		    ccs_csi_data_formats[i].code == code)
+			return &ccs_csi_data_formats[i];
+	}
+
+	return sensor->csi_format;
+}
+
 static int ccs_enum_mbus_code(struct v4l2_subdev *subdev,
 			      struct v4l2_subdev_state *sd_state,
 			      struct v4l2_subdev_mbus_code_enum *code)
@@ -2114,20 +2128,6 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
 	}
 }
 
-static const struct ccs_csi_data_format
-*ccs_validate_csi_data_format(struct ccs_sensor *sensor, u32 code)
-{
-	unsigned int i;
-
-	for (i = 0; i < ARRAY_SIZE(ccs_csi_data_formats); i++) {
-		if (sensor->mbus_frame_fmts & (1 << i) &&
-		    ccs_csi_data_formats[i].code == code)
-			return &ccs_csi_data_formats[i];
-	}
-
-	return sensor->csi_format;
-}
-
 static int ccs_set_format_source(struct v4l2_subdev *subdev,
 				 struct v4l2_subdev_state *sd_state,
 				 struct v4l2_subdev_format *fmt)
-- 
2.39.2

[PATCH v8 22/38] media: ccs: Support frame descriptors

[Sakari Ailus]

Provide information on the frame layout using frame descriptors.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ccs/ccs-core.c  | 60 +++++++++++++++++++++++++++++++
 drivers/media/i2c/ccs/ccs-quirk.h |  7 ++++
 drivers/media/i2c/ccs/ccs.h       |  4 +++
 3 files changed, 71 insertions(+)

diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
index 0efbc63534bc..9cc2080b73ec 100644
--- a/drivers/media/i2c/ccs/ccs-core.c
+++ b/drivers/media/i2c/ccs/ccs-core.c
@@ -25,6 +25,7 @@
 #include <linux/slab.h>
 #include <linux/smiapp.h>
 #include <linux/v4l2-mediabus.h>
+#include <media/mipi-csi2.h>
 #include <media/v4l2-cci.h>
 #include <media/v4l2-device.h>
 #include <media/v4l2-fwnode.h>
@@ -245,6 +246,33 @@ static int ccs_read_all_limits(struct ccs_sensor *sensor)
 	return ret;
 }
 
+static u8 ccs_mipi_csi2_data_type(unsigned int bpp)
+{
+	switch (bpp) {
+	case 6:
+		return MIPI_CSI2_DT_RAW6;
+	case 7:
+		return MIPI_CSI2_DT_RAW7;
+	case 8:
+		return MIPI_CSI2_DT_RAW8;
+	case 10:
+		return MIPI_CSI2_DT_RAW10;
+	case 12:
+		return MIPI_CSI2_DT_RAW12;
+	case 14:
+		return MIPI_CSI2_DT_RAW14;
+	case 16:
+		return MIPI_CSI2_DT_RAW16;
+	case 20:
+		return MIPI_CSI2_DT_RAW20;
+	case 24:
+		return MIPI_CSI2_DT_RAW24;
+	default:
+		WARN_ON(1);
+		return 0;
+	}
+}
+
 static int ccs_read_frame_fmt(struct ccs_sensor *sensor)
 {
 	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
@@ -2632,6 +2660,37 @@ static int ccs_set_selection(struct v4l2_subdev *subdev,
 	return ret;
 }
 
+static int ccs_get_frame_desc(struct v4l2_subdev *subdev, unsigned int pad,
+				 struct v4l2_mbus_frame_desc *desc)
+{
+	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
+	struct v4l2_mbus_frame_desc_entry *entry = desc->entry;
+
+	if (ccs_has_quirk(sensor, frame_desc))
+		return ccs_call_quirk(sensor, frame_desc, desc);
+
+	switch (sensor->hwcfg.csi_signalling_mode) {
+	case CCS_CSI_SIGNALING_MODE_CSI_2_DPHY:
+	case CCS_CSI_SIGNALING_MODE_CSI_2_CPHY:
+		desc->type = V4L2_MBUS_FRAME_DESC_TYPE_CSI2;
+		break;
+	default:
+		/* FIXME: CCP2 support */
+		return -EINVAL;
+	}
+
+	entry->pixelcode = sensor->csi_format->code;
+	entry->stream = CCS_STREAM_PIXEL;
+	entry->bus.csi2.dt =
+		sensor->csi_format->width == sensor->csi_format->compressed ?
+		ccs_mipi_csi2_data_type(sensor->csi_format->compressed) :
+		CCS_DEFAULT_COMPRESSED_DT;
+	entry++;
+	desc->num_entries++;
+
+	return 0;
+}
+
 static int ccs_get_skip_frames(struct v4l2_subdev *subdev, u32 *frames)
 {
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
@@ -3054,6 +3113,7 @@ static const struct v4l2_subdev_pad_ops ccs_pad_ops = {
 	.set_selection = ccs_set_selection,
 	.enable_streams = ccs_enable_streams,
 	.disable_streams = ccs_disable_streams,
+	.get_frame_desc = ccs_get_frame_desc,
 };
 
 static const struct v4l2_subdev_sensor_ops ccs_sensor_ops = {
diff --git a/drivers/media/i2c/ccs/ccs-quirk.h b/drivers/media/i2c/ccs/ccs-quirk.h
index 392c97109617..3e1d9eaa33fa 100644
--- a/drivers/media/i2c/ccs/ccs-quirk.h
+++ b/drivers/media/i2c/ccs/ccs-quirk.h
@@ -36,6 +36,7 @@ struct ccs_sensor;
  *			 access may be done by the caller (default read
  *			 value is zero), else negative error code on error
  * @flags: Quirk flags
+ * @frame_desc: Obtain the frame descriptor
  */
 struct ccs_quirk {
 	int (*limits)(struct ccs_sensor *sensor);
@@ -46,6 +47,8 @@ struct ccs_quirk {
 	int (*init)(struct ccs_sensor *sensor);
 	int (*reg_access)(struct ccs_sensor *sensor, bool write, u32 *reg,
 			  u32 *val);
+	int (*frame_desc)(struct ccs_sensor *sensor,
+			  struct v4l2_mbus_frame_desc *desc);
 	unsigned long flags;
 };
 
@@ -62,6 +65,10 @@ struct ccs_reg_8 {
 		.val = _val,		\
 	}
 
+#define ccs_has_quirk(sensor, _quirk)					\
+	((sensor)->minfo.quirk &&					\
+	 (sensor)->minfo.quirk->_quirk)
+
 #define ccs_call_quirk(sensor, _quirk, ...)				\
 	((sensor)->minfo.quirk &&					\
 	 (sensor)->minfo.quirk->_quirk ?				\
diff --git a/drivers/media/i2c/ccs/ccs.h b/drivers/media/i2c/ccs/ccs.h
index 4725e6eca8d0..adb152366ea2 100644
--- a/drivers/media/i2c/ccs/ccs.h
+++ b/drivers/media/i2c/ccs/ccs.h
@@ -46,6 +46,8 @@
 
 #define CCS_COLOUR_COMPONENTS		4
 
+#define CCS_DEFAULT_COMPRESSED_DT	0x30
+
 #define SMIAPP_NAME			"smiapp"
 #define CCS_NAME			"ccs"
 
@@ -175,6 +177,8 @@ struct ccs_csi_data_format {
 #define CCS_PAD_SRC			1
 #define CCS_PADS			2
 
+#define CCS_STREAM_PIXEL		0
+
 struct ccs_binning_subtype {
 	u8 horizontal:4;
 	u8 vertical:4;
-- 
2.39.2

[PATCH v8 23/38] media: uapi: ccs: Add media bus code for MIPI CCS embedded data

[Sakari Ailus]

Add new MIPI CCS embedded data media bus code
(MEDIA_BUS_FMT_CCS_EMBEDDED).

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../media/v4l/subdev-formats.rst              | 28 +++++++++++++++++++
 include/uapi/linux/media-bus-format.h         |  3 ++
 2 files changed, 31 insertions(+)

diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
index cbd475f7cae9..c8f982411e70 100644
--- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
+++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
@@ -8564,3 +8564,31 @@ and finally the bit number in subscript. "X" indicates a padding bit.
       - X
       - X
       - X
+
+.. _MEDIA-BUS-FMT-CCS-EMBEDDED:
+
+MIPI CCS Embedded Data Formats
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+`MIPI CCS <https://www.mipi.org/specifications/camera-command-set>`_ defines a
+metadata format for sensor embedded data, which is used to store the register
+configuration used for capturing a given frame. The format is defined in the CCS
+specification. The media bus code for this format is
+``MEDIA_BUS_FMT_CCS_EMBEDDED``.
+
+The CCS embedded data format definition includes three levels:
+
+1. Padding within CSI-2 bus :ref:`Data unit <media-glossary-data-unit>` as
+   documented in the MIPI CCS specification.
+
+2. The tagged data format as documented in the MIPI CCS specification.
+
+3. Register addresses and register documentation as documented in the MIPI CCS
+   specification.
+
+The format definition shall be used only by devices that fulfill all three
+levels above.
+
+This mbus code are only used for "2-byte simplified tagged data format" (code
+0xa) but their use may be extended further in the future, to cover other CCS
+embedded data format codes.
diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
index d4c1d991014b..03f7e9ab517b 100644
--- a/include/uapi/linux/media-bus-format.h
+++ b/include/uapi/linux/media-bus-format.h
@@ -183,4 +183,7 @@
 #define MEDIA_BUS_FMT_META_20			0x8006
 #define MEDIA_BUS_FMT_META_24			0x8007
 
+/* Specific metadata formats. Next is 0x9002. */
+#define MEDIA_BUS_FMT_CCS_EMBEDDED		0x9001
+
 #endif /* __LINUX_MEDIA_BUS_FORMAT_H */
-- 
2.39.2

[PATCH v8 24/38] media: ccs: Add support for embedded data stream

[Sakari Ailus]

Add support for embedded data stream, in UAPI and frame descriptor.

This patch adds also a new embedded data pad (2) to the source sub-device.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ccs/ccs-core.c | 370 +++++++++++++++++++++++++++++--
 drivers/media/i2c/ccs/ccs.h      |  18 +-
 2 files changed, 360 insertions(+), 28 deletions(-)

diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
index 9cc2080b73ec..1f91af25eac6 100644
--- a/drivers/media/i2c/ccs/ccs-core.c
+++ b/drivers/media/i2c/ccs/ccs-core.c
@@ -1902,6 +1902,13 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
 	if (rval < 0)
 		goto err_pm_put;
 
+	/* Configure embedded data */
+	if (sensor->csi_format->compressed >= 16) {
+		rval = ccs_write(sensor, EMB_DATA_CTRL, sensor->emb_data_ctrl);
+		if (rval < 0)
+			goto err_pm_put;
+	}
+
 	if (CCS_LIM(sensor, FLASH_MODE_CAPABILITY) &
 	    (CCS_FLASH_MODE_CAPABILITY_SINGLE_STROBE |
 	     SMIAPP_FLASH_MODE_CAPABILITY_MULTIPLE_STROBE) &&
@@ -2021,6 +2028,57 @@ static const struct ccs_csi_data_format
 	return sensor->csi_format;
 }
 
+#define CCS_EMBEDDED_CODE_DEPTH(depth, half_depth)			\
+	depth,								\
+	CCS_EMB_DATA_CAPABILITY_TWO_BYTES_PER_RAW##depth,		\
+	CCS_EMB_DATA_CAPABILITY_NO_ONE_BYTE_PER_RAW##depth,		\
+	CCS_EMB_DATA_CTRL_RAW##half_depth##_PACKING_FOR_RAW##depth,	\
+	MEDIA_BUS_FMT_META_##half_depth,				\
+	MEDIA_BUS_FMT_META_##depth,					\
+
+static const struct ccs_embedded_code {
+	u8 depth;
+	u8 cap_two_bytes_per_sample;
+	u8 cap_no_legacy;
+	u8 ctrl;
+	u32 code_two_bytes;
+	u32 code_legacy;
+} ccs_embedded_codes[] = {
+	{ CCS_EMBEDDED_CODE_DEPTH(16, 8) },
+	{ CCS_EMBEDDED_CODE_DEPTH(20, 10) },
+	{ CCS_EMBEDDED_CODE_DEPTH(24, 12) },
+};
+
+static const struct ccs_embedded_code *ccs_embedded_code(unsigned int bpp)
+{
+	unsigned int i;
+
+	for (i = 0; i < ARRAY_SIZE(ccs_embedded_codes); i++)
+		if (ccs_embedded_codes[i].depth == bpp)
+			return ccs_embedded_codes + i;
+
+	WARN_ON(1);
+
+	return ccs_embedded_codes;
+}
+
+static u32
+ccs_default_embedded_code(struct ccs_sensor *sensor,
+			  const struct ccs_embedded_code *embedded_code)
+{
+	if (CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
+	    BIT(embedded_code->cap_two_bytes_per_sample))
+		return embedded_code->code_two_bytes;
+
+	if (!(CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
+	      BIT(embedded_code->cap_no_legacy)))
+		return embedded_code->code_legacy;
+
+	WARN_ON(1);
+
+	return embedded_code->code_legacy;
+}
+
 static int ccs_enum_mbus_code(struct v4l2_subdev *subdev,
 			      struct v4l2_subdev_state *sd_state,
 			      struct v4l2_subdev_mbus_code_enum *code)
@@ -2036,6 +2094,69 @@ static int ccs_enum_mbus_code(struct v4l2_subdev *subdev,
 	dev_err(&client->dev, "subdev %s, pad %u, index %u\n",
 		subdev->name, code->pad, code->index);
 
+	if (subdev == &sensor->src->sd) {
+		if (code->pad == CCS_PAD_META) {
+			if (code->index)
+				goto out;
+
+			code->code = MEDIA_BUS_FMT_CCS_EMBEDDED;
+
+			rval = 0;
+			goto out;
+		}
+		if (code->stream == CCS_STREAM_META) {
+			struct v4l2_mbus_framefmt *pix_fmt =
+				v4l2_subdev_state_get_format(sd_state,
+							     CCS_PAD_SRC,
+							     CCS_STREAM_PIXEL);
+			const struct ccs_csi_data_format *csi_format =
+				ccs_validate_csi_data_format(sensor,
+							     pix_fmt->code);
+			unsigned int i = 0;
+			u32 codes[2];
+
+			switch (csi_format->compressed) {
+			case 8:
+				codes[i++] = MEDIA_BUS_FMT_META_8;
+				break;
+			case 10:
+				codes[i++] = MEDIA_BUS_FMT_META_10;
+				break;
+			case 12:
+				codes[i++] = MEDIA_BUS_FMT_META_12;
+				break;
+			case 14:
+				codes[i++] = MEDIA_BUS_FMT_META_14;
+				break;
+			case 16:
+			case 20:
+			case 24: {
+				const struct ccs_embedded_code *embedded_code =
+					ccs_embedded_code(csi_format->compressed);
+
+				if (CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
+				    BIT(embedded_code->cap_two_bytes_per_sample))
+					codes[i++] =
+						embedded_code->code_two_bytes;
+
+				if (!(CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
+				      BIT(embedded_code->cap_no_legacy)))
+					codes[i++] = embedded_code->code_legacy;
+				break;
+			}
+			default:
+				WARN_ON(1);
+			}
+
+			if (WARN_ON(i > ARRAY_SIZE(codes)) || code->index >= i)
+				goto out;
+
+			code->code = codes[code->index];
+			rval = 0;
+			goto out;
+		}
+	}
+
 	if (subdev != &sensor->src->sd || code->pad != CCS_PAD_SRC) {
 		if (code->index)
 			goto out;
@@ -2078,8 +2199,11 @@ static int __ccs_get_format(struct v4l2_subdev *subdev,
 			    struct v4l2_subdev_state *sd_state,
 			    struct v4l2_subdev_format *fmt)
 {
-	fmt->format = *v4l2_subdev_state_get_format(sd_state, fmt->pad);
-	fmt->format.code = __ccs_get_mbus_code(subdev, fmt->pad);
+	fmt->format = *v4l2_subdev_state_get_format(sd_state, fmt->pad,
+						    fmt->stream);
+
+	if (fmt->pad != CCS_PAD_META && fmt->stream != CCS_STREAM_META)
+		fmt->format.code = __ccs_get_mbus_code(subdev, fmt->pad);
 
 	return 0;
 }
@@ -2109,10 +2233,11 @@ static void ccs_get_crop_compose(struct v4l2_subdev *subdev,
 	if (crops)
 		for (i = 0; i < subdev->entity.num_pads; i++)
 			crops[i] =
-				v4l2_subdev_state_get_crop(sd_state, i);
+				v4l2_subdev_state_get_crop(sd_state, i,
+							   CCS_STREAM_PIXEL);
 	if (comps)
-		*comps = v4l2_subdev_state_get_compose(sd_state,
-						       ssd->sink_pad);
+		*comps = v4l2_subdev_state_get_compose(sd_state, ssd->sink_pad,
+						       CCS_STREAM_PIXEL);
 }
 
 /* Changes require propagation only on sink pad. */
@@ -2145,7 +2270,8 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
 		fallthrough;
 	case V4L2_SEL_TGT_COMPOSE:
 		*crops[CCS_PAD_SRC] = *comp;
-		fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC);
+		fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC,
+						   CCS_STREAM_PIXEL);
 		fmt->width = comp->width;
 		fmt->height = comp->height;
 		if (which == V4L2_SUBDEV_FORMAT_ACTIVE && ssd == sensor->src)
@@ -2209,6 +2335,81 @@ static int ccs_set_format_source(struct v4l2_subdev *subdev,
 	return ccs_pll_update(sensor);
 }
 
+static int ccs_set_format_meta(struct v4l2_subdev *subdev,
+			       struct v4l2_subdev_state *sd_state,
+			       struct v4l2_mbus_framefmt *fmt)
+{
+	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
+	const struct ccs_csi_data_format *csi_format;
+	struct v4l2_mbus_framefmt *pix_fmt;
+	struct v4l2_mbus_framefmt *meta_fmt;
+	struct v4l2_mbus_framefmt *meta_out_fmt;
+	u32 code;
+
+	pix_fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC,
+					       CCS_STREAM_PIXEL);
+	meta_fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_META, 0);
+	meta_out_fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC,
+						    CCS_STREAM_META);
+
+	code = fmt ? fmt->code : meta_out_fmt->code;
+
+	meta_out_fmt->width = meta_fmt->width = pix_fmt->width;
+	meta_out_fmt->height = meta_fmt->height =
+		sensor->embedded_end - sensor->embedded_start;
+	meta_fmt->code = MEDIA_BUS_FMT_CCS_EMBEDDED;
+
+	csi_format = ccs_validate_csi_data_format(sensor, pix_fmt->code);
+
+	switch (csi_format->compressed) {
+	case 8:
+		meta_out_fmt->code = MEDIA_BUS_FMT_META_8;
+		break;
+	case 10:
+		meta_out_fmt->code = MEDIA_BUS_FMT_META_10;
+		break;
+	case 12:
+		meta_out_fmt->code = MEDIA_BUS_FMT_META_12;
+		break;
+	case 14:
+		meta_out_fmt->code = MEDIA_BUS_FMT_META_14;
+		break;
+	case 16:
+	case 20:
+	case 24: {
+		const struct ccs_embedded_code *embedded_code;
+
+		embedded_code = ccs_embedded_code(csi_format->compressed);
+		meta_out_fmt->code =
+			ccs_default_embedded_code(sensor, embedded_code);
+
+		if (!(CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
+		      BIT(embedded_code->cap_no_legacy)) &&
+		    code == embedded_code->code_legacy) {
+			meta_out_fmt->code = embedded_code->code_legacy;
+			sensor->emb_data_ctrl = 0;
+		}
+
+		if (CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
+		    BIT(embedded_code->cap_two_bytes_per_sample) &&
+		    code == embedded_code->code_two_bytes) {
+			meta_out_fmt->code = embedded_code->code_two_bytes;
+			sensor->emb_data_ctrl = embedded_code->ctrl;
+		}
+
+		break;
+	}
+	default:
+		WARN_ON(1);
+		return 0;
+	}
+
+	if (fmt)
+		*fmt = *meta_out_fmt;
+
+	return 0;
+}
+
 static int ccs_set_format(struct v4l2_subdev *subdev,
 			  struct v4l2_subdev_state *sd_state,
 			  struct v4l2_subdev_format *fmt)
@@ -2217,12 +2418,25 @@ static int ccs_set_format(struct v4l2_subdev *subdev,
 	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
 	struct v4l2_rect *crops[CCS_PADS];
 
+	if (subdev == &sensor->src->sd && fmt->pad == CCS_PAD_META)
+		return ccs_get_format(subdev, sd_state, fmt);
+
 	mutex_lock(&sensor->mutex);
 
+	if (subdev == &sensor->src->sd && fmt->stream == CCS_STREAM_META) {
+		ccs_set_format_meta(subdev, sd_state, &fmt->format);
+
+		mutex_unlock(&sensor->mutex);
+
+		return 0;
+	}
+
 	if (fmt->pad == ssd->source_pad) {
 		int rval;
 
 		rval = ccs_set_format_source(subdev, sd_state, fmt);
+		if (sensor->embedded_start != sensor->embedded_end)
+			ccs_set_format_meta(subdev, sd_state, NULL);
 
 		mutex_unlock(&sensor->mutex);
 
@@ -2497,6 +2711,12 @@ static int ccs_sel_supported(struct v4l2_subdev *subdev,
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
 	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
 
+	if (sel->stream != CCS_STREAM_PIXEL)
+		return -EINVAL;
+
+	if (sel->pad == CCS_PAD_META)
+		return -EINVAL;
+
 	/* We only implement crop in three places. */
 	switch (sel->target) {
 	case V4L2_SEL_TGT_CROP:
@@ -2541,7 +2761,8 @@ static int ccs_set_crop(struct v4l2_subdev *subdev,
 
 	if (sel->pad == ssd->sink_pad) {
 		struct v4l2_mbus_framefmt *mfmt =
-			v4l2_subdev_state_get_format(sd_state, sel->pad);
+			v4l2_subdev_state_get_format(sd_state, sel->pad,
+						     CCS_STREAM_PIXEL);
 
 		src_size.width = mfmt->width;
 		src_size.height = mfmt->height;
@@ -2602,7 +2823,9 @@ static int ccs_get_selection(struct v4l2_subdev *subdev,
 		} else if (sel->pad == ssd->sink_pad) {
 			struct v4l2_mbus_framefmt *sink_fmt =
 				v4l2_subdev_state_get_format(sd_state,
-							     ssd->sink_pad);
+							     ssd->sink_pad,
+							     CCS_STREAM_PIXEL);
+
 			sel->r.top = sel->r.left = 0;
 			sel->r.width = sink_fmt->width;
 			sel->r.height = sink_fmt->height;
@@ -2688,6 +2911,14 @@ static int ccs_get_frame_desc(struct v4l2_subdev *subdev, unsigned int pad,
 	entry++;
 	desc->num_entries++;
 
+	if (sensor->embedded_start != sensor->embedded_end) {
+		entry->pixelcode = MEDIA_BUS_FMT_CCS_EMBEDDED;
+		entry->stream = CCS_STREAM_META;
+		entry->bus.csi2.dt = MIPI_CSI2_DT_EMBEDDED_8B;
+		entry++;
+		desc->num_entries++;
+	}
+
 	return 0;
 }
 
@@ -3006,6 +3237,8 @@ static void ccs_cleanup(struct ccs_sensor *sensor)
 	ccs_free_controls(sensor);
 }
 
+static const struct v4l2_subdev_internal_ops ccs_internal_ops;
+
 static int ccs_init_subdev(struct ccs_sensor *sensor,
 			   struct ccs_subdev *ssd, const char *name,
 			   unsigned short num_pads, u32 function,
@@ -3018,15 +3251,18 @@ static int ccs_init_subdev(struct ccs_sensor *sensor,
 	if (!ssd)
 		return 0;
 
-	if (ssd != sensor->src)
+	if (ssd != sensor->src) {
 		v4l2_subdev_init(&ssd->sd, &ccs_ops);
+		ssd->sd.internal_ops = &ccs_internal_ops;
+	}
 
 	ssd->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
 	ssd->sd.entity.function = function;
 	ssd->sensor = sensor;
 
 	ssd->npads = num_pads;
-	ssd->source_pad = num_pads - 1;
+	ssd->source_pad =
+		ssd == sensor->pixel_array ? CCS_PA_PAD_SRC : CCS_PAD_SRC;
 
 	v4l2_i2c_subdev_set_name(&ssd->sd, client, sensor->minfo.name, name);
 
@@ -3040,6 +3276,10 @@ static int ccs_init_subdev(struct ccs_sensor *sensor,
 		ssd->sd.owner = THIS_MODULE;
 		ssd->sd.dev = &client->dev;
 		v4l2_set_subdevdata(&ssd->sd, client);
+	} else {
+		ssd->sd.flags |= V4L2_SUBDEV_FL_STREAMS;
+		ssd->pads[CCS_PAD_META].flags =
+			MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL;
 	}
 
 	rval = media_entity_pads_init(&ssd->sd.entity, ssd->npads, ssd->pads);
@@ -3057,21 +3297,19 @@ static int ccs_init_subdev(struct ccs_sensor *sensor,
 	return 0;
 }
 
-static int ccs_init_state(struct v4l2_subdev *sd,
-			  struct v4l2_subdev_state *sd_state)
+static int __ccs_init_state(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_state *sd_state)
 {
 	struct ccs_subdev *ssd = to_ccs_subdev(sd);
 	struct ccs_sensor *sensor = ssd->sensor;
 	unsigned int pad = ssd == sensor->pixel_array ?
 		CCS_PA_PAD_SRC : CCS_PAD_SINK;
 	struct v4l2_mbus_framefmt *fmt =
-		v4l2_subdev_state_get_format(sd_state, pad);
+		v4l2_subdev_state_get_format(sd_state, pad, CCS_STREAM_PIXEL);
 	struct v4l2_rect *crop =
-		v4l2_subdev_state_get_crop(sd_state, pad);
+		v4l2_subdev_state_get_crop(sd_state, pad, CCS_STREAM_PIXEL);
 	bool is_active = !sd->active_state || sd->active_state == sd_state;
 
-	mutex_lock(&sensor->mutex);
-
 	ccs_get_native_size(ssd, crop);
 
 	fmt->width = crop->width;
@@ -3083,20 +3321,78 @@ static int ccs_init_state(struct v4l2_subdev *sd,
 		if (is_active)
 			sensor->pa_src = *crop;
 
-		mutex_unlock(&sensor->mutex);
 		return 0;
 	}
 
-	fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC);
+	fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC,
+					   CCS_STREAM_PIXEL);
 	fmt->code = ssd == sensor->src ?
 		sensor->csi_format->code : sensor->internal_csi_format->code;
 	fmt->field = V4L2_FIELD_NONE;
 
 	ccs_propagate(sd, sd_state, is_active, V4L2_SEL_TGT_CROP);
 
+	return 0;
+}
+
+static int ccs_init_state(struct v4l2_subdev *sd,
+			  struct v4l2_subdev_state *sd_state)
+{
+	struct ccs_subdev *ssd = to_ccs_subdev(sd);
+	struct ccs_sensor *sensor = ssd->sensor;
+	int rval;
+
+	mutex_lock(&sensor->mutex);
+	rval = __ccs_init_state(sd, sd_state);
 	mutex_unlock(&sensor->mutex);
 
-	return 0;
+	return rval;
+}
+
+static int ccs_src_init_state(struct v4l2_subdev *sd,
+			      struct v4l2_subdev_state *sd_state)
+{
+	struct v4l2_subdev_route routes[] = {
+		{
+			.sink_pad = CCS_PAD_SINK,
+			.source_pad = CCS_PAD_SRC,
+			.source_stream = CCS_STREAM_PIXEL,
+			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
+		}, {
+			.sink_pad = CCS_PAD_META,
+			.source_pad = CCS_PAD_SRC,
+			.source_stream = CCS_STREAM_META,
+			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
+		}
+	};
+	struct v4l2_subdev_krouting routing = {
+		.routes = routes,
+		.num_routes = 1,
+	};
+	struct ccs_subdev *ssd = to_ccs_subdev(sd);
+	struct ccs_sensor *sensor = ssd->sensor;
+	int rval;
+
+	mutex_lock(&sensor->mutex);
+
+	if (sensor->embedded_start != sensor->embedded_end)
+		routing.num_routes++;
+
+	rval = v4l2_subdev_set_routing(sd, sd_state, &routing);
+	if (rval)
+		goto out;
+
+	rval = __ccs_init_state(sd, sd_state);
+	if (rval)
+		goto out;
+
+	if (sensor->embedded_start != sensor->embedded_end)
+		ccs_set_format_meta(sd, sd_state, NULL);
+
+out:
+	mutex_unlock(&sensor->mutex);
+
+	return rval;
 }
 
 static const struct v4l2_subdev_video_ops ccs_video_ops = {
@@ -3111,6 +3407,14 @@ static const struct v4l2_subdev_pad_ops ccs_pad_ops = {
 	.set_fmt = ccs_set_format,
 	.get_selection = ccs_get_selection,
 	.set_selection = ccs_set_selection,
+};
+
+static const struct v4l2_subdev_pad_ops ccs_src_pad_ops = {
+	.enum_mbus_code = ccs_enum_mbus_code,
+	.get_fmt = ccs_get_format,
+	.set_fmt = ccs_set_format,
+	.get_selection = ccs_get_selection,
+	.set_selection = ccs_set_selection,
 	.enable_streams = ccs_enable_streams,
 	.disable_streams = ccs_disable_streams,
 	.get_frame_desc = ccs_get_frame_desc,
@@ -3127,12 +3431,22 @@ static const struct v4l2_subdev_ops ccs_ops = {
 	.sensor = &ccs_sensor_ops,
 };
 
+static const struct v4l2_subdev_ops ccs_src_ops = {
+	.video = &ccs_video_ops,
+	.pad = &ccs_src_pad_ops,
+	.sensor = &ccs_sensor_ops,
+};
+
 static const struct media_entity_operations ccs_entity_ops = {
 	.link_validate = v4l2_subdev_link_validate,
 };
 
-static const struct v4l2_subdev_internal_ops ccs_internal_src_ops = {
+static const struct v4l2_subdev_internal_ops ccs_internal_ops = {
 	.init_state = ccs_init_state,
+};
+
+static const struct v4l2_subdev_internal_ops ccs_src_internal_ops = {
+	.init_state = ccs_src_init_state,
 	.registered = ccs_registered,
 	.unregistered = ccs_unregistered,
 };
@@ -3283,8 +3597,8 @@ static int ccs_probe(struct i2c_client *client)
 
 	sensor->src = &sensor->ssds[sensor->ssds_used];
 
-	v4l2_i2c_subdev_init(&sensor->src->sd, client, &ccs_ops);
-	sensor->src->sd.internal_ops = &ccs_internal_src_ops;
+	v4l2_i2c_subdev_init(&sensor->src->sd, client, &ccs_src_ops);
+	sensor->src->sd.internal_ops = &ccs_src_internal_ops;
 
 	sensor->regulators = devm_kcalloc(&client->dev,
 					  ARRAY_SIZE(ccs_regulators),
@@ -3555,12 +3869,20 @@ static int ccs_probe(struct i2c_client *client)
 		goto out_cleanup;
 	}
 
-	rval = ccs_init_subdev(sensor, sensor->scaler, " scaler", 2,
+	rval = ccs_init_subdev(sensor, sensor->scaler, " scaler",
+			       sensor->ssds_used != CCS_SUBDEVS ?
+			       CCS_PADS_NOMETA :
+			       sensor->embedded_start == sensor->embedded_end ?
+			       CCS_PADS_NOMETA : CCS_PADS,
 			       MEDIA_ENT_F_PROC_VIDEO_SCALER,
 			       "ccs scaler mutex", &scaler_lock_key);
 	if (rval)
 		goto out_cleanup;
-	rval = ccs_init_subdev(sensor, sensor->binner, " binner", 2,
+	rval = ccs_init_subdev(sensor, sensor->binner, " binner",
+			       sensor->ssds_used == CCS_SUBDEVS ?
+			       CCS_PADS_NOMETA :
+			       sensor->embedded_start == sensor->embedded_end ?
+			       CCS_PADS_NOMETA : CCS_PADS,
 			       MEDIA_ENT_F_PROC_VIDEO_SCALER,
 			       "ccs binner mutex", &binner_lock_key);
 	if (rval)
diff --git a/drivers/media/i2c/ccs/ccs.h b/drivers/media/i2c/ccs/ccs.h
index adb152366ea2..207d33de3eee 100644
--- a/drivers/media/i2c/ccs/ccs.h
+++ b/drivers/media/i2c/ccs/ccs.h
@@ -173,11 +173,18 @@ struct ccs_csi_data_format {
 #define CCS_SUBDEVS			3
 
 #define CCS_PA_PAD_SRC			0
-#define CCS_PAD_SINK			0
-#define CCS_PAD_SRC			1
-#define CCS_PADS			2
+enum {
+	CCS_PAD_SINK,
+	CCS_PAD_SRC,
+	CCS_PAD_META,
+	CCS_PADS_NOMETA = CCS_PAD_META,
+	CCS_PADS,
+};
 
-#define CCS_STREAM_PIXEL		0
+enum {
+	CCS_STREAM_PIXEL,
+	CCS_STREAM_META,
+};
 
 struct ccs_binning_subtype {
 	u8 horizontal:4;
@@ -228,6 +235,9 @@ struct ccs_sensor {
 	int default_pixel_order;
 	struct ccs_data_container sdata, mdata;
 
+	u32 embedded_mbus_code;
+	u8 emb_data_ctrl;
+
 	u8 binning_horizontal;
 	u8 binning_vertical;
 
-- 
2.39.2

[PATCH v8 25/38] media: ccs: Remove ccs_get_crop_compose helper

[Sakari Ailus]

As it's now easier to obtain the necessary information on crop and compose
rectangles after moving to sub-device state, remove the
ccs_get_crop_compose helper.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ccs/ccs-core.c | 130 ++++++++++++++-----------------
 1 file changed, 58 insertions(+), 72 deletions(-)

diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
index 1f91af25eac6..6c7eb4f04070 100644
--- a/drivers/media/i2c/ccs/ccs-core.c
+++ b/drivers/media/i2c/ccs/ccs-core.c
@@ -2222,24 +2222,6 @@ static int ccs_get_format(struct v4l2_subdev *subdev,
 	return rval;
 }
 
-static void ccs_get_crop_compose(struct v4l2_subdev *subdev,
-				 struct v4l2_subdev_state *sd_state,
-				 struct v4l2_rect **crops,
-				 struct v4l2_rect **comps)
-{
-	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
-	unsigned int i;
-
-	if (crops)
-		for (i = 0; i < subdev->entity.num_pads; i++)
-			crops[i] =
-				v4l2_subdev_state_get_crop(sd_state, i,
-							   CCS_STREAM_PIXEL);
-	if (comps)
-		*comps = v4l2_subdev_state_get_compose(sd_state, ssd->sink_pad,
-						       CCS_STREAM_PIXEL);
-}
-
 /* Changes require propagation only on sink pad. */
 static void ccs_propagate(struct v4l2_subdev *subdev,
 			  struct v4l2_subdev_state *sd_state, int which,
@@ -2247,15 +2229,17 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
 {
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
 	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
-	struct v4l2_rect *comp, *crops[CCS_PADS];
+	struct v4l2_rect *comp, *crop;
 	struct v4l2_mbus_framefmt *fmt;
 
-	ccs_get_crop_compose(subdev, sd_state, crops, &comp);
-
+	comp = v4l2_subdev_state_get_compose(sd_state, ssd->sink_pad,
+					     CCS_STREAM_PIXEL);
 	switch (target) {
 	case V4L2_SEL_TGT_CROP:
-		comp->width = crops[CCS_PAD_SINK]->width;
-		comp->height = crops[CCS_PAD_SINK]->height;
+		crop = v4l2_subdev_state_get_crop(sd_state, CCS_PAD_SINK,
+						  CCS_STREAM_PIXEL);
+		comp->width = crop->width;
+		comp->height = crop->height;
 		if (which == V4L2_SUBDEV_FORMAT_ACTIVE) {
 			if (ssd == sensor->scaler) {
 				sensor->scale_m = CCS_LIM(sensor, SCALER_N_MIN);
@@ -2269,13 +2253,15 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
 		}
 		fallthrough;
 	case V4L2_SEL_TGT_COMPOSE:
-		*crops[CCS_PAD_SRC] = *comp;
+		crop = v4l2_subdev_state_get_crop(sd_state, CCS_PAD_SRC,
+						  CCS_STREAM_PIXEL);
+		*crop = *comp;
 		fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC,
 						   CCS_STREAM_PIXEL);
 		fmt->width = comp->width;
 		fmt->height = comp->height;
 		if (which == V4L2_SUBDEV_FORMAT_ACTIVE && ssd == sensor->src)
-			sensor->src_src = *crops[CCS_PAD_SRC];
+			sensor->src_src = *crop;
 		break;
 	default:
 		WARN_ON_ONCE(1);
@@ -2416,7 +2402,7 @@ static int ccs_set_format(struct v4l2_subdev *subdev,
 {
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
 	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
-	struct v4l2_rect *crops[CCS_PADS];
+	struct v4l2_rect *crop;
 
 	if (subdev == &sensor->src->sd && fmt->pad == CCS_PAD_META)
 		return ccs_get_format(subdev, sd_state, fmt);
@@ -2458,12 +2444,13 @@ static int ccs_set_format(struct v4l2_subdev *subdev,
 		      CCS_LIM(sensor, MIN_Y_OUTPUT_SIZE),
 		      CCS_LIM(sensor, MAX_Y_OUTPUT_SIZE));
 
-	ccs_get_crop_compose(subdev, sd_state, crops, NULL);
+	crop = v4l2_subdev_state_get_crop(sd_state, ssd->sink_pad,
+					  CCS_STREAM_PIXEL);
 
-	crops[ssd->sink_pad]->left = 0;
-	crops[ssd->sink_pad]->top = 0;
-	crops[ssd->sink_pad]->width = fmt->format.width;
-	crops[ssd->sink_pad]->height = fmt->format.height;
+	crop->left = 0;
+	crop->top = 0;
+	crop->width = fmt->format.width;
+	crop->height = fmt->format.height;
 	ccs_propagate(subdev, sd_state, fmt->which, V4L2_SEL_TGT_CROP);
 
 	mutex_unlock(&sensor->mutex);
@@ -2518,24 +2505,23 @@ static int scaling_goodness(struct v4l2_subdev *subdev, int w, int ask_w,
 static void ccs_set_compose_binner(struct v4l2_subdev *subdev,
 				   struct v4l2_subdev_state *sd_state,
 				   struct v4l2_subdev_selection *sel,
-				   struct v4l2_rect **crops,
+				   struct v4l2_rect *sink_crop,
 				   struct v4l2_rect *comp)
 {
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
 	unsigned int i;
 	unsigned int binh = 1, binv = 1;
 	int best = scaling_goodness(
-		subdev,
-		crops[CCS_PAD_SINK]->width, sel->r.width,
-		crops[CCS_PAD_SINK]->height, sel->r.height, sel->flags);
+		subdev,	sink_crop->width, sel->r.width,
+		sink_crop->height, sel->r.height, sel->flags);
 
 	for (i = 0; i < sensor->nbinning_subtypes; i++) {
 		int this = scaling_goodness(
 			subdev,
-			crops[CCS_PAD_SINK]->width
+			sink_crop->width
 			/ sensor->binning_subtypes[i].horizontal,
 			sel->r.width,
-			crops[CCS_PAD_SINK]->height
+			sink_crop->height
 			/ sensor->binning_subtypes[i].vertical,
 			sel->r.height, sel->flags);
 
@@ -2550,8 +2536,8 @@ static void ccs_set_compose_binner(struct v4l2_subdev *subdev,
 		sensor->binning_horizontal = binh;
 	}
 
-	sel->r.width = (crops[CCS_PAD_SINK]->width / binh) & ~1;
-	sel->r.height = (crops[CCS_PAD_SINK]->height / binv) & ~1;
+	sel->r.width = (sink_crop->width / binh) & ~1;
+	sel->r.height = (sink_crop->height / binv) & ~1;
 }
 
 /*
@@ -2566,7 +2552,7 @@ static void ccs_set_compose_binner(struct v4l2_subdev *subdev,
 static void ccs_set_compose_scaler(struct v4l2_subdev *subdev,
 				   struct v4l2_subdev_state *sd_state,
 				   struct v4l2_subdev_selection *sel,
-				   struct v4l2_rect **crops,
+				   struct v4l2_rect *sink_crop,
 				   struct v4l2_rect *comp)
 {
 	struct i2c_client *client = v4l2_get_subdevdata(subdev);
@@ -2579,17 +2565,12 @@ static void ccs_set_compose_scaler(struct v4l2_subdev *subdev,
 	unsigned int i;
 	int best = INT_MIN;
 
-	sel->r.width = min_t(unsigned int, sel->r.width,
-			     crops[CCS_PAD_SINK]->width);
-	sel->r.height = min_t(unsigned int, sel->r.height,
-			      crops[CCS_PAD_SINK]->height);
-
-	a = crops[CCS_PAD_SINK]->width
-		* CCS_LIM(sensor, SCALER_N_MIN) / sel->r.width;
-	b = crops[CCS_PAD_SINK]->height
-		* CCS_LIM(sensor, SCALER_N_MIN) / sel->r.height;
-	max_m = crops[CCS_PAD_SINK]->width
-		* CCS_LIM(sensor, SCALER_N_MIN)
+	sel->r.width = min_t(unsigned int, sel->r.width, sink_crop->width);
+	sel->r.height = min_t(unsigned int, sel->r.height, sink_crop->height);
+
+	a = sink_crop->width * CCS_LIM(sensor, SCALER_N_MIN) / sel->r.width;
+	b = sink_crop->height * CCS_LIM(sensor, SCALER_N_MIN) / sel->r.height;
+	max_m = sink_crop->width * CCS_LIM(sensor, SCALER_N_MIN)
 		/ CCS_LIM(sensor, MIN_X_OUTPUT_SIZE);
 
 	a = clamp(a, CCS_LIM(sensor, SCALER_M_MIN),
@@ -2622,10 +2603,10 @@ static void ccs_set_compose_scaler(struct v4l2_subdev *subdev,
 	for (i = 0; i < ntry; i++) {
 		int this = scaling_goodness(
 			subdev,
-			crops[CCS_PAD_SINK]->width
+			sink_crop->width
 			/ try[i] * CCS_LIM(sensor, SCALER_N_MIN),
 			sel->r.width,
-			crops[CCS_PAD_SINK]->height,
+			sink_crop->height,
 			sel->r.height,
 			sel->flags);
 
@@ -2642,12 +2623,10 @@ static void ccs_set_compose_scaler(struct v4l2_subdev *subdev,
 			continue;
 
 		this = scaling_goodness(
-			subdev, crops[CCS_PAD_SINK]->width
-			/ try[i]
+			subdev, sink_crop->width / try[i]
 			* CCS_LIM(sensor, SCALER_N_MIN),
 			sel->r.width,
-			crops[CCS_PAD_SINK]->height
-			/ try[i]
+			sink_crop->height / try[i]
 			* CCS_LIM(sensor, SCALER_N_MIN),
 			sel->r.height,
 			sel->flags);
@@ -2660,17 +2639,15 @@ static void ccs_set_compose_scaler(struct v4l2_subdev *subdev,
 	}
 
 	sel->r.width =
-		(crops[CCS_PAD_SINK]->width
-		 / scale_m
+		(sink_crop->width / scale_m
 		 * CCS_LIM(sensor, SCALER_N_MIN)) & ~1;
 	if (mode == SMIAPP_SCALING_MODE_BOTH)
 		sel->r.height =
-			(crops[CCS_PAD_SINK]->height
-			 / scale_m
+			(sink_crop->height / scale_m
 			 * CCS_LIM(sensor, SCALER_N_MIN))
 			& ~1;
 	else
-		sel->r.height = crops[CCS_PAD_SINK]->height;
+		sel->r.height = sink_crop->height;
 
 	if (sel->which == V4L2_SUBDEV_FORMAT_ACTIVE) {
 		sensor->scale_m = scale_m;
@@ -2684,17 +2661,20 @@ static int ccs_set_compose(struct v4l2_subdev *subdev,
 {
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
 	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
-	struct v4l2_rect *comp, *crops[CCS_PADS];
+	struct v4l2_rect *comp, *sink_crop;
 
-	ccs_get_crop_compose(subdev, sd_state, crops, &comp);
+	sink_crop = v4l2_subdev_state_get_crop(sd_state, CCS_PAD_SINK,
+					       CCS_STREAM_PIXEL);
+	comp = v4l2_subdev_state_get_compose(sd_state, ssd->sink_pad,
+					     CCS_STREAM_PIXEL);
 
 	sel->r.top = 0;
 	sel->r.left = 0;
 
 	if (ssd == sensor->binner)
-		ccs_set_compose_binner(subdev, sd_state, sel, crops, comp);
+		ccs_set_compose_binner(subdev, sd_state, sel, sink_crop, comp);
 	else
-		ccs_set_compose_scaler(subdev, sd_state, sel, crops, comp);
+		ccs_set_compose_scaler(subdev, sd_state, sel, sink_crop, comp);
 
 	*comp = sel->r;
 	ccs_propagate(subdev, sd_state, sel->which, V4L2_SEL_TGT_COMPOSE);
@@ -2755,9 +2735,12 @@ static int ccs_set_crop(struct v4l2_subdev *subdev,
 {
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
 	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
-	struct v4l2_rect src_size = { 0 }, *crops[CCS_PADS], *comp;
+	struct v4l2_rect src_size = { 0 }, *crop, *comp;
 
-	ccs_get_crop_compose(subdev, sd_state, crops, &comp);
+	crop = v4l2_subdev_state_get_crop(sd_state, sel->pad,
+					  CCS_STREAM_PIXEL);
+	comp = v4l2_subdev_state_get_compose(sd_state, ssd->sink_pad,
+					     CCS_STREAM_PIXEL);
 
 	if (sel->pad == ssd->sink_pad) {
 		struct v4l2_mbus_framefmt *mfmt =
@@ -2781,7 +2764,7 @@ static int ccs_set_crop(struct v4l2_subdev *subdev,
 	sel->r.left = min_t(int, sel->r.left, src_size.width - sel->r.width);
 	sel->r.top = min_t(int, sel->r.top, src_size.height - sel->r.height);
 
-	*crops[sel->pad] = sel->r;
+	*crop = sel->r;
 
 	if (ssd != sensor->pixel_array && sel->pad == CCS_PAD_SINK)
 		ccs_propagate(subdev, sd_state, sel->which, V4L2_SEL_TGT_CROP);
@@ -2806,14 +2789,17 @@ static int ccs_get_selection(struct v4l2_subdev *subdev,
 {
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
 	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
-	struct v4l2_rect *comp, *crops[CCS_PADS];
+	struct v4l2_rect *crop, *comp;
 	int ret;
 
 	ret = ccs_sel_supported(subdev, sel);
 	if (ret)
 		return ret;
 
-	ccs_get_crop_compose(subdev, sd_state, crops, &comp);
+	crop = v4l2_subdev_state_get_crop(sd_state, sel->pad,
+					  CCS_STREAM_PIXEL);
+	comp = v4l2_subdev_state_get_compose(sd_state, ssd->sink_pad,
+					     CCS_STREAM_PIXEL);
 
 	switch (sel->target) {
 	case V4L2_SEL_TGT_CROP_BOUNDS:
@@ -2835,7 +2821,7 @@ static int ccs_get_selection(struct v4l2_subdev *subdev,
 		break;
 	case V4L2_SEL_TGT_CROP:
 	case V4L2_SEL_TGT_COMPOSE_BOUNDS:
-		sel->r = *crops[sel->pad];
+		sel->r = *crop;
 		break;
 	case V4L2_SEL_TGT_COMPOSE:
 		sel->r = *comp;
-- 
2.39.2

[PATCH v8 26/38] media: ccs: Rely on sub-device state locking

[Sakari Ailus]

Rely on sub-device state locking to serialise access to driver's data
structures. The driver-provided mutex is used as the state lock for all
driver sub-devices.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ccs/ccs-core.c | 192 +++++++++++++------------------
 drivers/media/i2c/ccs/ccs.h      |   1 -
 2 files changed, 81 insertions(+), 112 deletions(-)

diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
index 6c7eb4f04070..a8c48abd2e30 100644
--- a/drivers/media/i2c/ccs/ccs-core.c
+++ b/drivers/media/i2c/ccs/ccs-core.c
@@ -541,12 +541,13 @@ static int ccs_pll_update(struct ccs_sensor *sensor)
  *
  */
 
-static void __ccs_update_exposure_limits(struct ccs_sensor *sensor)
+static void __ccs_update_exposure_limits(struct ccs_sensor *sensor,
+					 struct v4l2_rect *pa_src)
 {
 	struct v4l2_ctrl *ctrl = sensor->exposure;
 	int max;
 
-	max = sensor->pa_src.height + sensor->vblank->val -
+	max = pa_src->height + sensor->vblank->val -
 		CCS_LIM(sensor, COARSE_INTEGRATION_TIME_MAX_MARGIN);
 
 	__v4l2_ctrl_modify_range(ctrl, ctrl->minimum, max, ctrl->step, max);
@@ -649,12 +650,20 @@ static int ccs_set_ctrl(struct v4l2_ctrl *ctrl)
 		container_of(ctrl->handler, struct ccs_subdev, ctrl_handler)
 			->sensor;
 	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
+	struct v4l2_subdev_state *state;
+	struct v4l2_rect *pa_src;
 	int pm_status;
 	u32 orient = 0;
 	unsigned int i;
 	int exposure;
 	int rval;
 
+	if (ctrl->id == V4L2_CID_VBLANK || ctrl->id == V4L2_CID_HBLANK) {
+		state = v4l2_subdev_get_locked_active_state(&sensor->pixel_array->sd);
+		pa_src = v4l2_subdev_state_get_crop(state, CCS_PA_PAD_SRC,
+						    CCS_STREAM_PIXEL);
+	}
+
 	switch (ctrl->id) {
 	case V4L2_CID_HFLIP:
 	case V4L2_CID_VFLIP:
@@ -673,7 +682,7 @@ static int ccs_set_ctrl(struct v4l2_ctrl *ctrl)
 	case V4L2_CID_VBLANK:
 		exposure = sensor->exposure->val;
 
-		__ccs_update_exposure_limits(sensor);
+		__ccs_update_exposure_limits(sensor, pa_src);
 
 		if (exposure > sensor->exposure->maximum) {
 			sensor->exposure->val =	sensor->exposure->maximum;
@@ -765,12 +774,12 @@ static int ccs_set_ctrl(struct v4l2_ctrl *ctrl)
 		break;
 	case V4L2_CID_VBLANK:
 		rval = ccs_write(sensor, FRAME_LENGTH_LINES,
-				 sensor->pa_src.height + ctrl->val);
+				 pa_src->height + ctrl->val);
 
 		break;
 	case V4L2_CID_HBLANK:
 		rval = ccs_write(sensor, LINE_LENGTH_PCK,
-				 sensor->pa_src.width + ctrl->val);
+				 pa_src->width + ctrl->val);
 
 		break;
 	case V4L2_CID_TEST_PATTERN:
@@ -1225,7 +1234,8 @@ static int ccs_get_mbus_formats(struct ccs_sensor *sensor)
 	return 0;
 }
 
-static void ccs_update_blanking(struct ccs_sensor *sensor)
+static void ccs_update_blanking(struct ccs_sensor *sensor,
+				struct v4l2_rect *pa_src)
 {
 	struct v4l2_ctrl *vblank = sensor->vblank;
 	struct v4l2_ctrl *hblank = sensor->hblank;
@@ -1248,21 +1258,26 @@ static void ccs_update_blanking(struct ccs_sensor *sensor)
 
 	min = max_t(int,
 		    CCS_LIM(sensor, MIN_FRAME_BLANKING_LINES),
-		    min_fll - sensor->pa_src.height);
-	max = max_fll -	sensor->pa_src.height;
+		    min_fll - pa_src->height);
+	max = max_fll -	pa_src->height;
 
 	__v4l2_ctrl_modify_range(vblank, min, max, vblank->step, min);
 
-	min = max_t(int, min_llp - sensor->pa_src.width, min_lbp);
-	max = max_llp - sensor->pa_src.width;
+	min = max_t(int, min_llp - pa_src->width, min_lbp);
+	max = max_llp - pa_src->width;
 
 	__v4l2_ctrl_modify_range(hblank, min, max, hblank->step, min);
 
-	__ccs_update_exposure_limits(sensor);
+	__ccs_update_exposure_limits(sensor, pa_src);
 }
 
 static int ccs_pll_blanking_update(struct ccs_sensor *sensor)
 {
+	struct v4l2_subdev_state *state =
+		v4l2_subdev_get_locked_active_state(&sensor->pixel_array->sd);
+	struct v4l2_rect *pa_src =
+		v4l2_subdev_state_get_crop(state, CCS_PA_PAD_SRC,
+					   CCS_STREAM_PIXEL);
 	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
 	int rval;
 
@@ -1271,15 +1286,15 @@ static int ccs_pll_blanking_update(struct ccs_sensor *sensor)
 		return rval;
 
 	/* Output from pixel array, including blanking */
-	ccs_update_blanking(sensor);
+	ccs_update_blanking(sensor, pa_src);
 
 	dev_dbg(&client->dev, "vblank\t\t%d\n", sensor->vblank->val);
 	dev_dbg(&client->dev, "hblank\t\t%d\n", sensor->hblank->val);
 
 	dev_dbg(&client->dev, "real timeperframe\t100/%d\n",
 		sensor->pll.pixel_rate_pixel_array /
-		((sensor->pa_src.width + sensor->hblank->val) *
-		 (sensor->pa_src.height + sensor->vblank->val) / 100));
+		((pa_src->width + sensor->hblank->val) *
+		 (pa_src->height + sensor->vblank->val) / 100));
 
 	return 0;
 }
@@ -1787,6 +1802,16 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
 			      u64 streams_mask)
 {
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
+	struct v4l2_subdev_state *pa_state =
+		v4l2_subdev_get_locked_active_state(&sensor->pixel_array->sd);
+	struct v4l2_subdev_state *src_state =
+		v4l2_subdev_get_locked_active_state(&sensor->src->sd);
+	struct v4l2_rect *pa_src =
+		v4l2_subdev_state_get_crop(pa_state, CCS_PA_PAD_SRC,
+					   CCS_STREAM_PIXEL);
+	struct v4l2_rect *src_src =
+		v4l2_subdev_state_get_crop(src_state, CCS_PAD_SRC,
+					   CCS_STREAM_PIXEL);
 	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
 	unsigned int binning_mode;
 	int rval;
@@ -1834,22 +1859,20 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
 		goto err_pm_put;
 
 	/* Analog crop start coordinates */
-	rval = ccs_write(sensor, X_ADDR_START, sensor->pa_src.left);
+	rval = ccs_write(sensor, X_ADDR_START, pa_src->left);
 	if (rval < 0)
 		goto err_pm_put;
 
-	rval = ccs_write(sensor, Y_ADDR_START, sensor->pa_src.top);
+	rval = ccs_write(sensor, Y_ADDR_START, pa_src->top);
 	if (rval < 0)
 		goto err_pm_put;
 
 	/* Analog crop end coordinates */
-	rval = ccs_write(sensor, X_ADDR_END,
-			 sensor->pa_src.left + sensor->pa_src.width - 1);
+	rval = ccs_write(sensor, X_ADDR_END, pa_src->left + pa_src->width - 1);
 	if (rval < 0)
 		goto err_pm_put;
 
-	rval = ccs_write(sensor, Y_ADDR_END,
-			 sensor->pa_src.top + sensor->pa_src.height - 1);
+	rval = ccs_write(sensor, Y_ADDR_END, pa_src->top + pa_src->height - 1);
 	if (rval < 0)
 		goto err_pm_put;
 
@@ -1861,23 +1884,30 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
 	/* Digital crop */
 	if (CCS_LIM(sensor, DIGITAL_CROP_CAPABILITY)
 	    == CCS_DIGITAL_CROP_CAPABILITY_INPUT_CROP) {
+		struct v4l2_subdev_state *scaler_state =
+			v4l2_subdev_get_locked_active_state(&sensor->scaler->sd);
+		struct v4l2_rect *scaler_sink =
+			v4l2_subdev_state_get_crop(scaler_state,
+						   sensor->scaler->sink_pad,
+						   CCS_STREAM_PIXEL);
+
 		rval = ccs_write(sensor, DIGITAL_CROP_X_OFFSET,
-				 sensor->scaler_sink.left);
+				 scaler_sink->left);
 		if (rval < 0)
 			goto err_pm_put;
 
 		rval = ccs_write(sensor, DIGITAL_CROP_Y_OFFSET,
-				 sensor->scaler_sink.top);
+				 scaler_sink->top);
 		if (rval < 0)
 			goto err_pm_put;
 
 		rval = ccs_write(sensor, DIGITAL_CROP_IMAGE_WIDTH,
-				 sensor->scaler_sink.width);
+				 scaler_sink->width);
 		if (rval < 0)
 			goto err_pm_put;
 
 		rval = ccs_write(sensor, DIGITAL_CROP_IMAGE_HEIGHT,
-				 sensor->scaler_sink.height);
+				 scaler_sink->height);
 		if (rval < 0)
 			goto err_pm_put;
 	}
@@ -1895,10 +1925,10 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
 	}
 
 	/* Output size from sensor */
-	rval = ccs_write(sensor, X_OUTPUT_SIZE, sensor->src_src.width);
+	rval = ccs_write(sensor, X_OUTPUT_SIZE, src_src->width);
 	if (rval < 0)
 		goto err_pm_put;
-	rval = ccs_write(sensor, Y_OUTPUT_SIZE, sensor->src_src.height);
+	rval = ccs_write(sensor, Y_OUTPUT_SIZE, src_src->height);
 	if (rval < 0)
 		goto err_pm_put;
 
@@ -2087,9 +2117,6 @@ static int ccs_enum_mbus_code(struct v4l2_subdev *subdev,
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
 	unsigned int i;
 	int idx = -1;
-	int rval = -EINVAL;
-
-	mutex_lock(&sensor->mutex);
 
 	dev_err(&client->dev, "subdev %s, pad %u, index %u\n",
 		subdev->name, code->pad, code->index);
@@ -2097,12 +2124,11 @@ static int ccs_enum_mbus_code(struct v4l2_subdev *subdev,
 	if (subdev == &sensor->src->sd) {
 		if (code->pad == CCS_PAD_META) {
 			if (code->index)
-				goto out;
+				return -EINVAL;
 
 			code->code = MEDIA_BUS_FMT_CCS_EMBEDDED;
 
-			rval = 0;
-			goto out;
+			return 0;
 		}
 		if (code->stream == CCS_STREAM_META) {
 			struct v4l2_mbus_framefmt *pix_fmt =
@@ -2149,21 +2175,21 @@ static int ccs_enum_mbus_code(struct v4l2_subdev *subdev,
 			}
 
 			if (WARN_ON(i > ARRAY_SIZE(codes)) || code->index >= i)
-				goto out;
+				return -EINVAL;
 
 			code->code = codes[code->index];
-			rval = 0;
-			goto out;
+
+			return 0;
 		}
 	}
 
 	if (subdev != &sensor->src->sd || code->pad != CCS_PAD_SRC) {
 		if (code->index)
-			goto out;
+			return -EINVAL;
 
 		code->code = sensor->internal_csi_format->code;
-		rval = 0;
-		goto out;
+
+		return 0;
 	}
 
 	for (i = 0; i < ARRAY_SIZE(ccs_csi_data_formats); i++) {
@@ -2174,18 +2200,14 @@ static int ccs_enum_mbus_code(struct v4l2_subdev *subdev,
 			code->code = ccs_csi_data_formats[i].code;
 			dev_err(&client->dev, "found index %u, i %u, code %x\n",
 				code->index, i, code->code);
-			rval = 0;
-			break;
+			return 0;
 		}
 	}
 
-out:
-	mutex_unlock(&sensor->mutex);
-
-	return rval;
+	return -EINVAL;
 }
 
-static u32 __ccs_get_mbus_code(struct v4l2_subdev *subdev, unsigned int pad)
+static u32 ccs_get_mbus_code(struct v4l2_subdev *subdev, unsigned int pad)
 {
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
 
@@ -2195,33 +2217,19 @@ static u32 __ccs_get_mbus_code(struct v4l2_subdev *subdev, unsigned int pad)
 		return sensor->internal_csi_format->code;
 }
 
-static int __ccs_get_format(struct v4l2_subdev *subdev,
-			    struct v4l2_subdev_state *sd_state,
-			    struct v4l2_subdev_format *fmt)
+static int ccs_get_format(struct v4l2_subdev *subdev,
+			  struct v4l2_subdev_state *sd_state,
+			  struct v4l2_subdev_format *fmt)
 {
 	fmt->format = *v4l2_subdev_state_get_format(sd_state, fmt->pad,
 						    fmt->stream);
 
 	if (fmt->pad != CCS_PAD_META && fmt->stream != CCS_STREAM_META)
-		fmt->format.code = __ccs_get_mbus_code(subdev, fmt->pad);
+		fmt->format.code = ccs_get_mbus_code(subdev, fmt->pad);
 
 	return 0;
 }
 
-static int ccs_get_format(struct v4l2_subdev *subdev,
-			  struct v4l2_subdev_state *sd_state,
-			  struct v4l2_subdev_format *fmt)
-{
-	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
-	int rval;
-
-	mutex_lock(&sensor->mutex);
-	rval = __ccs_get_format(subdev, sd_state, fmt);
-	mutex_unlock(&sensor->mutex);
-
-	return rval;
-}
-
 /* Changes require propagation only on sink pad. */
 static void ccs_propagate(struct v4l2_subdev *subdev,
 			  struct v4l2_subdev_state *sd_state, int which,
@@ -2245,7 +2253,6 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
 				sensor->scale_m = CCS_LIM(sensor, SCALER_N_MIN);
 				sensor->scaling_mode =
 					CCS_SCALING_MODE_NO_SCALING;
-				sensor->scaler_sink = *comp;
 			} else if (ssd == sensor->binner) {
 				sensor->binning_horizontal = 1;
 				sensor->binning_vertical = 1;
@@ -2260,8 +2267,6 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
 						   CCS_STREAM_PIXEL);
 		fmt->width = comp->width;
 		fmt->height = comp->height;
-		if (which == V4L2_SUBDEV_FORMAT_ACTIVE && ssd == sensor->src)
-			sensor->src_src = *crop;
 		break;
 	default:
 		WARN_ON_ONCE(1);
@@ -2280,7 +2285,7 @@ static int ccs_set_format_source(struct v4l2_subdev *subdev,
 	unsigned int i;
 	int rval;
 
-	rval = __ccs_get_format(subdev, sd_state, fmt);
+	rval = ccs_get_format(subdev, sd_state, fmt);
 	if (rval)
 		return rval;
 
@@ -2407,13 +2412,9 @@ static int ccs_set_format(struct v4l2_subdev *subdev,
 	if (subdev == &sensor->src->sd && fmt->pad == CCS_PAD_META)
 		return ccs_get_format(subdev, sd_state, fmt);
 
-	mutex_lock(&sensor->mutex);
-
 	if (subdev == &sensor->src->sd && fmt->stream == CCS_STREAM_META) {
 		ccs_set_format_meta(subdev, sd_state, &fmt->format);
 
-		mutex_unlock(&sensor->mutex);
-
 		return 0;
 	}
 
@@ -2424,13 +2425,12 @@ static int ccs_set_format(struct v4l2_subdev *subdev,
 		if (sensor->embedded_start != sensor->embedded_end)
 			ccs_set_format_meta(subdev, sd_state, NULL);
 
-		mutex_unlock(&sensor->mutex);
-
 		return rval;
 	}
 
 	/* Sink pad. Width and height are changeable here. */
-	fmt->format.code = __ccs_get_mbus_code(subdev, fmt->pad);
+	fmt->format.code = ccs_get_mbus_code(subdev, fmt->pad);
+
 	fmt->format.width &= ~1;
 	fmt->format.height &= ~1;
 	fmt->format.field = V4L2_FIELD_NONE;
@@ -2453,8 +2453,6 @@ static int ccs_set_format(struct v4l2_subdev *subdev,
 	crop->height = fmt->format.height;
 	ccs_propagate(subdev, sd_state, fmt->which, V4L2_SEL_TGT_CROP);
 
-	mutex_unlock(&sensor->mutex);
-
 	return 0;
 }
 
@@ -2768,9 +2766,6 @@ static int ccs_set_crop(struct v4l2_subdev *subdev,
 
 	if (ssd != sensor->pixel_array && sel->pad == CCS_PAD_SINK)
 		ccs_propagate(subdev, sd_state, sel->which, V4L2_SEL_TGT_CROP);
-	else if (sel->which == V4L2_SUBDEV_FORMAT_ACTIVE &&
-		 ssd == sensor->pixel_array)
-		sensor->pa_src = sel->r;
 
 	return 0;
 }
@@ -2842,8 +2837,6 @@ static int ccs_set_selection(struct v4l2_subdev *subdev,
 	if (ret)
 		return ret;
 
-	mutex_lock(&sensor->mutex);
-
 	sel->r.left = max(0, sel->r.left & ~1);
 	sel->r.top = max(0, sel->r.top & ~1);
 	sel->r.width = CCS_ALIGN_DIM(sel->r.width, sel->flags);
@@ -2865,7 +2858,6 @@ static int ccs_set_selection(struct v4l2_subdev *subdev,
 		ret = -EINVAL;
 	}
 
-	mutex_unlock(&sensor->mutex);
 	return ret;
 }
 
@@ -3244,6 +3236,7 @@ static int ccs_init_subdev(struct ccs_sensor *sensor,
 
 	ssd->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
 	ssd->sd.entity.function = function;
+	ssd->sd.state_lock = &sensor->mutex;
 	ssd->sensor = sensor;
 
 	ssd->npads = num_pads;
@@ -3283,8 +3276,8 @@ static int ccs_init_subdev(struct ccs_sensor *sensor,
 	return 0;
 }
 
-static int __ccs_init_state(struct v4l2_subdev *sd,
-			    struct v4l2_subdev_state *sd_state)
+static int ccs_init_state(struct v4l2_subdev *sd,
+			  struct v4l2_subdev_state *sd_state)
 {
 	struct ccs_subdev *ssd = to_ccs_subdev(sd);
 	struct ccs_sensor *sensor = ssd->sensor;
@@ -3303,12 +3296,8 @@ static int __ccs_init_state(struct v4l2_subdev *sd,
 	fmt->code = sensor->internal_csi_format->code;
 	fmt->field = V4L2_FIELD_NONE;
 
-	if (ssd == sensor->pixel_array) {
-		if (is_active)
-			sensor->pa_src = *crop;
-
+	if (ssd == sensor->pixel_array)
 		return 0;
-	}
 
 	fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC,
 					   CCS_STREAM_PIXEL);
@@ -3321,20 +3310,6 @@ static int __ccs_init_state(struct v4l2_subdev *sd,
 	return 0;
 }
 
-static int ccs_init_state(struct v4l2_subdev *sd,
-			  struct v4l2_subdev_state *sd_state)
-{
-	struct ccs_subdev *ssd = to_ccs_subdev(sd);
-	struct ccs_sensor *sensor = ssd->sensor;
-	int rval;
-
-	mutex_lock(&sensor->mutex);
-	rval = __ccs_init_state(sd, sd_state);
-	mutex_unlock(&sensor->mutex);
-
-	return rval;
-}
-
 static int ccs_src_init_state(struct v4l2_subdev *sd,
 			      struct v4l2_subdev_state *sd_state)
 {
@@ -3359,25 +3334,20 @@ static int ccs_src_init_state(struct v4l2_subdev *sd,
 	struct ccs_sensor *sensor = ssd->sensor;
 	int rval;
 
-	mutex_lock(&sensor->mutex);
-
 	if (sensor->embedded_start != sensor->embedded_end)
 		routing.num_routes++;
 
 	rval = v4l2_subdev_set_routing(sd, sd_state, &routing);
 	if (rval)
-		goto out;
+		return 0;
 
-	rval = __ccs_init_state(sd, sd_state);
+	rval = ccs_init_state(sd, sd_state);
 	if (rval)
-		goto out;
+		return 0;
 
 	if (sensor->embedded_start != sensor->embedded_end)
 		ccs_set_format_meta(sd, sd_state, NULL);
 
-out:
-	mutex_unlock(&sensor->mutex);
-
 	return rval;
 }
 
diff --git a/drivers/media/i2c/ccs/ccs.h b/drivers/media/i2c/ccs/ccs.h
index 207d33de3eee..a2a461cda338 100644
--- a/drivers/media/i2c/ccs/ccs.h
+++ b/drivers/media/i2c/ccs/ccs.h
@@ -230,7 +230,6 @@ struct ccs_sensor {
 	u32 mbus_frame_fmts;
 	const struct ccs_csi_data_format *csi_format;
 	const struct ccs_csi_data_format *internal_csi_format;
-	struct v4l2_rect pa_src, scaler_sink, src_src;
 	u32 default_mbus_frame_fmts;
 	int default_pixel_order;
 	struct ccs_data_container sdata, mdata;
-- 
2.39.2

[PATCH v8 27/38] media: ccs: Compute binning configuration from sub-device state

[Sakari Ailus]

Calculate binning configuration from sub-device state so the state related
configuration can be removed from the driver's device context struct.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ccs/ccs-core.c | 72 ++++++++++++++++++++++----------
 drivers/media/i2c/ccs/ccs.h      |  3 --
 2 files changed, 49 insertions(+), 26 deletions(-)

diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
index a8c48abd2e30..3b80c54453cc 100644
--- a/drivers/media/i2c/ccs/ccs-core.c
+++ b/drivers/media/i2c/ccs/ccs-core.c
@@ -511,13 +511,52 @@ static int ccs_pll_try(struct ccs_sensor *sensor, struct ccs_pll *pll)
 	return ccs_pll_calculate(&client->dev, &lim, pll);
 }
 
+static void
+ccs_get_binning(struct ccs_sensor *sensor, u8 *binning_mode, u8 *binh, u8 *binv)
+{
+	struct v4l2_subdev_state *binner_state =
+		v4l2_subdev_get_locked_active_state(&sensor->binner->sd);
+	struct v4l2_rect *binner_sink_crop =
+		v4l2_subdev_state_get_crop(binner_state, CCS_PAD_SINK,
+					   CCS_STREAM_PIXEL);
+	struct v4l2_rect *binner_sink_comp =
+		v4l2_subdev_state_get_compose(binner_state, CCS_PAD_SINK,
+					      CCS_STREAM_PIXEL);
+
+	if (binner_sink_crop->width == binner_sink_comp->width &&
+	    binner_sink_crop->height == binner_sink_comp->height) {
+		if (binning_mode)
+			*binning_mode = 0;
+
+		if (binh)
+			*binh = 1;
+
+		if (binv)
+			*binv = 1;
+
+		return;
+	}
+
+	if (binning_mode)
+		*binning_mode = 1;
+
+	if (binh)
+		*binh = binner_sink_crop->width / binner_sink_comp->width;
+
+	if (binv)
+		*binv = binner_sink_crop->height / binner_sink_comp->height;
+}
+
 static int ccs_pll_update(struct ccs_sensor *sensor)
 {
 	struct ccs_pll *pll = &sensor->pll;
+	u8 binh, binv;
 	int rval;
 
-	pll->binning_horizontal = sensor->binning_horizontal;
-	pll->binning_vertical = sensor->binning_vertical;
+	ccs_get_binning(sensor, NULL, &binh, &binv);
+
+	pll->binning_horizontal = binh;
+	pll->binning_vertical = binv;
 	pll->link_freq =
 		sensor->link_freq->qmenu_int[sensor->link_freq->val];
 	pll->scale_m = sensor->scale_m;
@@ -1241,8 +1280,11 @@ static void ccs_update_blanking(struct ccs_sensor *sensor,
 	struct v4l2_ctrl *hblank = sensor->hblank;
 	u16 min_fll, max_fll, min_llp, max_llp, min_lbp;
 	int min, max;
+	u8 binh, binv;
+
+	ccs_get_binning(sensor, NULL, &binh, &binv);
 
-	if (sensor->binning_vertical > 1 || sensor->binning_horizontal > 1) {
+	if (binv > 1 || binh > 1) {
 		min_fll = CCS_LIM(sensor, MIN_FRAME_LENGTH_LINES_BIN);
 		max_fll = CCS_LIM(sensor, MAX_FRAME_LENGTH_LINES_BIN);
 		min_llp = CCS_LIM(sensor, MIN_LINE_LENGTH_PCK_BIN);
@@ -1813,7 +1855,7 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
 		v4l2_subdev_state_get_crop(src_state, CCS_PAD_SRC,
 					   CCS_STREAM_PIXEL);
 	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
-	unsigned int binning_mode;
+	u8 binning_mode, binh, binv;
 	int rval;
 
 	if (pad != CCS_PAD_SRC)
@@ -1835,19 +1877,12 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
 		goto err_pm_put;
 
 	/* Binning configuration */
-	if (sensor->binning_horizontal == 1 &&
-	    sensor->binning_vertical == 1) {
-		binning_mode = 0;
-	} else {
-		u8 binning_type =
-			(sensor->binning_horizontal << 4)
-			| sensor->binning_vertical;
+	ccs_get_binning(sensor,	&binning_mode, &binh, &binv);
 
-		rval = ccs_write(sensor, BINNING_TYPE, binning_type);
+	if (binning_mode) {
+		rval = ccs_write(sensor, BINNING_TYPE, (binh << 4) | binv);
 		if (rval < 0)
 			goto err_pm_put;
-
-		binning_mode = 1;
 	}
 	rval = ccs_write(sensor, BINNING_MODE, binning_mode);
 	if (rval < 0)
@@ -2253,9 +2288,6 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
 				sensor->scale_m = CCS_LIM(sensor, SCALER_N_MIN);
 				sensor->scaling_mode =
 					CCS_SCALING_MODE_NO_SCALING;
-			} else if (ssd == sensor->binner) {
-				sensor->binning_horizontal = 1;
-				sensor->binning_vertical = 1;
 			}
 		}
 		fallthrough;
@@ -2529,10 +2561,6 @@ static void ccs_set_compose_binner(struct v4l2_subdev *subdev,
 			best = this;
 		}
 	}
-	if (sel->which == V4L2_SUBDEV_FORMAT_ACTIVE) {
-		sensor->binning_vertical = binv;
-		sensor->binning_horizontal = binh;
-	}
 
 	sel->r.width = (sink_crop->width / binh) & ~1;
 	sel->r.height = (sink_crop->height / binv) & ~1;
@@ -3721,8 +3749,6 @@ static int ccs_probe(struct i2c_client *client)
 				sensor->binning_subtypes[i].vertical);
 		}
 	}
-	sensor->binning_horizontal = 1;
-	sensor->binning_vertical = 1;
 
 	if (device_create_file(&client->dev, &dev_attr_ident) != 0) {
 		dev_err(&client->dev, "sysfs ident entry creation failed\n");
diff --git a/drivers/media/i2c/ccs/ccs.h b/drivers/media/i2c/ccs/ccs.h
index a2a461cda338..e6fc00a9fa11 100644
--- a/drivers/media/i2c/ccs/ccs.h
+++ b/drivers/media/i2c/ccs/ccs.h
@@ -237,9 +237,6 @@ struct ccs_sensor {
 	u32 embedded_mbus_code;
 	u8 emb_data_ctrl;
 
-	u8 binning_horizontal;
-	u8 binning_vertical;
-
 	u8 scale_m;
 	u8 scaling_mode;
 
-- 
2.39.2

[PATCH v8 28/38] media: ccs: Compute scaling configuration from sub-device state

[Sakari Ailus]

Compute scaling configuration from sub-device state instead of storing it
to the driver's device context struct.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ccs/ccs-core.c | 60 ++++++++++++++++++++++----------
 drivers/media/i2c/ccs/ccs.h      |  3 --
 2 files changed, 41 insertions(+), 22 deletions(-)

diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
index 3b80c54453cc..a147dbb9f362 100644
--- a/drivers/media/i2c/ccs/ccs-core.c
+++ b/drivers/media/i2c/ccs/ccs-core.c
@@ -547,19 +547,52 @@ ccs_get_binning(struct ccs_sensor *sensor, u8 *binning_mode, u8 *binh, u8 *binv)
 		*binv = binner_sink_crop->height / binner_sink_comp->height;
 }
 
+static void ccs_get_scaling(struct ccs_sensor *sensor,
+			    u8 *scaling_mode, u8 *scale_m)
+{
+	struct v4l2_subdev_state *scaler_state =
+		v4l2_subdev_get_locked_active_state(&sensor->scaler->sd);
+	struct v4l2_rect *scaler_sink_crop =
+		v4l2_subdev_state_get_crop(scaler_state, CCS_PAD_SINK,
+					   CCS_STREAM_PIXEL);
+	struct v4l2_rect *scaler_sink_comp =
+		v4l2_subdev_state_get_compose(scaler_state, CCS_PAD_SINK,
+					      CCS_STREAM_PIXEL);
+
+	if (scale_m)
+		*scale_m = scaler_sink_crop->width *
+			CCS_LIM(sensor, SCALER_N_MIN) /
+			scaler_sink_comp->width;
+
+	if (scaling_mode) {
+		if (scaler_sink_crop->width == scaler_sink_comp->width)
+			*scaling_mode = CCS_SCALING_MODE_NO_SCALING;
+		else if (scaler_sink_crop->height == scaler_sink_comp->height)
+			*scaling_mode = CCS_SCALING_MODE_HORIZONTAL;
+		else
+			*scaling_mode = SMIAPP_SCALING_MODE_BOTH;
+	}
+}
+
 static int ccs_pll_update(struct ccs_sensor *sensor)
 {
 	struct ccs_pll *pll = &sensor->pll;
 	u8 binh, binv;
+	u8 scale_m;
 	int rval;
 
 	ccs_get_binning(sensor, NULL, &binh, &binv);
 
+	if (sensor->scaler)
+		ccs_get_scaling(sensor, NULL, &scale_m);
+	else
+		scale_m = CCS_LIM(sensor, SCALER_N_MIN);
+
 	pll->binning_horizontal = binh;
 	pll->binning_vertical = binv;
 	pll->link_freq =
 		sensor->link_freq->qmenu_int[sensor->link_freq->val];
-	pll->scale_m = sensor->scale_m;
+	pll->scale_m = scale_m;
 	pll->bits_per_pixel = sensor->csi_format->compressed;
 
 	rval = ccs_pll_try(sensor, pll);
@@ -1202,7 +1235,7 @@ static int ccs_get_mbus_formats(struct ccs_sensor *sensor)
 	/* Figure out which BPP values can be used with which formats. */
 	pll->binning_horizontal = 1;
 	pll->binning_vertical = 1;
-	pll->scale_m = sensor->scale_m;
+	pll->scale_m = CCS_LIM(sensor, SCALER_N_MIN);
 
 	for (i = 0; i < ARRAY_SIZE(ccs_csi_data_formats); i++) {
 		sensor->compressed_min_bpp =
@@ -1950,11 +1983,15 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
 	/* Scaling */
 	if (CCS_LIM(sensor, SCALING_CAPABILITY)
 	    != CCS_SCALING_CAPABILITY_NONE) {
-		rval = ccs_write(sensor, SCALING_MODE, sensor->scaling_mode);
+		u8 scaling_mode, scale_m;
+
+		ccs_get_scaling(sensor, &scaling_mode, &scale_m);
+
+		rval = ccs_write(sensor, SCALING_MODE, scaling_mode);
 		if (rval < 0)
 			goto err_pm_put;
 
-		rval = ccs_write(sensor, SCALE_M, sensor->scale_m);
+		rval = ccs_write(sensor, SCALE_M, scale_m);
 		if (rval < 0)
 			goto err_pm_put;
 	}
@@ -2270,7 +2307,6 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
 			  struct v4l2_subdev_state *sd_state, int which,
 			  int target)
 {
-	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
 	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
 	struct v4l2_rect *comp, *crop;
 	struct v4l2_mbus_framefmt *fmt;
@@ -2283,13 +2319,6 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
 						  CCS_STREAM_PIXEL);
 		comp->width = crop->width;
 		comp->height = crop->height;
-		if (which == V4L2_SUBDEV_FORMAT_ACTIVE) {
-			if (ssd == sensor->scaler) {
-				sensor->scale_m = CCS_LIM(sensor, SCALER_N_MIN);
-				sensor->scaling_mode =
-					CCS_SCALING_MODE_NO_SCALING;
-			}
-		}
 		fallthrough;
 	case V4L2_SEL_TGT_COMPOSE:
 		crop = v4l2_subdev_state_get_crop(sd_state, CCS_PAD_SRC,
@@ -2674,11 +2703,6 @@ static void ccs_set_compose_scaler(struct v4l2_subdev *subdev,
 			& ~1;
 	else
 		sel->r.height = sink_crop->height;
-
-	if (sel->which == V4L2_SUBDEV_FORMAT_ACTIVE) {
-		sensor->scale_m = scale_m;
-		sensor->scaling_mode = mode;
-	}
 }
 /* We're only called on source pads. This function sets scaling. */
 static int ccs_set_compose(struct v4l2_subdev *subdev,
@@ -3785,8 +3809,6 @@ static int ccs_probe(struct i2c_client *client)
 	sensor->pixel_array = &sensor->ssds[sensor->ssds_used];
 	sensor->ssds_used++;
 
-	sensor->scale_m = CCS_LIM(sensor, SCALER_N_MIN);
-
 	/* prepare PLL configuration input values */
 	sensor->pll.bus_type = CCS_PLL_BUS_TYPE_CSI2_DPHY;
 	sensor->pll.csi2.lanes = sensor->hwcfg.lanes;
diff --git a/drivers/media/i2c/ccs/ccs.h b/drivers/media/i2c/ccs/ccs.h
index e6fc00a9fa11..d33014f2710b 100644
--- a/drivers/media/i2c/ccs/ccs.h
+++ b/drivers/media/i2c/ccs/ccs.h
@@ -237,9 +237,6 @@ struct ccs_sensor {
 	u32 embedded_mbus_code;
 	u8 emb_data_ctrl;
 
-	u8 scale_m;
-	u8 scaling_mode;
-
 	u8 frame_skip;
 	u16 embedded_start; /* embedded data start line */
 	u16 embedded_end;
-- 
2.39.2

[PATCH v8 29/38] media: ccs: Remove which parameter from ccs_propagate

[Sakari Ailus]

ccs_propagate() no longer stores information in the driver's context
struct. The which parameter can thus be removed.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ccs/ccs-core.c | 12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)

diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
index a147dbb9f362..838daab212f2 100644
--- a/drivers/media/i2c/ccs/ccs-core.c
+++ b/drivers/media/i2c/ccs/ccs-core.c
@@ -2304,8 +2304,7 @@ static int ccs_get_format(struct v4l2_subdev *subdev,
 
 /* Changes require propagation only on sink pad. */
 static void ccs_propagate(struct v4l2_subdev *subdev,
-			  struct v4l2_subdev_state *sd_state, int which,
-			  int target)
+			  struct v4l2_subdev_state *sd_state, int target)
 {
 	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
 	struct v4l2_rect *comp, *crop;
@@ -2512,7 +2511,7 @@ static int ccs_set_format(struct v4l2_subdev *subdev,
 	crop->top = 0;
 	crop->width = fmt->format.width;
 	crop->height = fmt->format.height;
-	ccs_propagate(subdev, sd_state, fmt->which, V4L2_SEL_TGT_CROP);
+	ccs_propagate(subdev, sd_state, V4L2_SEL_TGT_CROP);
 
 	return 0;
 }
@@ -2727,7 +2726,7 @@ static int ccs_set_compose(struct v4l2_subdev *subdev,
 		ccs_set_compose_scaler(subdev, sd_state, sel, sink_crop, comp);
 
 	*comp = sel->r;
-	ccs_propagate(subdev, sd_state, sel->which, V4L2_SEL_TGT_COMPOSE);
+	ccs_propagate(subdev, sd_state, V4L2_SEL_TGT_COMPOSE);
 
 	if (sel->which == V4L2_SUBDEV_FORMAT_ACTIVE)
 		return ccs_pll_blanking_update(sensor);
@@ -2817,7 +2816,7 @@ static int ccs_set_crop(struct v4l2_subdev *subdev,
 	*crop = sel->r;
 
 	if (ssd != sensor->pixel_array && sel->pad == CCS_PAD_SINK)
-		ccs_propagate(subdev, sd_state, sel->which, V4L2_SEL_TGT_CROP);
+		ccs_propagate(subdev, sd_state, V4L2_SEL_TGT_CROP);
 
 	return 0;
 }
@@ -3339,7 +3338,6 @@ static int ccs_init_state(struct v4l2_subdev *sd,
 		v4l2_subdev_state_get_format(sd_state, pad, CCS_STREAM_PIXEL);
 	struct v4l2_rect *crop =
 		v4l2_subdev_state_get_crop(sd_state, pad, CCS_STREAM_PIXEL);
-	bool is_active = !sd->active_state || sd->active_state == sd_state;
 
 	ccs_get_native_size(ssd, crop);
 
@@ -3357,7 +3355,7 @@ static int ccs_init_state(struct v4l2_subdev *sd,
 		sensor->csi_format->code : sensor->internal_csi_format->code;
 	fmt->field = V4L2_FIELD_NONE;
 
-	ccs_propagate(sd, sd_state, is_active, V4L2_SEL_TGT_CROP);
+	ccs_propagate(sd, sd_state, V4L2_SEL_TGT_CROP);
 
 	return 0;
 }
-- 
2.39.2

[PATCH v8 30/38] media: Documentation: ccs: Document routing

[Sakari Ailus]

Document which routes are available for the CCS driver (source) sub-device
and what configuration are possible.

Also update copyright.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../userspace-api/media/drivers/ccs.rst       | 34 ++++++++++++++++++-
 .../media/v4l/subdev-formats.rst              |  2 ++
 2 files changed, 35 insertions(+), 1 deletion(-)

diff --git a/Documentation/userspace-api/media/drivers/ccs.rst b/Documentation/userspace-api/media/drivers/ccs.rst
index 03015b33d5ab..bc2804ec663b 100644
--- a/Documentation/userspace-api/media/drivers/ccs.rst
+++ b/Documentation/userspace-api/media/drivers/ccs.rst
@@ -111,4 +111,36 @@ than in the centre.
 Shading correction needs to be enabled for luminance correction level to have an
 effect.
 
-**Copyright** |copy| 2020 Intel Corporation
+.. _media-ccs-routes:
+
+Routes
+------
+
+The CCS driver implements one or two :ref:`routes <subdev-routing>` in
+its source sub-device (scaler sub-device if exists for the device, otherwise
+binner) depending on whether the sensor supports embedded data. (All CCS
+compliant sensors do but the CCS driver supports preceding standards that did
+not require embedded data support, too.)
+
+The first route of the CCS source sub-device is for pixel data (internal pad
+1/stream 0 -> pad 0/stream 0) and the second one is for embedded data (internal
+pad 2/stream 0 -> pad 0/stream 1).
+
+Embedded data
+~~~~~~~~~~~~~
+
+MIPI CCS supports generation of camera sensor embedded data. The media bus code
+used for this format is :ref:`MEDIA_BUS_FMT_CCS_EMBEDDDED
+<MEDIA-BUS-FMT-CCS-EMBEDDED>`.
+
+The bit depth of the CCS pixel data affects how the sensor will output the
+embedded data, adding padding to align with CSI-2 bus :ref:`Data units
+<media-glossary-data-unit>` for that particular bit depth. This is indicated by
+the generic metadata format on the sensor's source sub-device's source pad.
+
+Embedded data for bit depths greater than or equal to 16 may support more dense
+packing or legacy single metadata byte per data unit, or both of these,
+depending on the device. The supported embedded data formats can be enumerated
+and configured on stream 1 of the source pad (1) of the CCS source sub-device.
+
+**Copyright** |copy| 2020, 2023 Intel Corporation
diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
index c8f982411e70..ca4da6a400ff 100644
--- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
+++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
@@ -8592,3 +8592,5 @@ levels above.
 This mbus code are only used for "2-byte simplified tagged data format" (code
 0xa) but their use may be extended further in the future, to cover other CCS
 embedded data format codes.
+
+Also see :ref:`CCS driver documentation <media-ccs-routes>`.
-- 
2.39.2

[PATCH v8 31/38] media: uapi: v4l: subdev: Enable streams API

[Sakari Ailus]

Remove v4l2_subdev_enable_streams_api variable that was used to easily
enable streams API for development, and conditions that use the variable.

This patch enables the streams API for V4L2 sub-device interface which
allows transporting multiple streams on a single MC link.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 30 ---------------------------
 1 file changed, 30 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index c8c435df92c8..b3a697df4b66 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -26,15 +26,6 @@
 #include <media/v4l2-fh.h>
 #include <media/v4l2-ioctl.h>
 
-#if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
-/*
- * The Streams API is an experimental feature. To use the Streams API, set
- * 'v4l2_subdev_enable_streams_api' to 1 below.
- */
-
-static bool v4l2_subdev_enable_streams_api;
-#endif
-
 /*
  * Maximum stream ID is 63 for now, as we use u64 bitmask to represent a set
  * of streams.
@@ -570,13 +561,6 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 				       V4L2_SUBDEV_CLIENT_CAP_STREAMS;
 	int rval;
 
-	/*
-	 * If the streams API is not enabled, remove V4L2_SUBDEV_CAP_STREAMS.
-	 * Remove this when the API is no longer experimental.
-	 */
-	if (!v4l2_subdev_enable_streams_api)
-		streams_subdev = false;
-
 	switch (cmd) {
 	case VIDIOC_SUBDEV_QUERYCAP: {
 		struct v4l2_subdev_capability *cap = arg;
@@ -916,9 +900,6 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 		struct v4l2_subdev_krouting krouting = {};
 		unsigned int i;
 
-		if (!v4l2_subdev_enable_streams_api)
-			return -ENOIOCTLCMD;
-
 		if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS))
 			return -ENOIOCTLCMD;
 
@@ -989,9 +970,6 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 		struct v4l2_subdev_routing *routing = arg;
 		struct v4l2_subdev_krouting *krouting;
 
-		if (!v4l2_subdev_enable_streams_api)
-			return -ENOIOCTLCMD;
-
 		if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS))
 			return -ENOIOCTLCMD;
 
@@ -1019,14 +997,6 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 	case VIDIOC_SUBDEV_S_CLIENT_CAP: {
 		struct v4l2_subdev_client_capability *client_cap = arg;
 
-		/*
-		 * Clear V4L2_SUBDEV_CLIENT_CAP_STREAMS if streams API is not
-		 * enabled. Remove this when streams API is no longer
-		 * experimental.
-		 */
-		if (!v4l2_subdev_enable_streams_api)
-			client_cap->capabilities &= ~V4L2_SUBDEV_CLIENT_CAP_STREAMS;
-
 		/* Filter out unsupported capabilities */
 		client_cap->capabilities &= (V4L2_SUBDEV_CLIENT_CAP_STREAMS |
 					     V4L2_SUBDEV_CLIENT_CAP_INTERVAL_USES_WHICH);
-- 
2.39.2

[PATCH v8 32/38] media: uapi: Add media bus code for ov2740 embedded data

[Sakari Ailus]

Add a media bus code for ov2740 camera sensor embedded data and document
it.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../media/v4l/subdev-formats.rst              | 66 +++++++++++++++++++
 include/uapi/linux/media-bus-format.h         |  3 +-
 2 files changed, 68 insertions(+), 1 deletion(-)

diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
index ca4da6a400ff..a875868ed951 100644
--- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
+++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
@@ -8594,3 +8594,69 @@ This mbus code are only used for "2-byte simplified tagged data format" (code
 embedded data format codes.
 
 Also see :ref:`CCS driver documentation <media-ccs-routes>`.
+
+Omnivision OV2740 Embedded Data Format
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The Omnivision OV2740 camera sensor produces the following embedded data format.
+
+.. flat-table:: Omnivision OV2740 Embedded Data Format. Octets at indices marked
+                reserved or unused have been omitted from the table.
+    :header-rows: 1
+
+    * - Byte
+      - Concent description
+    * - 0
+      - Sensor info
+    * - 4
+      - Analogue gain (bits 10--8)
+    * - 5
+      - Analogue gain (bits 7--0)
+    * - 6
+      - Coarse integration time (bits 15--8)
+    * - 7
+      - Coarse integration time (bits 7--0)
+    * - 10
+      - Dpc correction threshold (bits 9--2)
+    * - 15
+      - Output image width (bits 15--8)
+    * - 16
+      - Output image width (bits 7--0)
+    * - 17
+      - Output image height (bits 15--8)
+    * - 18
+      - Output image height (bits 7--0)
+    * - 23
+      - MIPI header revision number
+    * - 31
+      - Vertical (bit 1) and horizontal flip (bit 0)
+    * - 32
+      - Frame duration A
+    * - 33
+      - Frame duration B
+    * - 34
+      - Context count
+    * - 35
+      - Context select
+    * - 54
+      - Data pedestal (bits 9--2)
+    * - 63
+      - Frame average (bits 9--2)
+    * - 64
+      - Digital gain red
+    * - 65
+      - Digital gain red
+    * - 66
+      - Digital gain greenr
+    * - 67
+      - Digital gain greenr
+    * - 68
+      - Digital gain blue
+    * - 69
+      - Digital gain blue
+    * - 70
+      - Digital gain greenb
+    * - 71
+      - Digital gain greenb
+    * - 89
+      - Frame counter
diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
index 03f7e9ab517b..13e68c2ccb61 100644
--- a/include/uapi/linux/media-bus-format.h
+++ b/include/uapi/linux/media-bus-format.h
@@ -183,7 +183,8 @@
 #define MEDIA_BUS_FMT_META_20			0x8006
 #define MEDIA_BUS_FMT_META_24			0x8007
 
-/* Specific metadata formats. Next is 0x9002. */
+/* Specific metadata formats. Next is 0x9003. */
 #define MEDIA_BUS_FMT_CCS_EMBEDDED		0x9001
+#define MEDIA_BUS_FMT_OV2740_EMBEDDED		0x9002
 
 #endif /* __LINUX_MEDIA_BUS_FORMAT_H */
-- 
2.39.2

[PATCH v8 33/38] media: ov2740: Switch to {enable,disable}_streams

[Sakari Ailus]

Switch from s_stream to enable_streams and disable_streams callbacks.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ov2740.c | 72 +++++++++++++++++---------------------
 1 file changed, 32 insertions(+), 40 deletions(-)

diff --git a/drivers/media/i2c/ov2740.c b/drivers/media/i2c/ov2740.c
index 552935ccb4a9..44c6724a102c 100644
--- a/drivers/media/i2c/ov2740.c
+++ b/drivers/media/i2c/ov2740.c
@@ -919,16 +919,23 @@ static int ov2740_load_otp_data(struct nvm_data *nvm)
 	return ret;
 }
 
-static int ov2740_start_streaming(struct ov2740 *ov2740)
+static int ov2740_enable_streams(struct v4l2_subdev *sd,
+				 struct v4l2_subdev_state *state, u32 pad,
+				 u64 streams_mask)
 {
-	struct i2c_client *client = v4l2_get_subdevdata(&ov2740->sd);
+	struct i2c_client *client = v4l2_get_subdevdata(sd);
+	struct ov2740 *ov2740 = to_ov2740(sd);
 	const struct ov2740_reg_list *reg_list;
 	int link_freq_index;
 	int ret;
 
+	ret = pm_runtime_resume_and_get(&client->dev);
+	if (ret < 0)
+		return ret;
+
 	ret = ov2740_identify_module(ov2740);
 	if (ret)
-		return ret;
+		goto out_pm_put;
 
 	if (ov2740->nvm)
 		ov2740_load_otp_data(ov2740->nvm);
@@ -937,7 +944,7 @@ static int ov2740_start_streaming(struct ov2740 *ov2740)
 	ret = ov2740_write_reg(ov2740, 0x0103, 1, 0x01);
 	if (ret) {
 		dev_err(&client->dev, "failed to reset\n");
-		return ret;
+		goto out_pm_put;
 	}
 
 	usleep_range(10000, 15000);
@@ -947,64 +954,47 @@ static int ov2740_start_streaming(struct ov2740 *ov2740)
 	ret = ov2740_write_reg_list(ov2740, reg_list);
 	if (ret) {
 		dev_err(&client->dev, "failed to set plls\n");
-		return ret;
+		goto out_pm_put;
 	}
 
 	reg_list = &ov2740->cur_mode->reg_list;
 	ret = ov2740_write_reg_list(ov2740, reg_list);
 	if (ret) {
 		dev_err(&client->dev, "failed to set mode\n");
-		return ret;
+		goto out_pm_put;
 	}
 
 	ret = __v4l2_ctrl_handler_setup(ov2740->sd.ctrl_handler);
 	if (ret)
-		return ret;
+		goto out_pm_put;
 
 	ret = ov2740_write_reg(ov2740, OV2740_REG_MODE_SELECT, 1,
 			       OV2740_MODE_STREAMING);
-	if (ret)
+	if (ret) {
 		dev_err(&client->dev, "failed to start streaming\n");
+		goto out_pm_put;
+	}
 
-	return ret;
-}
+	return 0;
 
-static void ov2740_stop_streaming(struct ov2740 *ov2740)
-{
-	struct i2c_client *client = v4l2_get_subdevdata(&ov2740->sd);
+out_pm_put:
+	pm_runtime_put(&client->dev);
 
-	if (ov2740_write_reg(ov2740, OV2740_REG_MODE_SELECT, 1,
-			     OV2740_MODE_STANDBY))
-		dev_err(&client->dev, "failed to stop streaming\n");
+	return ret;
 }
 
-static int ov2740_set_stream(struct v4l2_subdev *sd, int enable)
+static int ov2740_disable_streams(struct v4l2_subdev *sd,
+				  struct v4l2_subdev_state *state, u32 pad,
+				  u64 streams_mask)
 {
-	struct ov2740 *ov2740 = to_ov2740(sd);
 	struct i2c_client *client = v4l2_get_subdevdata(sd);
-	struct v4l2_subdev_state *sd_state;
-	int ret = 0;
-
-	sd_state = v4l2_subdev_lock_and_get_active_state(&ov2740->sd);
-
-	if (enable) {
-		ret = pm_runtime_resume_and_get(&client->dev);
-		if (ret < 0)
-			goto out_unlock;
+	struct ov2740 *ov2740 = to_ov2740(sd);
+	int ret;
 
-		ret = ov2740_start_streaming(ov2740);
-		if (ret) {
-			enable = 0;
-			ov2740_stop_streaming(ov2740);
-			pm_runtime_put(&client->dev);
-		}
-	} else {
-		ov2740_stop_streaming(ov2740);
-		pm_runtime_put(&client->dev);
-	}
+	ret = ov2740_write_reg(ov2740, OV2740_REG_MODE_SELECT, 1,
+			       OV2740_MODE_STANDBY);
 
-out_unlock:
-	v4l2_subdev_unlock_state(sd_state);
+	pm_runtime_put(&client->dev);
 
 	return ret;
 }
@@ -1089,7 +1079,7 @@ static int ov2740_init_state(struct v4l2_subdev *sd,
 }
 
 static const struct v4l2_subdev_video_ops ov2740_video_ops = {
-	.s_stream = ov2740_set_stream,
+	.s_stream = v4l2_subdev_s_stream_helper,
 };
 
 static const struct v4l2_subdev_pad_ops ov2740_pad_ops = {
@@ -1097,6 +1087,8 @@ static const struct v4l2_subdev_pad_ops ov2740_pad_ops = {
 	.set_fmt = ov2740_set_format,
 	.enum_mbus_code = ov2740_enum_mbus_code,
 	.enum_frame_size = ov2740_enum_frame_size,
+	.enable_streams = ov2740_enable_streams,
+	.disable_streams = ov2740_disable_streams,
 };
 
 static const struct v4l2_subdev_ops ov2740_subdev_ops = {
-- 
2.39.2

[PATCH v8 34/38] media: ov2740: Track streaming state

[Sakari Ailus]

With enable_streams and disable_streams, the driver for a device where
streams are not independently started and stopped needs to maintain state
information on streams that have been requested to be started. Do that
now.

In the future, a helper function in the framework is a desirable way to do
this instead.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ov2740.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/drivers/media/i2c/ov2740.c b/drivers/media/i2c/ov2740.c
index 44c6724a102c..df57f0096e98 100644
--- a/drivers/media/i2c/ov2740.c
+++ b/drivers/media/i2c/ov2740.c
@@ -539,6 +539,9 @@ struct ov2740 {
 
 	/* True if the device has been identified */
 	bool identified;
+
+	/* Track streaming state */
+	u8 streaming;
 };
 
 static inline struct ov2740 *to_ov2740(struct v4l2_subdev *subdev)
@@ -929,6 +932,11 @@ static int ov2740_enable_streams(struct v4l2_subdev *sd,
 	int link_freq_index;
 	int ret;
 
+	if (ov2740->streaming) {
+		ov2740->streaming |= streams_mask;
+		return 0;
+	}
+
 	ret = pm_runtime_resume_and_get(&client->dev);
 	if (ret < 0)
 		return ret;
@@ -975,6 +983,8 @@ static int ov2740_enable_streams(struct v4l2_subdev *sd,
 		goto out_pm_put;
 	}
 
+	ov2740->streaming |= streams_mask;
+
 	return 0;
 
 out_pm_put:
@@ -991,6 +1001,10 @@ static int ov2740_disable_streams(struct v4l2_subdev *sd,
 	struct ov2740 *ov2740 = to_ov2740(sd);
 	int ret;
 
+	ov2740->streaming &= ~streams_mask;
+	if (ov2740->streaming)
+		return 0;
+
 	ret = ov2740_write_reg(ov2740, OV2740_REG_MODE_SELECT, 1,
 			       OV2740_MODE_STANDBY);
 
-- 
2.39.2

[PATCH v8 35/38] media: ov2740: Add support for embedded data

[Sakari Ailus]

Add support for embedded data. This introduces two internal pads for pixel
and embedded data streams. As the driver supports a single mode only,
there's no need for backward compatibility in mode selection.

The embedded data is configured to be placed before the image data whereas
after the image data is the default.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ov2740.c | 150 +++++++++++++++++++++++++++++++++----
 1 file changed, 137 insertions(+), 13 deletions(-)

diff --git a/drivers/media/i2c/ov2740.c b/drivers/media/i2c/ov2740.c
index df57f0096e98..7488b2535071 100644
--- a/drivers/media/i2c/ov2740.c
+++ b/drivers/media/i2c/ov2740.c
@@ -11,6 +11,7 @@
 #include <linux/pm_runtime.h>
 #include <linux/nvmem-provider.h>
 #include <linux/regmap.h>
+#include <media/mipi-csi2.h>
 #include <media/v4l2-ctrls.h>
 #include <media/v4l2-device.h>
 #include <media/v4l2-fwnode.h>
@@ -71,11 +72,31 @@
 #define OV2740_REG_ISP_CTRL00		0x5000
 /* ISP CTRL01 */
 #define OV2740_REG_ISP_CTRL01		0x5001
+
+/* Embedded data line location control */
+#define OV2740_REG_EMBEDDED_FLAG	0x5a08
+#define OV2740_EMBEDDED_FLAG_FOOTER	BIT(2) /* otherwise it's in header */
+#define OV2740_EMBEDDED_FLAG_MYSTERY	BIT(1)
 /* Customer Addresses: 0x7010 - 0x710F */
 #define CUSTOMER_USE_OTP_SIZE		0x100
 /* OTP registers from sensor */
 #define OV2740_REG_OTP_CUSTOMER		0x7010
 
+enum {
+	OV2740_PAD_SOURCE,
+	OV2740_PAD_PIXEL,
+	OV2740_PAD_META,
+	OV2740_NUM_PADS,
+};
+
+enum {
+	OV2740_STREAM_PIXEL,
+	OV2740_STREAM_META,
+};
+
+#define OV2740_META_WIDTH		100U /* 97 bytes of actual data */
+#define OV2740_META_HEIGHT		1U
+
 struct nvm_data {
 	struct nvmem_device *nvmem;
 	struct regmap *regmap;
@@ -513,7 +534,7 @@ static const struct ov2740_mode supported_modes_180mhz[] = {
 
 struct ov2740 {
 	struct v4l2_subdev sd;
-	struct media_pad pad;
+	struct media_pad pads[OV2740_NUM_PADS];
 	struct v4l2_ctrl_handler ctrl_handler;
 
 	/* V4L2 Controls */
@@ -976,6 +997,11 @@ static int ov2740_enable_streams(struct v4l2_subdev *sd,
 	if (ret)
 		goto out_pm_put;
 
+	ret = ov2740_write_reg(ov2740, OV2740_REG_EMBEDDED_FLAG, 1,
+			       OV2740_EMBEDDED_FLAG_MYSTERY);
+	if (ret)
+		return ret;
+
 	ret = ov2740_write_reg(ov2740, OV2740_REG_MODE_SELECT, 1,
 			       OV2740_MODE_STREAMING);
 	if (ret) {
@@ -1013,23 +1039,49 @@ static int ov2740_disable_streams(struct v4l2_subdev *sd,
 	return ret;
 }
 
-static int ov2740_set_format(struct v4l2_subdev *sd,
-			     struct v4l2_subdev_state *sd_state,
-			     struct v4l2_subdev_format *fmt)
+static int __ov2740_set_format(struct v4l2_subdev *sd,
+			       struct v4l2_subdev_state *sd_state,
+			       struct v4l2_mbus_framefmt *format,
+			       enum v4l2_subdev_format_whence which,
+			       unsigned int pad, unsigned int stream)
 {
+	struct v4l2_mbus_framefmt *src_pix_fmt, *src_meta_fmt, *pix_fmt,
+		*meta_fmt;
 	struct ov2740 *ov2740 = to_ov2740(sd);
 	const struct ov2740_mode *mode;
 	s32 vblank_def, h_blank;
 
+	/*
+	 * Allow setting format on internal pixel pad as well as the source
+	 * pad's pixel stream (for compatibility).
+	 */
+	if (pad == OV2740_PAD_SOURCE || pad == OV2740_PAD_META ||
+	    stream == OV2740_STREAM_META) {
+		*format = *v4l2_subdev_state_get_format(sd_state, pad, stream);
+		return 0;
+	}
+
+	pix_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_PIXEL, 0);
+	meta_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_META, 0);
+	src_pix_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_SOURCE,
+						   OV2740_STREAM_PIXEL);
+	src_meta_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_SOURCE,
+						    OV2740_STREAM_META);
+
 	mode = v4l2_find_nearest_size(ov2740->supported_modes,
 				      ov2740->supported_modes_count,
 				      width, height,
-				      fmt->format.width, fmt->format.height);
+				      format->width, format->height);
+	ov2740_update_pad_format(mode, pix_fmt);
+	*format = *src_pix_fmt = *pix_fmt;
 
-	ov2740_update_pad_format(mode, &fmt->format);
-	*v4l2_subdev_state_get_format(sd_state, fmt->pad) = fmt->format;
+	meta_fmt->code = MEDIA_BUS_FMT_OV2740_EMBEDDED;
+	meta_fmt->width = OV2740_META_WIDTH;
+	meta_fmt->height = OV2740_META_HEIGHT;
+	*src_meta_fmt = *meta_fmt;
+	src_meta_fmt->code = MEDIA_BUS_FMT_META_10;
 
-	if (fmt->which == V4L2_SUBDEV_FORMAT_TRY)
+	if (which == V4L2_SUBDEV_FORMAT_TRY)
 		return 0;
 
 	ov2740->cur_mode = mode;
@@ -1049,6 +1101,14 @@ static int ov2740_set_format(struct v4l2_subdev *sd,
 	return 0;
 }
 
+static int ov2740_set_format(struct v4l2_subdev *sd,
+			     struct v4l2_subdev_state *sd_state,
+			     struct v4l2_subdev_format *fmt)
+{
+	return __ov2740_set_format(sd, sd_state, &fmt->format, fmt->which,
+				   fmt->pad, fmt->stream);
+}
+
 static int ov2740_enum_mbus_code(struct v4l2_subdev *sd,
 				 struct v4l2_subdev_state *sd_state,
 				 struct v4l2_subdev_mbus_code_enum *code)
@@ -1085,10 +1145,68 @@ static int ov2740_enum_frame_size(struct v4l2_subdev *sd,
 static int ov2740_init_state(struct v4l2_subdev *sd,
 			     struct v4l2_subdev_state *sd_state)
 {
+	struct v4l2_subdev_route routes[] = {
+		{
+			.sink_pad = OV2740_PAD_PIXEL,
+			.source_pad = OV2740_PAD_SOURCE,
+			.source_stream = OV2740_STREAM_PIXEL,
+			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
+		}, {
+			.sink_pad = OV2740_PAD_META,
+			.source_pad = OV2740_PAD_SOURCE,
+			.source_stream = OV2740_STREAM_META,
+			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
+		},
+	};
+	struct v4l2_subdev_krouting routing = {
+		.routes = routes,
+		.num_routes = ARRAY_SIZE(routes),
+	};
+	struct v4l2_subdev_state *active_state;
+	struct v4l2_mbus_framefmt format = { 0 };
 	struct ov2740 *ov2740 = to_ov2740(sd);
+	int ret;
+
+	ret = v4l2_subdev_set_routing(sd, sd_state, &routing);
+	if (ret)
+		return ret;
+
+	active_state = v4l2_subdev_get_locked_active_state(sd);
+
+	ov2740_update_pad_format(&ov2740->supported_modes[0], &format);
+
+	return __ov2740_set_format(sd, sd_state, &format,
+				   active_state == sd_state ?
+				   V4L2_SUBDEV_FORMAT_ACTIVE :
+				   V4L2_SUBDEV_FORMAT_TRY, OV2740_PAD_PIXEL, 0);
+}
+
+static int ov2740_get_frame_desc(struct v4l2_subdev *sd, unsigned int pad,
+				 struct v4l2_mbus_frame_desc *desc)
+{
+	struct v4l2_mbus_frame_desc_entry *entry = desc->entry;
+	struct v4l2_subdev_state *sd_state;
+	struct v4l2_mbus_framefmt *fmt;
+
+	desc->type = V4L2_MBUS_FRAME_DESC_TYPE_CSI2;
+
+	sd_state = v4l2_subdev_lock_and_get_active_state(sd);
+	fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_SOURCE,
+					   OV2740_STREAM_PIXEL);
+	entry->pixelcode = fmt->code;
+	v4l2_subdev_unlock_state(sd_state);
+
+	entry->stream = OV2740_STREAM_PIXEL;
+	entry->bus.csi2.dt = MIPI_CSI2_DT_RAW10;
+	entry++;
+	desc->num_entries++;
+
+	entry->pixelcode = MEDIA_BUS_FMT_META_8;
+	entry->stream = OV2740_STREAM_META;
+	entry->bus.csi2.dt = MIPI_CSI2_DT_GENERIC_LONG(1);
+	entry++;
+	desc->num_entries++;
 
-	ov2740_update_pad_format(&ov2740->supported_modes[0],
-				 v4l2_subdev_state_get_format(sd_state, 0));
 	return 0;
 }
 
@@ -1103,6 +1221,7 @@ static const struct v4l2_subdev_pad_ops ov2740_pad_ops = {
 	.enum_frame_size = ov2740_enum_frame_size,
 	.enable_streams = ov2740_enable_streams,
 	.disable_streams = ov2740_disable_streams,
+	.get_frame_desc = ov2740_get_frame_desc,
 };
 
 static const struct v4l2_subdev_ops ov2740_subdev_ops = {
@@ -1369,11 +1488,16 @@ static int ov2740_probe(struct i2c_client *client)
 	}
 
 	ov2740->sd.state_lock = ov2740->ctrl_handler.lock;
-	ov2740->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
+	ov2740->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_STREAMS;
 	ov2740->sd.entity.ops = &ov2740_subdev_entity_ops;
 	ov2740->sd.entity.function = MEDIA_ENT_F_CAM_SENSOR;
-	ov2740->pad.flags = MEDIA_PAD_FL_SOURCE;
-	ret = media_entity_pads_init(&ov2740->sd.entity, 1, &ov2740->pad);
+	ov2740->pads[OV2740_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
+	ov2740->pads[OV2740_PAD_PIXEL].flags =
+		MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL;
+	ov2740->pads[OV2740_PAD_META].flags =
+		MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL;
+	ret = media_entity_pads_init(&ov2740->sd.entity,
+				     ARRAY_SIZE(ov2740->pads), ov2740->pads);
 	if (ret) {
 		dev_err_probe(dev, ret, "failed to init entity pads\n");
 		goto probe_error_v4l2_ctrl_handler_free;
-- 
2.39.2

[PATCH v8 36/38] media: v4l: Add V4L2_SUBDEV_ROUTE_FL_IMMUTABLE sub-device routing flag

[Sakari Ailus]

Add a flag to denote immutable routes, V4L2_SUBDEV_ROUTE_FL_IMMUTABLE.
Such routes cannot be changed and they're always active.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 Documentation/userspace-api/media/v4l/dev-subdev.rst         | 3 ++-
 .../userspace-api/media/v4l/vidioc-subdev-g-routing.rst      | 5 +++++
 include/uapi/linux/v4l2-subdev.h                             | 5 +++++
 3 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index 08495cc6f4a6..2f2423f676cf 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -572,7 +572,8 @@ internal pad always has a single stream only (0).
 Routes from an internal source pad to an external source pad are typically not
 modifiable but they can be activated and deactivated using the
 :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE <v4l2-subdev-routing-flags>` flag, depending
-on driver capabilities.
+on driver capabilities. This capatibility is indicated by the
+:ref:`V4L2_SUBDEV_ROUTE_FL_IMMUTABLE <v4l2-subdev-routing-flags>` flag.
 
 Interaction between routes, streams, formats and selections
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
index 08b8d17cef3f..cd7735f9104e 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
@@ -139,6 +139,11 @@ Also ``VIDIOC_SUBDEV_S_ROUTING`` may return more route than the user provided in
     * - V4L2_SUBDEV_ROUTE_FL_ACTIVE
       - 0x0001
       - The route is enabled. Set by applications.
+    * - V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
+      - 0x0002
+      - The route is immutable. Set by the driver. The
+	``V4L2_SUBDEV_ROUTE_FL_ACTIVE`` flag of an immutable route may not be
+	changed.
 
 Return Value
 ============
diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
index ca543982460c..7e501cb45e4e 100644
--- a/include/uapi/linux/v4l2-subdev.h
+++ b/include/uapi/linux/v4l2-subdev.h
@@ -200,6 +200,11 @@ struct v4l2_subdev_capability {
  * on a video node.
  */
 #define V4L2_SUBDEV_ROUTE_FL_ACTIVE		(1U << 0)
+/*
+ * Is the route immutable. The ACTIVE flag of an immutable route may not be
+ * changed.
+ */
+#define V4L2_SUBDEV_ROUTE_FL_IMMUTABLE		(1U << 1)
 
 /**
  * struct v4l2_subdev_route - A route inside a subdev
-- 
2.39.2

[PATCH v8 37/38] media: ccs: Add IMMUTABLE route flag

[Sakari Ailus]

Add immutable route flag to the routing table.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ccs/ccs-core.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
index 838daab212f2..5205d1ecf8f2 100644
--- a/drivers/media/i2c/ccs/ccs-core.c
+++ b/drivers/media/i2c/ccs/ccs-core.c
@@ -3368,12 +3368,14 @@ static int ccs_src_init_state(struct v4l2_subdev *sd,
 			.sink_pad = CCS_PAD_SINK,
 			.source_pad = CCS_PAD_SRC,
 			.source_stream = CCS_STREAM_PIXEL,
-			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
+			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE |
+				 V4L2_SUBDEV_ROUTE_FL_IMMUTABLE,
 		}, {
 			.sink_pad = CCS_PAD_META,
 			.source_pad = CCS_PAD_SRC,
 			.source_stream = CCS_STREAM_META,
-			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
+			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE |
+				 V4L2_SUBDEV_ROUTE_FL_IMMUTABLE,
 		}
 	};
 	struct v4l2_subdev_krouting routing = {
-- 
2.39.2

[PATCH v8 38/38] media: ov2740: Add IMMUTABLE route flag

[Sakari Ailus]

Add immutable route flag to the routing table.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ov2740.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/drivers/media/i2c/ov2740.c b/drivers/media/i2c/ov2740.c
index 7488b2535071..3e2585eb6ec9 100644
--- a/drivers/media/i2c/ov2740.c
+++ b/drivers/media/i2c/ov2740.c
@@ -1150,12 +1150,14 @@ static int ov2740_init_state(struct v4l2_subdev *sd,
 			.sink_pad = OV2740_PAD_PIXEL,
 			.source_pad = OV2740_PAD_SOURCE,
 			.source_stream = OV2740_STREAM_PIXEL,
-			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
+			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE |
+				 V4L2_SUBDEV_ROUTE_FL_IMMUTABLE,
 		}, {
 			.sink_pad = OV2740_PAD_META,
 			.source_pad = OV2740_PAD_SOURCE,
 			.source_stream = OV2740_STREAM_META,
-			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
+			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE |
+				 V4L2_SUBDEV_ROUTE_FL_IMMUTABLE,
 		},
 	};
 	struct v4l2_subdev_krouting routing = {
-- 
2.39.2

Re: [PATCH v8 36/38] media: v4l: Add V4L2_SUBDEV_ROUTE_FL_IMMUTABLE sub-device routing flag

[Tomi Valkeinen]

On 13/03/2024 09:25, Sakari Ailus wrote:
> Add a flag to denote immutable routes, V4L2_SUBDEV_ROUTE_FL_IMMUTABLE.
> Such routes cannot be changed and they're always active.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>   Documentation/userspace-api/media/v4l/dev-subdev.rst         | 3 ++-
>   .../userspace-api/media/v4l/vidioc-subdev-g-routing.rst      | 5 +++++
>   include/uapi/linux/v4l2-subdev.h                             | 5 +++++
>   3 files changed, 12 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> index 08495cc6f4a6..2f2423f676cf 100644
> --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> @@ -572,7 +572,8 @@ internal pad always has a single stream only (0).
>   Routes from an internal source pad to an external source pad are typically not
>   modifiable but they can be activated and deactivated using the
>   :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE <v4l2-subdev-routing-flags>` flag, depending
> -on driver capabilities.
> +on driver capabilities. This capatibility is indicated by the
> +:ref:`V4L2_SUBDEV_ROUTE_FL_IMMUTABLE <v4l2-subdev-routing-flags>` flag.
>   
>   Interaction between routes, streams, formats and selections
>   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> index 08b8d17cef3f..cd7735f9104e 100644
> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> @@ -139,6 +139,11 @@ Also ``VIDIOC_SUBDEV_S_ROUTING`` may return more route than the user provided in
>       * - V4L2_SUBDEV_ROUTE_FL_ACTIVE
>         - 0x0001
>         - The route is enabled. Set by applications.
> +    * - V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
> +      - 0x0002
> +      - The route is immutable. Set by the driver. The
> +	``V4L2_SUBDEV_ROUTE_FL_ACTIVE`` flag of an immutable route may not be
> +	changed.
>   
>   Return Value
>   ============
> diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
> index ca543982460c..7e501cb45e4e 100644
> --- a/include/uapi/linux/v4l2-subdev.h
> +++ b/include/uapi/linux/v4l2-subdev.h
> @@ -200,6 +200,11 @@ struct v4l2_subdev_capability {
>    * on a video node.
>    */
>   #define V4L2_SUBDEV_ROUTE_FL_ACTIVE		(1U << 0)
> +/*
> + * Is the route immutable. The ACTIVE flag of an immutable route may not be
> + * changed.
> + */
> +#define V4L2_SUBDEV_ROUTE_FL_IMMUTABLE		(1U << 1)
>   
>   /**
>    * struct v4l2_subdev_route - A route inside a subdev

Is the route fully immutable? The sink/source stream ID cannot be 
changed (or any new fields we might come up with in the future)?

Hmm, or would a route with different stream IDs be a, well, different 
route...

The docs here only talk about the ACTIVE flag. Would 
V4L2_SUBDEV_ROUTE_FL_ALWAYS_ACTIVE be a better name, to be more explicit 
on the meaning?

  Tomi

Re: [PATCH v8 36/38] media: v4l: Add V4L2_SUBDEV_ROUTE_FL_IMMUTABLE sub-device routing flag

[Sakari Ailus]

Huomenta Tomi,

On Wed, Mar 13, 2024 at 09:34:13AM +0200, Tomi Valkeinen wrote:
> On 13/03/2024 09:25, Sakari Ailus wrote:
> > Add a flag to denote immutable routes, V4L2_SUBDEV_ROUTE_FL_IMMUTABLE.
> > Such routes cannot be changed and they're always active.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >   Documentation/userspace-api/media/v4l/dev-subdev.rst         | 3 ++-
> >   .../userspace-api/media/v4l/vidioc-subdev-g-routing.rst      | 5 +++++
> >   include/uapi/linux/v4l2-subdev.h                             | 5 +++++
> >   3 files changed, 12 insertions(+), 1 deletion(-)
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > index 08495cc6f4a6..2f2423f676cf 100644
> > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > @@ -572,7 +572,8 @@ internal pad always has a single stream only (0).
> >   Routes from an internal source pad to an external source pad are typically not
> >   modifiable but they can be activated and deactivated using the
> >   :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE <v4l2-subdev-routing-flags>` flag, depending
> > -on driver capabilities.
> > +on driver capabilities. This capatibility is indicated by the
> > +:ref:`V4L2_SUBDEV_ROUTE_FL_IMMUTABLE <v4l2-subdev-routing-flags>` flag.
> >   Interaction between routes, streams, formats and selections
> >   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > index 08b8d17cef3f..cd7735f9104e 100644
> > --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > @@ -139,6 +139,11 @@ Also ``VIDIOC_SUBDEV_S_ROUTING`` may return more route than the user provided in
> >       * - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> >         - 0x0001
> >         - The route is enabled. Set by applications.
> > +    * - V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
> > +      - 0x0002
> > +      - The route is immutable. Set by the driver. The
> > +	``V4L2_SUBDEV_ROUTE_FL_ACTIVE`` flag of an immutable route may not be
> > +	changed.
> >   Return Value
> >   ============
> > diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
> > index ca543982460c..7e501cb45e4e 100644
> > --- a/include/uapi/linux/v4l2-subdev.h
> > +++ b/include/uapi/linux/v4l2-subdev.h
> > @@ -200,6 +200,11 @@ struct v4l2_subdev_capability {
> >    * on a video node.
> >    */
> >   #define V4L2_SUBDEV_ROUTE_FL_ACTIVE		(1U << 0)
> > +/*
> > + * Is the route immutable. The ACTIVE flag of an immutable route may not be
> > + * changed.
> > + */
> > +#define V4L2_SUBDEV_ROUTE_FL_IMMUTABLE		(1U << 1)
> >   /**
> >    * struct v4l2_subdev_route - A route inside a subdev
> 
> Is the route fully immutable? The sink/source stream ID cannot be changed
> (or any new fields we might come up with in the future)?

I think the new fields should be considered separately when they're added.
This also applies to the stream IDs, I'll add this to the documentation.

The naming of the flag is aligned with MC link flag with a similar purpose.

> 
> Hmm, or would a route with different stream IDs be a, well, different
> route...
> 
> The docs here only talk about the ACTIVE flag. Would
> V4L2_SUBDEV_ROUTE_FL_ALWAYS_ACTIVE be a better name, to be more explicit on
> the meaning?

I prefer immutable. I wonder what Laurent and Hans think.

-- 
Terveisin,

Sakari Ailus

Re: [PATCH v8 18/38] media: ccs: No need to set streaming to false in power off

[Kieran Bingham]

Quoting Sakari Ailus (2024-03-13 07:24:56)
> Streaming will have been stopped by the sensor is powered off, and so

by the time the sensor

or

before the sensor

> sensor->streaming is also false already. Do not set it as part of the
> runtime suspend callback.
> 

Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>

> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  drivers/media/i2c/ccs/ccs-core.c | 1 -
>  1 file changed, 1 deletion(-)
> 
> diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
> index e21287d50c15..671540a8ab6a 100644
> --- a/drivers/media/i2c/ccs/ccs-core.c
> +++ b/drivers/media/i2c/ccs/ccs-core.c
> @@ -1710,7 +1710,6 @@ static int ccs_power_off(struct device *dev)
>         usleep_range(5000, 5000);
>         regulator_bulk_disable(ARRAY_SIZE(ccs_regulators),
>                                sensor->regulators);
> -       sensor->streaming = false;
>  
>         return 0;
>  }
> -- 
> 2.39.2
>

Re: [PATCH v8 35/38] media: ov2740: Add support for embedded data

[Bingbu Cao]

Sakari,

Thanks for the patch.

On 3/13/24 3:25 PM, Sakari Ailus wrote:
> Add support for embedded data. This introduces two internal pads for pixel
> and embedded data streams. As the driver supports a single mode only,
> there's no need for backward compatibility in mode selection.
> 
> The embedded data is configured to be placed before the image data whereas
> after the image data is the default.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  drivers/media/i2c/ov2740.c | 150 +++++++++++++++++++++++++++++++++----
>  1 file changed, 137 insertions(+), 13 deletions(-)
> 
> diff --git a/drivers/media/i2c/ov2740.c b/drivers/media/i2c/ov2740.c
> index df57f0096e98..7488b2535071 100644
> --- a/drivers/media/i2c/ov2740.c
> +++ b/drivers/media/i2c/ov2740.c
> @@ -11,6 +11,7 @@
>  #include <linux/pm_runtime.h>
>  #include <linux/nvmem-provider.h>
>  #include <linux/regmap.h>
> +#include <media/mipi-csi2.h>
>  #include <media/v4l2-ctrls.h>
>  #include <media/v4l2-device.h>
>  #include <media/v4l2-fwnode.h>
> @@ -71,11 +72,31 @@
>  #define OV2740_REG_ISP_CTRL00		0x5000
>  /* ISP CTRL01 */
>  #define OV2740_REG_ISP_CTRL01		0x5001
> +
> +/* Embedded data line location control */
> +#define OV2740_REG_EMBEDDED_FLAG	0x5a08
> +#define OV2740_EMBEDDED_FLAG_FOOTER	BIT(2) /* otherwise it's in header */
> +#define OV2740_EMBEDDED_FLAG_MYSTERY	BIT(1)
>  /* Customer Addresses: 0x7010 - 0x710F */
>  #define CUSTOMER_USE_OTP_SIZE		0x100
>  /* OTP registers from sensor */
>  #define OV2740_REG_OTP_CUSTOMER		0x7010
>  
> +enum {
> +	OV2740_PAD_SOURCE,
> +	OV2740_PAD_PIXEL,
> +	OV2740_PAD_META,
> +	OV2740_NUM_PADS,
> +};
> +
> +enum {
> +	OV2740_STREAM_PIXEL,
> +	OV2740_STREAM_META,
> +};
> +
> +#define OV2740_META_WIDTH		100U /* 97 bytes of actual data */
> +#define OV2740_META_HEIGHT		1U
> +
>  struct nvm_data {
>  	struct nvmem_device *nvmem;
>  	struct regmap *regmap;
> @@ -513,7 +534,7 @@ static const struct ov2740_mode supported_modes_180mhz[] = {
>  
>  struct ov2740 {
>  	struct v4l2_subdev sd;
> -	struct media_pad pad;
> +	struct media_pad pads[OV2740_NUM_PADS];
>  	struct v4l2_ctrl_handler ctrl_handler;
>  
>  	/* V4L2 Controls */
> @@ -976,6 +997,11 @@ static int ov2740_enable_streams(struct v4l2_subdev *sd,
>  	if (ret)
>  		goto out_pm_put;
>  
> +	ret = ov2740_write_reg(ov2740, OV2740_REG_EMBEDDED_FLAG, 1,
> +			       OV2740_EMBEDDED_FLAG_MYSTERY);
> +	if (ret)
> +		return ret;
> +
>  	ret = ov2740_write_reg(ov2740, OV2740_REG_MODE_SELECT, 1,
>  			       OV2740_MODE_STREAMING);
>  	if (ret) {
> @@ -1013,23 +1039,49 @@ static int ov2740_disable_streams(struct v4l2_subdev *sd,
>  	return ret;
>  }
>  
> -static int ov2740_set_format(struct v4l2_subdev *sd,
> -			     struct v4l2_subdev_state *sd_state,
> -			     struct v4l2_subdev_format *fmt)
> +static int __ov2740_set_format(struct v4l2_subdev *sd,
> +			       struct v4l2_subdev_state *sd_state,
> +			       struct v4l2_mbus_framefmt *format,
> +			       enum v4l2_subdev_format_whence which,
> +			       unsigned int pad, unsigned int stream)
>  {
> +	struct v4l2_mbus_framefmt *src_pix_fmt, *src_meta_fmt, *pix_fmt,
> +		*meta_fmt;
>  	struct ov2740 *ov2740 = to_ov2740(sd);
>  	const struct ov2740_mode *mode;
>  	s32 vblank_def, h_blank;
>  
> +	/*
> +	 * Allow setting format on internal pixel pad as well as the source
> +	 * pad's pixel stream (for compatibility).
> +	 */
> +	if (pad == OV2740_PAD_SOURCE || pad == OV2740_PAD_META ||
> +	    stream == OV2740_STREAM_META) {
> +		*format = *v4l2_subdev_state_get_format(sd_state, pad, stream);
> +		return 0;
> +	}
> +
> +	pix_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_PIXEL, 0);
> +	meta_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_META, 0);
> +	src_pix_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_SOURCE,
> +						   OV2740_STREAM_PIXEL);
> +	src_meta_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_SOURCE,
> +						    OV2740_STREAM_META);
> +
>  	mode = v4l2_find_nearest_size(ov2740->supported_modes,
>  				      ov2740->supported_modes_count,
>  				      width, height,
> -				      fmt->format.width, fmt->format.height);
> +				      format->width, format->height);
> +	ov2740_update_pad_format(mode, pix_fmt);
> +	*format = *src_pix_fmt = *pix_fmt;
>  
> -	ov2740_update_pad_format(mode, &fmt->format);
> -	*v4l2_subdev_state_get_format(sd_state, fmt->pad) = fmt->format;
> +	meta_fmt->code = MEDIA_BUS_FMT_OV2740_EMBEDDED;
> +	meta_fmt->width = OV2740_META_WIDTH;
> +	meta_fmt->height = OV2740_META_HEIGHT;
> +	*src_meta_fmt = *meta_fmt;
> +	src_meta_fmt->code = MEDIA_BUS_FMT_META_10;
>  
> -	if (fmt->which == V4L2_SUBDEV_FORMAT_TRY)
> +	if (which == V4L2_SUBDEV_FORMAT_TRY)
>  		return 0;
>  
>  	ov2740->cur_mode = mode;
> @@ -1049,6 +1101,14 @@ static int ov2740_set_format(struct v4l2_subdev *sd,
>  	return 0;
>  }
>  
> +static int ov2740_set_format(struct v4l2_subdev *sd,
> +			     struct v4l2_subdev_state *sd_state,
> +			     struct v4l2_subdev_format *fmt)
> +{
> +	return __ov2740_set_format(sd, sd_state, &fmt->format, fmt->which,
> +				   fmt->pad, fmt->stream);
> +}
> +
>  static int ov2740_enum_mbus_code(struct v4l2_subdev *sd,
>  				 struct v4l2_subdev_state *sd_state,
>  				 struct v4l2_subdev_mbus_code_enum *code)
> @@ -1085,10 +1145,68 @@ static int ov2740_enum_frame_size(struct v4l2_subdev *sd,
>  static int ov2740_init_state(struct v4l2_subdev *sd,
>  			     struct v4l2_subdev_state *sd_state)
>  {
> +	struct v4l2_subdev_route routes[] = {
> +		{
> +			.sink_pad = OV2740_PAD_PIXEL,
> +			.source_pad = OV2740_PAD_SOURCE,
> +			.source_stream = OV2740_STREAM_PIXEL,
> +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> +		}, {
> +			.sink_pad = OV2740_PAD_META,
> +			.source_pad = OV2740_PAD_SOURCE,
> +			.source_stream = OV2740_STREAM_META,
> +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> +		},
> +	};
> +	struct v4l2_subdev_krouting routing = {
> +		.routes = routes,
> +		.num_routes = ARRAY_SIZE(routes),
> +	};
> +	struct v4l2_subdev_state *active_state;
> +	struct v4l2_mbus_framefmt format = { 0 };
>  	struct ov2740 *ov2740 = to_ov2740(sd);
> +	int ret;
> +
> +	ret = v4l2_subdev_set_routing(sd, sd_state, &routing);
> +	if (ret)
> +		return ret;
> +
> +	active_state = v4l2_subdev_get_locked_active_state(sd);
> +
> +	ov2740_update_pad_format(&ov2740->supported_modes[0], &format);
> +
> +	return __ov2740_set_format(sd, sd_state, &format,
> +				   active_state == sd_state ?
> +				   V4L2_SUBDEV_FORMAT_ACTIVE :
> +				   V4L2_SUBDEV_FORMAT_TRY, OV2740_PAD_PIXEL, 0);
> +}
> +
> +static int ov2740_get_frame_desc(struct v4l2_subdev *sd, unsigned int pad,
> +				 struct v4l2_mbus_frame_desc *desc)
> +{
> +	struct v4l2_mbus_frame_desc_entry *entry = desc->entry;
> +	struct v4l2_subdev_state *sd_state;
> +	struct v4l2_mbus_framefmt *fmt;
> +
> +	desc->type = V4L2_MBUS_FRAME_DESC_TYPE_CSI2;
> +
> +	sd_state = v4l2_subdev_lock_and_get_active_state(sd);
> +	fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_SOURCE,
> +					   OV2740_STREAM_PIXEL);
> +	entry->pixelcode = fmt->code;
> +	v4l2_subdev_unlock_state(sd_state);
> +
> +	entry->stream = OV2740_STREAM_PIXEL;
> +	entry->bus.csi2.dt = MIPI_CSI2_DT_RAW10;
> +	entry++;
> +	desc->num_entries++;
> +
> +	entry->pixelcode = MEDIA_BUS_FMT_META_8;

Should we change the pixelcode to META_10 also?

> +	entry->stream = OV2740_STREAM_META;
> +	entry->bus.csi2.dt = MIPI_CSI2_DT_GENERIC_LONG(1);

In the register setting array, the 0x4816 is set to 0x52 instead of
its default value 0x53, so the data type should be
MIPI_CSI2_DT_GENERIC_LONG(0) or MIPI_CSI2_DT_EMBEDDED_8B.

Or you can change back to the default settings. :)

> +	entry++;
> +	desc->num_entries++;
>  
> -	ov2740_update_pad_format(&ov2740->supported_modes[0],
> -				 v4l2_subdev_state_get_format(sd_state, 0));
>  	return 0;
>  }
>  
> @@ -1103,6 +1221,7 @@ static const struct v4l2_subdev_pad_ops ov2740_pad_ops = {
>  	.enum_frame_size = ov2740_enum_frame_size,
>  	.enable_streams = ov2740_enable_streams,
>  	.disable_streams = ov2740_disable_streams,
> +	.get_frame_desc = ov2740_get_frame_desc,
>  };
>  
>  static const struct v4l2_subdev_ops ov2740_subdev_ops = {
> @@ -1369,11 +1488,16 @@ static int ov2740_probe(struct i2c_client *client)
>  	}
>  
>  	ov2740->sd.state_lock = ov2740->ctrl_handler.lock;
> -	ov2740->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
> +	ov2740->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_STREAMS;
>  	ov2740->sd.entity.ops = &ov2740_subdev_entity_ops;
>  	ov2740->sd.entity.function = MEDIA_ENT_F_CAM_SENSOR;
> -	ov2740->pad.flags = MEDIA_PAD_FL_SOURCE;
> -	ret = media_entity_pads_init(&ov2740->sd.entity, 1, &ov2740->pad);
> +	ov2740->pads[OV2740_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
> +	ov2740->pads[OV2740_PAD_PIXEL].flags =
> +		MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL;
> +	ov2740->pads[OV2740_PAD_META].flags =
> +		MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL;
> +	ret = media_entity_pads_init(&ov2740->sd.entity,
> +				     ARRAY_SIZE(ov2740->pads), ov2740->pads);
>  	if (ret) {
>  		dev_err_probe(dev, ret, "failed to init entity pads\n");
>  		goto probe_error_v4l2_ctrl_handler_free;
> 

-- 
Best regards,
Bingbu Cao

Re: [PATCH v8 01/38] media: mc: Add INTERNAL pad flag

[Tomi Valkeinen]

On 13/03/2024 09:24, Sakari Ailus wrote:
> Internal source pads will be used as routing endpoints in V4L2
> [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
> Internal source pads are pads that have both SINK and INTERNAL flags set.
> 
> Also prevent creating links to pads that have been flagged as internal and
> initialising SOURCE pads with INTERNAL flag set.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>   .../userspace-api/media/mediactl/media-types.rst       |  8 ++++++++
>   drivers/media/mc/mc-entity.c                           | 10 ++++++++--
>   include/uapi/linux/media.h                             |  1 +
>   3 files changed, 17 insertions(+), 2 deletions(-)
> 
> diff --git a/Documentation/userspace-api/media/mediactl/media-types.rst b/Documentation/userspace-api/media/mediactl/media-types.rst
> index 6332e8395263..f55ef055bcf8 100644
> --- a/Documentation/userspace-api/media/mediactl/media-types.rst
> +++ b/Documentation/userspace-api/media/mediactl/media-types.rst
> @@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
>   .. _MEDIA-PAD-FL-SINK:
>   .. _MEDIA-PAD-FL-SOURCE:
>   .. _MEDIA-PAD-FL-MUST-CONNECT:
> +.. _MEDIA-PAD-FL-INTERNAL:
>   
>   .. flat-table:: Media pad flags
>       :header-rows:  0
> @@ -381,6 +382,13 @@ Types and flags used to represent the media graph elements
>   	  enabled links even when this flag isn't set; the absence of the flag
>   	  doesn't imply there is none.
>   
> +    *  -  ``MEDIA_PAD_FL_INTERNAL``
> +       -  The internal flag indicates an internal pad that has no external
> +	  connections. Such a pad shall not be connected with a link.
> +
> +	  The internal flag may currently be present only in a source pad where

s/source/sink/

Reviewed-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>

  Tomi


> +	  it indicates that the :ref:``stream <media-glossary-stream>``
> +	  originates from within the entity.
>   
>   One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
>   must be set for every pad.
> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> index 0e28b9a7936e..1973e9e1013e 100644
> --- a/drivers/media/mc/mc-entity.c
> +++ b/drivers/media/mc/mc-entity.c
> @@ -213,7 +213,9 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
>   		iter->index = i++;
>   
>   		if (hweight32(iter->flags & (MEDIA_PAD_FL_SINK |
> -					     MEDIA_PAD_FL_SOURCE)) != 1) {
> +					     MEDIA_PAD_FL_SOURCE)) != 1 ||
> +		    (iter->flags & MEDIA_PAD_FL_INTERNAL &&
> +		     !(iter->flags & MEDIA_PAD_FL_SINK))) {
>   			ret = -EINVAL;
>   			break;
>   		}
> @@ -1112,7 +1114,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
>   
>   	for (i = 0; i < entity->num_pads; i++) {
>   		if ((entity->pads[i].flags &
> -		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
> +		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
> +		      MEDIA_PAD_FL_INTERNAL)) != pad_type)
>   			continue;
>   
>   		if (entity->pads[i].sig_type == sig_type)
> @@ -1142,6 +1145,9 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
>   		return -EINVAL;
>   	if (WARN_ON(!(sink->pads[sink_pad].flags & MEDIA_PAD_FL_SINK)))
>   		return -EINVAL;
> +	if (WARN_ON(source->pads[source_pad].flags & MEDIA_PAD_FL_INTERNAL) ||
> +	    WARN_ON(sink->pads[sink_pad].flags & MEDIA_PAD_FL_INTERNAL))
> +		return -EINVAL;
>   
>   	link = media_add_link(&source->links);
>   	if (link == NULL)
> diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
> index 1c80b1d6bbaf..80cfd12a43fc 100644
> --- a/include/uapi/linux/media.h
> +++ b/include/uapi/linux/media.h
> @@ -208,6 +208,7 @@ struct media_entity_desc {
>   #define MEDIA_PAD_FL_SINK			(1U << 0)
>   #define MEDIA_PAD_FL_SOURCE			(1U << 1)
>   #define MEDIA_PAD_FL_MUST_CONNECT		(1U << 2)
> +#define MEDIA_PAD_FL_INTERNAL			(1U << 3)
>   
>   struct media_pad_desc {
>   	__u32 entity;		/* entity ID */

Re: [PATCH v8 02/38] media: Documentation: Add "stream" into glossary

[Tomi Valkeinen]

On 13/03/2024 09:24, Sakari Ailus wrote:
> Add term "stream" to the glossary of the Media subsystem documentation.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>   Documentation/userspace-api/media/glossary.rst | 6 ++++++
>   1 file changed, 6 insertions(+)
> 
> diff --git a/Documentation/userspace-api/media/glossary.rst b/Documentation/userspace-api/media/glossary.rst
> index 96a360edbf3b..ef0ab601b5bf 100644
> --- a/Documentation/userspace-api/media/glossary.rst
> +++ b/Documentation/userspace-api/media/glossary.rst
> @@ -173,6 +173,12 @@ Glossary
>   	An integrated circuit that integrates all components of a computer
>   	or other electronic systems.
>   
> +_media-glossary-stream:
> +    Stream
> +	A distinct flow of data (image data or metadata) from an initial source
> +	to a final sink. The initial source may be e.g. an image sensor and the
> +	final sink e.g. a memory buffer.
> +
>       V4L2 API
>   	**V4L2 userspace API**
>   

Reviewed-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>

  Tomi

Re: [PATCH v8 03/38] media: uapi: Add generic serial metadata mbus formats

[Tomi Valkeinen]

On 13/03/2024 09:24, Sakari Ailus wrote:
> Add generic serial metadata mbus formats. These formats describe data
> width and packing but not the content itself. The reason for specifying
> such formats is that the formats as such are fairly device specific but
> they are still handled by CSI-2 receiver drivers that should not be aware
> of device specific formats. What makes generic metadata formats possible
> is that these formats are parsed by software only, after capturing the
> data to system memory.
> 
> Also add a definition for "Data unit" to cover what is essentially a pixel
> but is not image data.

The CCS spec talks about legacy packing and optimized packing for 16+ 
bit formats. You cover only the "legacy" ones here. Did you look at those?

  Tomi

Re: [PATCH v8 35/38] media: ov2740: Add support for embedded data

[Julien Massot]

Hi Sakari,

On 3/13/24 08:25, Sakari Ailus wrote:
> Add support for embedded data. This introduces two internal pads for pixel
> and embedded data streams. As the driver supports a single mode only,
> there's no need for backward compatibility in mode selection.
> 
> The embedded data is configured to be placed before the image data whereas
> after the image data is the default.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>   drivers/media/i2c/ov2740.c | 150 +++++++++++++++++++++++++++++++++----
>   1 file changed, 137 insertions(+), 13 deletions(-)
> 
> diff --git a/drivers/media/i2c/ov2740.c b/drivers/media/i2c/ov2740.c
> index df57f0096e98..7488b2535071 100644
> --- a/drivers/media/i2c/ov2740.c
> +++ b/drivers/media/i2c/ov2740.c
> @@ -11,6 +11,7 @@
>   #include <linux/pm_runtime.h>
>   #include <linux/nvmem-provider.h>
>   #include <linux/regmap.h>
> +#include <media/mipi-csi2.h>
>   #include <media/v4l2-ctrls.h>
>   #include <media/v4l2-device.h>
>   #include <media/v4l2-fwnode.h>
> @@ -71,11 +72,31 @@
>   #define OV2740_REG_ISP_CTRL00		0x5000
>   /* ISP CTRL01 */
>   #define OV2740_REG_ISP_CTRL01		0x5001
> +
> +/* Embedded data line location control */
> +#define OV2740_REG_EMBEDDED_FLAG	0x5a08
> +#define OV2740_EMBEDDED_FLAG_FOOTER	BIT(2) /* otherwise it's in header */
> +#define OV2740_EMBEDDED_FLAG_MYSTERY	BIT(1)
>   /* Customer Addresses: 0x7010 - 0x710F */
>   #define CUSTOMER_USE_OTP_SIZE		0x100
>   /* OTP registers from sensor */
>   #define OV2740_REG_OTP_CUSTOMER		0x7010
>   
> +enum {
> +	OV2740_PAD_SOURCE,
> +	OV2740_PAD_PIXEL,
> +	OV2740_PAD_META,
> +	OV2740_NUM_PADS,
> +};
> +
> +enum {
> +	OV2740_STREAM_PIXEL,
> +	OV2740_STREAM_META,
> +};
> +
> +#define OV2740_META_WIDTH		100U /* 97 bytes of actual data */
> +#define OV2740_META_HEIGHT		1U
> +
>   struct nvm_data {
>   	struct nvmem_device *nvmem;
>   	struct regmap *regmap;
> @@ -513,7 +534,7 @@ static const struct ov2740_mode supported_modes_180mhz[] = {
>   
>   struct ov2740 {
>   	struct v4l2_subdev sd;
> -	struct media_pad pad;
> +	struct media_pad pads[OV2740_NUM_PADS];
>   	struct v4l2_ctrl_handler ctrl_handler;
>   
>   	/* V4L2 Controls */
> @@ -976,6 +997,11 @@ static int ov2740_enable_streams(struct v4l2_subdev *sd,
>   	if (ret)
>   		goto out_pm_put;
>   
> +	ret = ov2740_write_reg(ov2740, OV2740_REG_EMBEDDED_FLAG, 1,
> +			       OV2740_EMBEDDED_FLAG_MYSTERY);
> +	if (ret)
> +		return ret;
> +
>   	ret = ov2740_write_reg(ov2740, OV2740_REG_MODE_SELECT, 1,
>   			       OV2740_MODE_STREAMING);
>   	if (ret) {
> @@ -1013,23 +1039,49 @@ static int ov2740_disable_streams(struct v4l2_subdev *sd,
>   	return ret;
>   }
>   
> -static int ov2740_set_format(struct v4l2_subdev *sd,
> -			     struct v4l2_subdev_state *sd_state,
> -			     struct v4l2_subdev_format *fmt)
> +static int __ov2740_set_format(struct v4l2_subdev *sd,
> +			       struct v4l2_subdev_state *sd_state,
> +			       struct v4l2_mbus_framefmt *format,
> +			       enum v4l2_subdev_format_whence which,
> +			       unsigned int pad, unsigned int stream)
>   {
> +	struct v4l2_mbus_framefmt *src_pix_fmt, *src_meta_fmt, *pix_fmt,
> +		*meta_fmt;
>   	struct ov2740 *ov2740 = to_ov2740(sd);
>   	const struct ov2740_mode *mode;
>   	s32 vblank_def, h_blank;
>   
> +	/*
> +	 * Allow setting format on internal pixel pad as well as the source
> +	 * pad's pixel stream (for compatibility).
> +	 */
> +	if (pad == OV2740_PAD_SOURCE || pad == OV2740_PAD_META ||
> +	    stream == OV2740_STREAM_META) {
This is equivalent to
if (pad != OV2740_PAD_PIXEL)
Correct me if I'm wrong but this code doesn't allow to set the format on 
the source pad.

Should it be:
if ((pad == OV2740_PAD_SOURCE && stream == OV2740_STREAM_META) ||
	pad == OV2740_PAD_META) {


> +		*format = *v4l2_subdev_state_get_format(sd_state, pad, stream);
> +		return 0;
> +	}
> +
> +	pix_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_PIXEL, 0);
> +	meta_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_META, 0);
> +	src_pix_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_SOURCE,
> +						   OV2740_STREAM_PIXEL);
> +	src_meta_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_SOURCE,
> +						    OV2740_STREAM_META);
> +
>   	mode = v4l2_find_nearest_size(ov2740->supported_modes,
>   				      ov2740->supported_modes_count,
>   				      width, height,
> -				      fmt->format.width, fmt->format.height);
> +				      format->width, format->height);
> +	ov2740_update_pad_format(mode, pix_fmt);
> +	*format = *src_pix_fmt = *pix_fmt;
>   
> -	ov2740_update_pad_format(mode, &fmt->format);
> -	*v4l2_subdev_state_get_format(sd_state, fmt->pad) = fmt->format;
> +	meta_fmt->code = MEDIA_BUS_FMT_OV2740_EMBEDDED;
> +	meta_fmt->width = OV2740_META_WIDTH;
> +	meta_fmt->height = OV2740_META_HEIGHT;
> +	*src_meta_fmt = *meta_fmt;
> +	src_meta_fmt->code = MEDIA_BUS_FMT_META_10;
>   
> -	if (fmt->which == V4L2_SUBDEV_FORMAT_TRY)
> +	if (which == V4L2_SUBDEV_FORMAT_TRY)
>   		return 0;
>   
>   	ov2740->cur_mode = mode;
Regards,
-- 
Julien Massot
Senior Software Engineer
Collabora Ltd.
Platinum Building, St John's Innovation Park, Cambridge CB4 0DS, UK
Registered in England & Wales, no. 5513718

Re: [PATCH v8 04/38] media: uapi: Document which mbus format fields are valid for metadata

[Tomi Valkeinen]

On 13/03/2024 09:24, Sakari Ailus wrote:
> Now that metadata mbus formats have been added, it is necessary to define
> which fields in struct v4l2_mbus_format are applicable to them (not many).
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>   include/uapi/linux/v4l2-mediabus.h | 18 ++++++++++++------
>   1 file changed, 12 insertions(+), 6 deletions(-)
> 
> diff --git a/include/uapi/linux/v4l2-mediabus.h b/include/uapi/linux/v4l2-mediabus.h
> index 6b07b73473b5..de1d6161bf62 100644
> --- a/include/uapi/linux/v4l2-mediabus.h
> +++ b/include/uapi/linux/v4l2-mediabus.h
> @@ -19,12 +19,18 @@
>    * @width:	image width
>    * @height:	image height
>    * @code:	data format code (from enum v4l2_mbus_pixelcode)
> - * @field:	used interlacing type (from enum v4l2_field)
> - * @colorspace:	colorspace of the data (from enum v4l2_colorspace)
> - * @ycbcr_enc:	YCbCr encoding of the data (from enum v4l2_ycbcr_encoding)
> - * @hsv_enc:	HSV encoding of the data (from enum v4l2_hsv_encoding)
> - * @quantization: quantization of the data (from enum v4l2_quantization)
> - * @xfer_func:  transfer function of the data (from enum v4l2_xfer_func)
> + * @field:	used interlacing type (from enum v4l2_field), zero on metadata
> + *		mbus codes
> + * @colorspace:	colorspace of the data (from enum v4l2_colorspace), zero on
> + *		metadata mbus codes
> + * @ycbcr_enc:	YCbCr encoding of the data (from enum v4l2_ycbcr_encoding), zero
> + *		on metadata mbus codes
> + * @hsv_enc:	HSV encoding of the data (from enum v4l2_hsv_encoding), zero on
> + *		metadata mbus codes
> + * @quantization: quantization of the data (from enum v4l2_quantization), zero
> + *		on metadata mbus codes
> + * @xfer_func:  transfer function of the data (from enum v4l2_xfer_func), zero
> + *		on metadata mbus codes
>    * @flags:	flags (V4L2_MBUS_FRAMEFMT_*)
>    * @reserved:  reserved bytes that can be later used
>    */

Reviewed-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>

  Tomi

Re: [PATCH v8 08/38] media: Documentation: Document embedded data guidelines for camera sensors

[Julien Massot]

On 3/13/24 08:24, Sakari Ailus wrote:
> Document how embedded data support should be implemented for camera
> sensors, and when and how CCS embedded data format should be referenced.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>   .../media/drivers/camera-sensor.rst           | 29 +++++++++++++++++++
>   1 file changed, 29 insertions(+)
> 
> diff --git a/Documentation/userspace-api/media/drivers/camera-sensor.rst b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> index 919a50e8b9d9..92545538b855 100644
> --- a/Documentation/userspace-api/media/drivers/camera-sensor.rst
> +++ b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> @@ -102,3 +102,32 @@ register programming sequences shall initialize the :ref:`V4L2_CID_HFLIP
>   values programmed by the register sequences. The default values of these
>   controls shall be 0 (disabled). Especially these controls shall not be inverted,
>   independently of the sensor's mounting rotation.
> +
> +Embedded data
> +-------------
> +
> +Many sensors, mostly raw sensors, support embedded data which is used to convey
> +the sensor configuration for the captured frame back to the host. While CSI-2 is
> +the most common bus used by such sensors, embedded data can be available on
> +other bus types as well.
> +
> +Embedded data support shall use an internal source pad (a pad that has
> +``MEDIA_PAD_FL_SINK <MEDIA-PAD-FL-SINK>`` and ``MEDIA_PAD_FL_INTERNAL
> +<MEDIA-PAD-FL-INTERNAL>`` flags set) and route to the external pad. If embedded
> +data output can be disabled in hardware, it should be possible to disable the
> +embedded data route via ``VIDIOC_SUBDEV_S_ROUTING`` IOCTL.
> +
> +In general, changing the embedded data format from the driver-configured values
> +is not supported. The height of the metadata is hardware specific and the width
> +is that (or less of that) of the image width, as configured on the pixel data
> +stream.
> +
> +CCS and non-CCS embedded data
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Embedded data which is compliant with CCS definitions shall use ``CCS embedded
> +data format <MEDIA-BUS-FMT-CCS-EMBEDDED>``. Device specific embedded data which
> +is compliant up to MIPI CCS embedded data levels 1 or 2 only shall refer to CCS
> +embedded data formats and document the level of conformance. The rest of the
> +device specific embedded data format shall be documented in the context of the
> +data format itself.
Reviewed-by: Julien Massot <julien.massot@collabora.com>

Julien

Re: [PATCH v8 09/38] media: Documentation: v4l: Document internal source pads

[Julien Massot]

On 3/13/24 08:24, Sakari Ailus wrote:
> Document internal source pads, pads that have both SINK and INTERNAL flags
> set. Use the IMX219 camera sensor as an example.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>   .../userspace-api/media/v4l/dev-subdev.rst    | 145 ++++++++++++++++++
>   1 file changed, 145 insertions(+)
> 
> diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> index a387e8a15b8d..1808f40f63e3 100644
> --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> @@ -553,6 +553,27 @@ A stream at a specific point in the media pipeline is identified by the
>   sub-device and a (pad, stream) pair. For sub-devices that do not support
>   multiplexed streams the 'stream' field is always 0.
>   
> +.. _v4l2-subdev-internal-source-pads:
> +
> +Internal source pads and routing
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +Cases where a single sub-device source pad is traversed by multiple streams, one
> +or more of which originate from within the sub-device itself, are special as
> +there is no external sink pad for such routes. In those cases, the sources of
> +the internally generated streams are represented by internal source pads, which
> +are sink pads that have the :ref:`MEDIA_PAD_FL_INTERNAL <MEDIA-PAD-FL-INTERNAL>`
> +pad flag set.
> +
> +Internal pads have all the properties of an external pad, including formats and
> +selections. The format in this case is the source format of the stream. An
> +internal pad always has a single stream only (0).
> +
> +Routes from an internal source pad to an external source pad are typically not
> +modifiable but they can be activated and deactivated using the
> +:ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE <v4l2-subdev-routing-flags>` flag, depending
> +on driver capabilities.
> +
>   Interaction between routes, streams, formats and selections
>   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   
> @@ -668,3 +689,127 @@ To configure this pipeline, the userspace must take the following steps:
>      the configurations along the stream towards the receiver, using
>      :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each
>      stream endpoint in each sub-device.
> +
> +Internal pads setup example
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +A simple example of a multiplexed stream setup might be as follows:
> +
> +- An IMX219 camera sensor source sub-device, with one sink pad (0), one source pad
> +  (1), an internal sink pad (2) that represents the source of embedded
The pixel pad is an internal pad as well ?

> +  data. There are two routes, one from the sink pad to the source, and another
> +  from the internal sink pad to the source pad. Both streams are always active,
> +  i.e. there is no need to separately enable the embedded data stream. The
> +  sensor uses the CSI-2 bus.
> +
> +- A CSI-2 receiver in the SoC (Receiver). The receiver has a single sink pad
> +  (pad 0), connected to the bridge, and two source pads (pads 1-2), going to the
> +  DMA engine. The receiver demultiplexes the incoming streams to the source
> +  pads.
> +
> +- DMA Engines in the SoC (DMA Engine), one for each stream. Each DMA engine is
> +  connected to a single source pad in the receiver.
> +
> +The sensor, the bridge and the receiver are modeled as V4L2 sub-devices,
> +exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are
> +modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
> +
> +To configure this pipeline, the userspace must take the following steps:
> +
> +1) Set up media links between entities: connect the sensors to the bridge,
> +   bridge to the receiver, and the receiver to the DMA engines. This step does
> +   not differ from normal non-multiplexed media controller setup.
> +
> +2) Configure routing
> +
> +.. flat-table:: Camera sensor. There are no configurable routes.
> +    :header-rows: 1
> +
> +    * - Sink Pad/Stream
> +      - Source Pad/Stream
> +      - Routing Flags
> +      - Comments
> +    * - 0/0
> +      - 1/0
- 1/0
- 0/0
> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +      - Pixel data stream from the sink pad
> +    * - 2/0
> +      - 1/1
- 0/1
> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +      - Metadata stream from the internal sink pad

In latest patch "[PATCH v6 05/15] media: i2c: imx219: Add embedded data 
support"
we have:
- 0 -> source pad
- 1 -> pixel/image
- 2 -> EDATA

This is also what you did for ov2740.

With that fixed:
Reviewed-by Julien Massot <julien.massot@collabora.com>
Julien

Re: [PATCH v8 10/38] media: Documentation: Document S_ROUTING behaviour

[Julien Massot]

On 3/13/24 08:24, Sakari Ailus wrote:
> Document S_ROUTING behaviour for different devices.
> 
> Generally in devices that produce streams the streams are static and some
> can be enabled and disabled, whereas in devices that just transport them
> or write them to memory, more configurability is allowed. Document this.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>   .../userspace-api/media/v4l/dev-subdev.rst    | 24 +++++++++++++++++++
>   1 file changed, 24 insertions(+)
> 
> diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> index 1808f40f63e3..08495cc6f4a6 100644
> --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> @@ -593,6 +593,30 @@ Any configurations of a stream within a pad, such as format or selections,
>   are independent of similar configurations on other streams. This is
>   subject to change in the future.
>   
> +Device types and routing setup
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +Different kinds of sub-devices have differing behaviour for route activation,
> +depending on the hardware. In all cases, however, only routes that have the
> +``V4L2_SUBDEV_STREAM_FL_ACTIVE`` flag set are active.
> +
> +Devices generating the streams may allow enabling and disabling some of the
> +routes or the configuration is fixed. If the routes can be disabled, not
> +declaring the routes (or declaring them without
> +``VIDIOC_SUBDEV_STREAM_FL_ACTIVE`` flag set) in ``VIDIOC_SUBDEV_S_ROUTING`` will
> +disable the routes while the sub-device driver retains the streams and their
> +configuration. The ``VIDIOC_SUBDEV_S_ROUTING`` will still return such routes
> +back to the user in the routes array, with the ``V4L2_SUBDEV_STREAM_FL_ACTIVE``
> +flag unset.
> +
> +Devices transporting the streams almost always have more configurability with
> +respect to routing. Typically any route between the sub-device's sink and source
> +pads is possible, and multiple routes (usually up to certain limited number) may
> +be active simultaneously. For such devices, no routes are created by the driver
> +and user-created routes are fully replaced when ``VIDIOC_SUBDEV_S_ROUTING`` is
> +called on the sub-device. Such newly created routes have the device's default
> +configuration for format and selection rectangles.
> +
>   Configuring streams
>   ^^^^^^^^^^^^^^^^^^^
>   
Reviewed-by: Julien Massot <julien.massot@collabora.com>
Julien

Re: [PATCH v8 11/38] media: v4l: subdev: Add a function to lock two sub-device states, use it

[Julien Massot]

On 3/13/24 08:24, Sakari Ailus wrote:
> Add two new functions, v4l2_subdev_lock_states() and
> v4l2_subdev_unclock_states(), to acquire and release the state of two
> sub-devices. They differ from calling v4l2_subdev_{un,}lock_state() so
> that if the two states share the same lock, the lock is acquired only
> once.
> 
> Also use the new functions in v4l2_subdev_link_validate().
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Julien Massot <julien.massot@collabora.com>
> ---
>   drivers/media/v4l2-core/v4l2-subdev.c | 12 +++-----
>   include/media/v4l2-subdev.h           | 40 +++++++++++++++++++++++++++
>   2 files changed, 44 insertions(+), 8 deletions(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 4c6198c48dd6..6c34df19ea2f 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -1400,17 +1400,13 @@ int v4l2_subdev_link_validate(struct media_link *link)
>   
>   	states_locked = sink_state && source_state;
>   
> -	if (states_locked) {
> -		v4l2_subdev_lock_state(sink_state);
> -		v4l2_subdev_lock_state(source_state);
> -	}
> +	if (states_locked)
> +		v4l2_subdev_lock_states(sink_state, source_state);
>   
>   	ret = v4l2_subdev_link_validate_locked(link, states_locked);
>   
> -	if (states_locked) {
> -		v4l2_subdev_unlock_state(sink_state);
> -		v4l2_subdev_unlock_state(source_state);
> -	}
> +	if (states_locked)
> +		v4l2_subdev_unlock_states(sink_state, source_state);
>   
>   	return ret;
>   }
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index a9e6b8146279..9cce48365975 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -1724,6 +1724,46 @@ static inline void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state)
>   	mutex_unlock(state->lock);
>   }
>   
> +/**
> + * v4l2_subdev_lock_states - Lock two sub-device states
> + * @state1: One subdevice state
> + * @state2: The other subdevice state
> + *
> + * Locks the state of two sub-devices.
> + *
> + * The states must be unlocked with v4l2_subdev_unlock_states() after use.
> + *
> + * This differs from calling v4l2_subdev_lock_state() on both states so that if
> + * the states share the same lock, the lock is acquired only once (so no
> + * deadlock occurs). The caller is responsible for ensuring the locks will
> + * always be acquired in the same order.
> + */
> +static inline void v4l2_subdev_lock_states(struct v4l2_subdev_state *state1,
> +					   struct v4l2_subdev_state *state2)
> +{
> +	mutex_lock(state1->lock);
> +	if (state1->lock != state2->lock)
> +		mutex_lock(state2->lock);
> +}
> +
> +/**
> + * v4l2_subdev_unlock_states() - Unlock two sub-device states
> + * @state1: One subdevice state
> + * @state2: The other subdevice state
> + *
> + * Unlocks the state of two sub-devices.
> + *
> + * This differs from calling v4l2_subdev_unlock_state() on both states so that
> + * if the states share the same lock, the lock is released only once.
> + */
> +static inline void v4l2_subdev_unlock_states(struct v4l2_subdev_state *state1,
> +					     struct v4l2_subdev_state *state2)
> +{
> +	mutex_unlock(state1->lock);
> +	if (state1->lock != state2->lock)
> +		mutex_unlock(state2->lock);
> +}
> +
>   /**
>    * v4l2_subdev_get_unlocked_active_state() - Checks that the active subdev state
>    *					     is unlocked and returns it

-- 
Julien Massot
Senior Software Engineer
Collabora Ltd.
Platinum Building, St John's Innovation Park, Cambridge CB4 0DS, UK
Registered in England & Wales, no. 5513718

Re: [PATCH v8 12/38] media: v4l: subdev: Move G_ROUTING handling below S_ROUTING

[Julien Massot]

On 3/13/24 08:24, Sakari Ailus wrote:
> Move G_ROUTING IOCTL handling below that of S_ROUTING. G_ROUTING
> implementation will soon needed in handling S_ROUTING as well.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Julien Massot <julien.massot@collabora.com>
> ---
>   drivers/media/v4l2-core/v4l2-subdev.c | 54 +++++++++++++--------------
>   1 file changed, 27 insertions(+), 27 deletions(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 6c34df19ea2f..ec8de3f872b9 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -909,33 +909,6 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>   	case VIDIOC_SUBDEV_QUERYSTD:
>   		return v4l2_subdev_call(sd, video, querystd, arg);
>   
> -	case VIDIOC_SUBDEV_G_ROUTING: {
> -		struct v4l2_subdev_routing *routing = arg;
> -		struct v4l2_subdev_krouting *krouting;
> -
> -		if (!v4l2_subdev_enable_streams_api)
> -			return -ENOIOCTLCMD;
> -
> -		if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS))
> -			return -ENOIOCTLCMD;
> -
> -		memset(routing->reserved, 0, sizeof(routing->reserved));
> -
> -		krouting = &state->routing;
> -
> -		if (routing->num_routes < krouting->num_routes) {
> -			routing->num_routes = krouting->num_routes;
> -			return -ENOSPC;
> -		}
> -
> -		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
> -		       krouting->routes,
> -		       krouting->num_routes * sizeof(*krouting->routes));
> -		routing->num_routes = krouting->num_routes;
> -
> -		return 0;
> -	}
> -
>   	case VIDIOC_SUBDEV_S_ROUTING: {
>   		struct v4l2_subdev_routing *routing = arg;
>   		struct v4l2_subdev_route *routes =
> @@ -984,6 +957,33 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>   					routing->which, &krouting);
>   	}
>   
> +	case VIDIOC_SUBDEV_G_ROUTING: {
> +		struct v4l2_subdev_routing *routing = arg;
> +		struct v4l2_subdev_krouting *krouting;
> +
> +		if (!v4l2_subdev_enable_streams_api)
> +			return -ENOIOCTLCMD;
> +
> +		if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS))
> +			return -ENOIOCTLCMD;
> +
> +		memset(routing->reserved, 0, sizeof(routing->reserved));
> +
> +		krouting = &state->routing;
> +
> +		if (routing->num_routes < krouting->num_routes) {
> +			routing->num_routes = krouting->num_routes;
> +			return -ENOSPC;
> +		}
> +
> +		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
> +		       krouting->routes,
> +		       krouting->num_routes * sizeof(*krouting->routes));
> +		routing->num_routes = krouting->num_routes;
> +
> +		return 0;
> +	}
> +
>   	case VIDIOC_SUBDEV_G_CLIENT_CAP: {
>   		struct v4l2_subdev_client_capability *client_cap = arg;
>   

-- 
Julien Massot
Senior Software Engineer
Collabora Ltd.
Platinum Building, St John's Innovation Park, Cambridge CB4 0DS, UK
Registered in England & Wales, no. 5513718

Re: [PATCH v8 13/38] media: v4l: subdev: Copy argument back to user also for S_ROUTING

[Julien Massot]

On 3/13/24 08:24, Sakari Ailus wrote:
> As the user needs to know what went wrong for S_ROUTING, copy array
> arguments back to the user.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Julien Massot <julien.massot@collabora.com>
> ---
>   drivers/media/v4l2-core/v4l2-ioctl.c | 9 ++++++---
>   1 file changed, 6 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index d125d23e4e61..95bd56145d38 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -3415,11 +3415,14 @@ video_usercopy(struct file *file, unsigned int orig_cmd, unsigned long arg,
>   	 * FIXME: subdev IOCTLS are partially handled here and partially in
>   	 * v4l2-subdev.c and the 'always_copy' flag can only be set for IOCTLS
>   	 * defined here as part of the 'v4l2_ioctls' array. As
> -	 * VIDIOC_SUBDEV_G_ROUTING needs to return results to applications even
> -	 * in case of failure, but it is not defined here as part of the
> +	 * VIDIOC_SUBDEV_[GS]_ROUTING needs to return results to applications
> +	 * even in case of failure, but it is not defined here as part of the
>   	 * 'v4l2_ioctls' array, insert an ad-hoc check to address that.
>   	 */
> -	if (err < 0 && !always_copy && cmd != VIDIOC_SUBDEV_G_ROUTING)
> +	if (cmd == VIDIOC_SUBDEV_G_ROUTING || cmd == VIDIOC_SUBDEV_S_ROUTING)
> +		always_copy = true;
> +
> +	if (err < 0 && !always_copy)
>   		goto out;
>   
>   	if (has_array_args) {

-- 
Julien Massot
Senior Software Engineer
Collabora Ltd.
Platinum Building, St John's Innovation Park, Cambridge CB4 0DS, UK
Registered in England & Wales, no. 5513718

Re: [PATCH v8 17/38] media: v4l: subdev: Add trivial set_routing support

[Julien Massot]

On 3/13/24 08:24, Sakari Ailus wrote:
> Add trivial S_ROUTING IOCTL support for drivers where routing is static.
> Essentially this means returning the same information G_ROUTING call would
> have done.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Julien Massot <julien.massot@collabora.com>
> ---
>   drivers/media/v4l2-core/v4l2-subdev.c | 14 ++++++++++++++
>   1 file changed, 14 insertions(+)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index a6107e440ef0..c8c435df92c8 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -930,6 +930,20 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>   
>   		memset(routing->reserved, 0, sizeof(routing->reserved));
>   
> +		/*
> +		 * If the driver doesn't support setting routing, just return
> +		 * the routing table here.
> +		 */
> +		if (!v4l2_subdev_has_op(sd, pad, set_routing)) {
> +			memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
> +			       state->routing.routes,
> +			       min(state->routing.num_routes, routing->len_routes) *
> +			       sizeof(*state->routing.routes));
> +			routing->num_routes = state->routing.num_routes;
> +
> +			return 0;
> +		}
> +
>   		for (i = 0; i < routing->num_routes; ++i) {
>   			const struct v4l2_subdev_route *route = &routes[i];
>   			const struct media_pad *pads = sd->entity.pads;

-- 
Julien Massot
Senior Software Engineer
Collabora Ltd.
Platinum Building, St John's Innovation Park, Cambridge CB4 0DS, UK
Registered in England & Wales, no. 5513718

Re: [PATCH v8 20/38] media: ccs: Track streaming state

[Julien Massot]

On 3/13/24 08:24, Sakari Ailus wrote:
> With enable_streams and disable_streams, the driver for a device where
> streams are not independently started and stopped needs to maintain state
> information on streams that have been requested to be started. Do that
> now.
> 
> In the future, a helper function in the framework is a desirable way to do
> this instead.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Julien Massot <julien.massot@collabora.com>
> ---
>   drivers/media/i2c/ccs/ccs-core.c | 13 ++++++++++---
>   drivers/media/i2c/ccs/ccs.h      |  2 +-
>   2 files changed, 11 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
> index 9e70946653e9..0cd8ee957655 100644
> --- a/drivers/media/i2c/ccs/ccs-core.c
> +++ b/drivers/media/i2c/ccs/ccs-core.c
> @@ -1766,6 +1766,11 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
>   	if (pad != CCS_PAD_SRC)
>   		return -EINVAL;
>   
> +	if (sensor->streaming) {
> +		sensor->streaming |= streams_mask;
> +		return 0;
> +	}
> +
>   	rval = ccs_pm_get_init(sensor);
>   	if (rval)
>   		return rval;
> @@ -1887,7 +1892,7 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
>   
>   	rval = ccs_write(sensor, MODE_SELECT, CCS_MODE_SELECT_STREAMING);
>   
> -	sensor->streaming = true;
> +	sensor->streaming |= streams_mask;
>   
>   	return 0;
>   
> @@ -1909,6 +1914,10 @@ static int ccs_disable_streams(struct v4l2_subdev *subdev,
>   	if (pad != CCS_PAD_SRC)
>   		return -EINVAL;
>   
> +	sensor->streaming &= ~streams_mask;
> +	if (sensor->streaming)
> +		return 0;
> +
>   	rval = ccs_write(sensor, MODE_SELECT, CCS_MODE_SELECT_SOFTWARE_STANDBY);
>   	if (rval)
>   		return rval;
> @@ -1917,7 +1926,6 @@ static int ccs_disable_streams(struct v4l2_subdev *subdev,
>   	if (rval)
>   		dev_err(&client->dev, "post_streamoff quirks failed\n");
>   
> -	sensor->streaming = false;
>   	pm_runtime_mark_last_busy(&client->dev);
>   	pm_runtime_put_autosuspend(&client->dev);
>   
> @@ -3525,7 +3533,6 @@ static int ccs_probe(struct i2c_client *client)
>   		goto out_cleanup;
>   	}
>   
> -	sensor->streaming = false;
>   	sensor->dev_init_done = true;
>   	sensor->handler_setup_needed = true;
>   
> diff --git a/drivers/media/i2c/ccs/ccs.h b/drivers/media/i2c/ccs/ccs.h
> index 096573845a10..4725e6eca8d0 100644
> --- a/drivers/media/i2c/ccs/ccs.h
> +++ b/drivers/media/i2c/ccs/ccs.h
> @@ -236,7 +236,7 @@ struct ccs_sensor {
>   	u16 image_start; /* image data start line */
>   	u16 visible_pixel_start; /* start pixel of the visible image */
>   
> -	bool streaming;
> +	u8 streaming;
>   	bool dev_init_done;
>   	bool handler_setup_needed;
>   	u8 compressed_min_bpp;

-- 
Julien Massot
Senior Software Engineer
Collabora Ltd.
Platinum Building, St John's Innovation Park, Cambridge CB4 0DS, UK
Registered in England & Wales, no. 5513718

Re: [PATCH v8 21/38] media: ccs: Move ccs_validate_csi_data_format up

[Julien Massot]

On 3/13/24 08:24, Sakari Ailus wrote:
> ccs_validate_csi_data_format() will soon be needed elsewhere, above its
> current location. Move it up.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Julien Massot <julien.massot@collabora.com>
> ---
>   drivers/media/i2c/ccs/ccs-core.c | 28 ++++++++++++++--------------
>   1 file changed, 14 insertions(+), 14 deletions(-)
> 
> diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
> index 0cd8ee957655..0efbc63534bc 100644
> --- a/drivers/media/i2c/ccs/ccs-core.c
> +++ b/drivers/media/i2c/ccs/ccs-core.c
> @@ -1979,6 +1979,20 @@ static int ccs_post_streamoff(struct v4l2_subdev *subdev)
>   	return pm_runtime_put(&client->dev);
>   }
>   
> +static const struct ccs_csi_data_format
> +*ccs_validate_csi_data_format(struct ccs_sensor *sensor, u32 code)
> +{
> +	unsigned int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(ccs_csi_data_formats); i++) {
> +		if (sensor->mbus_frame_fmts & (1 << i) &&
> +		    ccs_csi_data_formats[i].code == code)
> +			return &ccs_csi_data_formats[i];
> +	}
> +
> +	return sensor->csi_format;
> +}
> +
>   static int ccs_enum_mbus_code(struct v4l2_subdev *subdev,
>   			      struct v4l2_subdev_state *sd_state,
>   			      struct v4l2_subdev_mbus_code_enum *code)
> @@ -2114,20 +2128,6 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
>   	}
>   }
>   
> -static const struct ccs_csi_data_format
> -*ccs_validate_csi_data_format(struct ccs_sensor *sensor, u32 code)
> -{
> -	unsigned int i;
> -
> -	for (i = 0; i < ARRAY_SIZE(ccs_csi_data_formats); i++) {
> -		if (sensor->mbus_frame_fmts & (1 << i) &&
> -		    ccs_csi_data_formats[i].code == code)
> -			return &ccs_csi_data_formats[i];
> -	}
> -
> -	return sensor->csi_format;
> -}
> -
>   static int ccs_set_format_source(struct v4l2_subdev *subdev,
>   				 struct v4l2_subdev_state *sd_state,
>   				 struct v4l2_subdev_format *fmt)

-- 
Julien Massot
Senior Software Engineer
Collabora Ltd.
Platinum Building, St John's Innovation Park, Cambridge CB4 0DS, UK
Registered in England & Wales, no. 5513718

Re: [PATCH v8 22/38] media: ccs: Support frame descriptors

[Julien Massot]

On 3/13/24 08:25, Sakari Ailus wrote:
> Provide information on the frame layout using frame descriptors.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Julien Massot <julien.massot@collabora.com>
> ---
>   drivers/media/i2c/ccs/ccs-core.c  | 60 +++++++++++++++++++++++++++++++
>   drivers/media/i2c/ccs/ccs-quirk.h |  7 ++++
>   drivers/media/i2c/ccs/ccs.h       |  4 +++
>   3 files changed, 71 insertions(+)
> 
> diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
> index 0efbc63534bc..9cc2080b73ec 100644
> --- a/drivers/media/i2c/ccs/ccs-core.c
> +++ b/drivers/media/i2c/ccs/ccs-core.c
> @@ -25,6 +25,7 @@
>   #include <linux/slab.h>
>   #include <linux/smiapp.h>
>   #include <linux/v4l2-mediabus.h>
> +#include <media/mipi-csi2.h>
>   #include <media/v4l2-cci.h>
>   #include <media/v4l2-device.h>
>   #include <media/v4l2-fwnode.h>
> @@ -245,6 +246,33 @@ static int ccs_read_all_limits(struct ccs_sensor *sensor)
>   	return ret;
>   }
>   
> +static u8 ccs_mipi_csi2_data_type(unsigned int bpp)
> +{
> +	switch (bpp) {
> +	case 6:
> +		return MIPI_CSI2_DT_RAW6;
> +	case 7:
> +		return MIPI_CSI2_DT_RAW7;
> +	case 8:
> +		return MIPI_CSI2_DT_RAW8;
> +	case 10:
> +		return MIPI_CSI2_DT_RAW10;
> +	case 12:
> +		return MIPI_CSI2_DT_RAW12;
> +	case 14:
> +		return MIPI_CSI2_DT_RAW14;
> +	case 16:
> +		return MIPI_CSI2_DT_RAW16;
> +	case 20:
> +		return MIPI_CSI2_DT_RAW20;
> +	case 24:
> +		return MIPI_CSI2_DT_RAW24;
> +	default:
> +		WARN_ON(1);
> +		return 0;
> +	}
> +}
> +
>   static int ccs_read_frame_fmt(struct ccs_sensor *sensor)
>   {
>   	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
> @@ -2632,6 +2660,37 @@ static int ccs_set_selection(struct v4l2_subdev *subdev,
>   	return ret;
>   }
>   
> +static int ccs_get_frame_desc(struct v4l2_subdev *subdev, unsigned int pad,
> +				 struct v4l2_mbus_frame_desc *desc)
> +{
> +	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
> +	struct v4l2_mbus_frame_desc_entry *entry = desc->entry;
> +
> +	if (ccs_has_quirk(sensor, frame_desc))
> +		return ccs_call_quirk(sensor, frame_desc, desc);
> +
> +	switch (sensor->hwcfg.csi_signalling_mode) {
> +	case CCS_CSI_SIGNALING_MODE_CSI_2_DPHY:
> +	case CCS_CSI_SIGNALING_MODE_CSI_2_CPHY:
> +		desc->type = V4L2_MBUS_FRAME_DESC_TYPE_CSI2;
> +		break;
> +	default:
> +		/* FIXME: CCP2 support */
> +		return -EINVAL;
> +	}
> +
> +	entry->pixelcode = sensor->csi_format->code;
> +	entry->stream = CCS_STREAM_PIXEL;
> +	entry->bus.csi2.dt =
> +		sensor->csi_format->width == sensor->csi_format->compressed ?
> +		ccs_mipi_csi2_data_type(sensor->csi_format->compressed) :
> +		CCS_DEFAULT_COMPRESSED_DT;
> +	entry++;
> +	desc->num_entries++;
> +
> +	return 0;
> +}
> +
>   static int ccs_get_skip_frames(struct v4l2_subdev *subdev, u32 *frames)
>   {
>   	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
> @@ -3054,6 +3113,7 @@ static const struct v4l2_subdev_pad_ops ccs_pad_ops = {
>   	.set_selection = ccs_set_selection,
>   	.enable_streams = ccs_enable_streams,
>   	.disable_streams = ccs_disable_streams,
> +	.get_frame_desc = ccs_get_frame_desc,
>   };
>   
>   static const struct v4l2_subdev_sensor_ops ccs_sensor_ops = {
> diff --git a/drivers/media/i2c/ccs/ccs-quirk.h b/drivers/media/i2c/ccs/ccs-quirk.h
> index 392c97109617..3e1d9eaa33fa 100644
> --- a/drivers/media/i2c/ccs/ccs-quirk.h
> +++ b/drivers/media/i2c/ccs/ccs-quirk.h
> @@ -36,6 +36,7 @@ struct ccs_sensor;
>    *			 access may be done by the caller (default read
>    *			 value is zero), else negative error code on error
>    * @flags: Quirk flags
> + * @frame_desc: Obtain the frame descriptor
>    */
>   struct ccs_quirk {
>   	int (*limits)(struct ccs_sensor *sensor);
> @@ -46,6 +47,8 @@ struct ccs_quirk {
>   	int (*init)(struct ccs_sensor *sensor);
>   	int (*reg_access)(struct ccs_sensor *sensor, bool write, u32 *reg,
>   			  u32 *val);
> +	int (*frame_desc)(struct ccs_sensor *sensor,
> +			  struct v4l2_mbus_frame_desc *desc);
>   	unsigned long flags;
>   };
>   
> @@ -62,6 +65,10 @@ struct ccs_reg_8 {
>   		.val = _val,		\
>   	}
>   
> +#define ccs_has_quirk(sensor, _quirk)					\
> +	((sensor)->minfo.quirk &&					\
> +	 (sensor)->minfo.quirk->_quirk)
> +
>   #define ccs_call_quirk(sensor, _quirk, ...)				\
>   	((sensor)->minfo.quirk &&					\
>   	 (sensor)->minfo.quirk->_quirk ?				\
> diff --git a/drivers/media/i2c/ccs/ccs.h b/drivers/media/i2c/ccs/ccs.h
> index 4725e6eca8d0..adb152366ea2 100644
> --- a/drivers/media/i2c/ccs/ccs.h
> +++ b/drivers/media/i2c/ccs/ccs.h
> @@ -46,6 +46,8 @@
>   
>   #define CCS_COLOUR_COMPONENTS		4
>   
> +#define CCS_DEFAULT_COMPRESSED_DT	0x30
> +
>   #define SMIAPP_NAME			"smiapp"
>   #define CCS_NAME			"ccs"
>   
> @@ -175,6 +177,8 @@ struct ccs_csi_data_format {
>   #define CCS_PAD_SRC			1
>   #define CCS_PADS			2
>   
> +#define CCS_STREAM_PIXEL		0
> +
>   struct ccs_binning_subtype {
>   	u8 horizontal:4;
>   	u8 vertical:4;

-- 
Julien Massot
Senior Software Engineer
Collabora Ltd.
Platinum Building, St John's Innovation Park, Cambridge CB4 0DS, UK
Registered in England & Wales, no. 5513718

Re: [PATCH v8 23/38] media: uapi: ccs: Add media bus code for MIPI CCS embedded data

[Julien Massot]

On 3/13/24 08:25, Sakari Ailus wrote:
> Add new MIPI CCS embedded data media bus code
> (MEDIA_BUS_FMT_CCS_EMBEDDED).
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Julien Massot <julien.massot@collabora.com>
> ---
>   .../media/v4l/subdev-formats.rst              | 28 +++++++++++++++++++
>   include/uapi/linux/media-bus-format.h         |  3 ++
>   2 files changed, 31 insertions(+)
> 
> diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> index cbd475f7cae9..c8f982411e70 100644
> --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
> +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> @@ -8564,3 +8564,31 @@ and finally the bit number in subscript. "X" indicates a padding bit.
>         - X
>         - X
>         - X
> +
> +.. _MEDIA-BUS-FMT-CCS-EMBEDDED:
> +
> +MIPI CCS Embedded Data Formats
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +`MIPI CCS <https://www.mipi.org/specifications/camera-command-set>`_ defines a
> +metadata format for sensor embedded data, which is used to store the register
> +configuration used for capturing a given frame. The format is defined in the CCS
> +specification. The media bus code for this format is
> +``MEDIA_BUS_FMT_CCS_EMBEDDED``.
> +
> +The CCS embedded data format definition includes three levels:
> +
> +1. Padding within CSI-2 bus :ref:`Data unit <media-glossary-data-unit>` as
> +   documented in the MIPI CCS specification.
> +
> +2. The tagged data format as documented in the MIPI CCS specification.
> +
> +3. Register addresses and register documentation as documented in the MIPI CCS
> +   specification.
> +
> +The format definition shall be used only by devices that fulfill all three
> +levels above.
> +
> +This mbus code are only used for "2-byte simplified tagged data format" (code
> +0xa) but their use may be extended further in the future, to cover other CCS
> +embedded data format codes.
> diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
> index d4c1d991014b..03f7e9ab517b 100644
> --- a/include/uapi/linux/media-bus-format.h
> +++ b/include/uapi/linux/media-bus-format.h
> @@ -183,4 +183,7 @@
>   #define MEDIA_BUS_FMT_META_20			0x8006
>   #define MEDIA_BUS_FMT_META_24			0x8007
>   
> +/* Specific metadata formats. Next is 0x9002. */
> +#define MEDIA_BUS_FMT_CCS_EMBEDDED		0x9001
> +
>   #endif /* __LINUX_MEDIA_BUS_FORMAT_H */

-- 
Julien Massot
Senior Software Engineer
Collabora Ltd.
Platinum Building, St John's Innovation Park, Cambridge CB4 0DS, UK
Registered in England & Wales, no. 5513718

Re: [PATCH v8 37/38] media: ccs: Add IMMUTABLE route flag

[Julien Massot]

On 3/13/24 08:25, Sakari Ailus wrote:
> Add immutable route flag to the routing table.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Julien Massot <julien.massot@collabora.com>
> ---
>   drivers/media/i2c/ccs/ccs-core.c | 6 ++++--
>   1 file changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
> index 838daab212f2..5205d1ecf8f2 100644
> --- a/drivers/media/i2c/ccs/ccs-core.c
> +++ b/drivers/media/i2c/ccs/ccs-core.c
> @@ -3368,12 +3368,14 @@ static int ccs_src_init_state(struct v4l2_subdev *sd,
>   			.sink_pad = CCS_PAD_SINK,
>   			.source_pad = CCS_PAD_SRC,
>   			.source_stream = CCS_STREAM_PIXEL,
> -			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE |
> +				 V4L2_SUBDEV_ROUTE_FL_IMMUTABLE,
>   		}, {
>   			.sink_pad = CCS_PAD_META,
>   			.source_pad = CCS_PAD_SRC,
>   			.source_stream = CCS_STREAM_META,
> -			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE |
> +				 V4L2_SUBDEV_ROUTE_FL_IMMUTABLE,
>   		}
>   	};
>   	struct v4l2_subdev_krouting routing = {

-- 
Julien Massot
Senior Software Engineer
Collabora Ltd.
Platinum Building, St John's Innovation Park, Cambridge CB4 0DS, UK
Registered in England & Wales, no. 5513718

Re: [PATCH v8 32/38] media: uapi: Add media bus code for ov2740 embedded data

[Julien Massot]

On 3/13/24 08:25, Sakari Ailus wrote:
> Add a media bus code for ov2740 camera sensor embedded data and document
> it.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Julien Massot <julien.massot@collabora.com>
> ---
>   .../media/v4l/subdev-formats.rst              | 66 +++++++++++++++++++
>   include/uapi/linux/media-bus-format.h         |  3 +-
>   2 files changed, 68 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> index ca4da6a400ff..a875868ed951 100644
> --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
> +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> @@ -8594,3 +8594,69 @@ This mbus code are only used for "2-byte simplified tagged data format" (code
>   embedded data format codes.
>   
>   Also see :ref:`CCS driver documentation <media-ccs-routes>`.
> +
> +Omnivision OV2740 Embedded Data Format
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +The Omnivision OV2740 camera sensor produces the following embedded data format.
> +
> +.. flat-table:: Omnivision OV2740 Embedded Data Format. Octets at indices marked
> +                reserved or unused have been omitted from the table.
> +    :header-rows: 1
> +
> +    * - Byte
> +      - Concent description
> +    * - 0
> +      - Sensor info
> +    * - 4
> +      - Analogue gain (bits 10--8)
> +    * - 5
> +      - Analogue gain (bits 7--0)
> +    * - 6
> +      - Coarse integration time (bits 15--8)
> +    * - 7
> +      - Coarse integration time (bits 7--0)
> +    * - 10
> +      - Dpc correction threshold (bits 9--2)
> +    * - 15
> +      - Output image width (bits 15--8)
> +    * - 16
> +      - Output image width (bits 7--0)
> +    * - 17
> +      - Output image height (bits 15--8)
> +    * - 18
> +      - Output image height (bits 7--0)
> +    * - 23
> +      - MIPI header revision number
> +    * - 31
> +      - Vertical (bit 1) and horizontal flip (bit 0)
> +    * - 32
> +      - Frame duration A
> +    * - 33
> +      - Frame duration B
> +    * - 34
> +      - Context count
> +    * - 35
> +      - Context select
> +    * - 54
> +      - Data pedestal (bits 9--2)
> +    * - 63
> +      - Frame average (bits 9--2)
> +    * - 64
> +      - Digital gain red
> +    * - 65
> +      - Digital gain red
> +    * - 66
> +      - Digital gain greenr
> +    * - 67
> +      - Digital gain greenr
> +    * - 68
> +      - Digital gain blue
> +    * - 69
> +      - Digital gain blue
> +    * - 70
> +      - Digital gain greenb
> +    * - 71
> +      - Digital gain greenb
> +    * - 89
> +      - Frame counter
> diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
> index 03f7e9ab517b..13e68c2ccb61 100644
> --- a/include/uapi/linux/media-bus-format.h
> +++ b/include/uapi/linux/media-bus-format.h
> @@ -183,7 +183,8 @@
>   #define MEDIA_BUS_FMT_META_20			0x8006
>   #define MEDIA_BUS_FMT_META_24			0x8007
>   
> -/* Specific metadata formats. Next is 0x9002. */
> +/* Specific metadata formats. Next is 0x9003. */
>   #define MEDIA_BUS_FMT_CCS_EMBEDDED		0x9001
> +#define MEDIA_BUS_FMT_OV2740_EMBEDDED		0x9002
>   
>   #endif /* __LINUX_MEDIA_BUS_FORMAT_H */

-- 
Julien Massot
Senior Software Engineer
Collabora Ltd.
Platinum Building, St John's Innovation Park, Cambridge CB4 0DS, UK
Registered in England & Wales, no. 5513718

Re: [PATCH v8 33/38] media: ov2740: Switch to {enable,disable}_streams

[Julien Massot]

On 3/13/24 08:25, Sakari Ailus wrote:
> Switch from s_stream to enable_streams and disable_streams callbacks.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Julien Massot <julien.massot@collabora.com>
> ---
>   drivers/media/i2c/ov2740.c | 72 +++++++++++++++++---------------------
>   1 file changed, 32 insertions(+), 40 deletions(-)
> 
> diff --git a/drivers/media/i2c/ov2740.c b/drivers/media/i2c/ov2740.c
> index 552935ccb4a9..44c6724a102c 100644
> --- a/drivers/media/i2c/ov2740.c
> +++ b/drivers/media/i2c/ov2740.c
> @@ -919,16 +919,23 @@ static int ov2740_load_otp_data(struct nvm_data *nvm)
>   	return ret;
>   }
>   
> -static int ov2740_start_streaming(struct ov2740 *ov2740)
> +static int ov2740_enable_streams(struct v4l2_subdev *sd,
> +				 struct v4l2_subdev_state *state, u32 pad,
> +				 u64 streams_mask)
>   {
> -	struct i2c_client *client = v4l2_get_subdevdata(&ov2740->sd);
> +	struct i2c_client *client = v4l2_get_subdevdata(sd);
> +	struct ov2740 *ov2740 = to_ov2740(sd);
>   	const struct ov2740_reg_list *reg_list;
>   	int link_freq_index;
>   	int ret;
>   
> +	ret = pm_runtime_resume_and_get(&client->dev);
> +	if (ret < 0)
> +		return ret;
> +
>   	ret = ov2740_identify_module(ov2740);
>   	if (ret)
> -		return ret;
> +		goto out_pm_put;
>   
>   	if (ov2740->nvm)
>   		ov2740_load_otp_data(ov2740->nvm);
> @@ -937,7 +944,7 @@ static int ov2740_start_streaming(struct ov2740 *ov2740)
>   	ret = ov2740_write_reg(ov2740, 0x0103, 1, 0x01);
>   	if (ret) {
>   		dev_err(&client->dev, "failed to reset\n");
> -		return ret;
> +		goto out_pm_put;
>   	}
>   
>   	usleep_range(10000, 15000);
> @@ -947,64 +954,47 @@ static int ov2740_start_streaming(struct ov2740 *ov2740)
>   	ret = ov2740_write_reg_list(ov2740, reg_list);
>   	if (ret) {
>   		dev_err(&client->dev, "failed to set plls\n");
> -		return ret;
> +		goto out_pm_put;
>   	}
>   
>   	reg_list = &ov2740->cur_mode->reg_list;
>   	ret = ov2740_write_reg_list(ov2740, reg_list);
>   	if (ret) {
>   		dev_err(&client->dev, "failed to set mode\n");
> -		return ret;
> +		goto out_pm_put;
>   	}
>   
>   	ret = __v4l2_ctrl_handler_setup(ov2740->sd.ctrl_handler);
>   	if (ret)
> -		return ret;
> +		goto out_pm_put;
>   
>   	ret = ov2740_write_reg(ov2740, OV2740_REG_MODE_SELECT, 1,
>   			       OV2740_MODE_STREAMING);
> -	if (ret)
> +	if (ret) {
>   		dev_err(&client->dev, "failed to start streaming\n");
> +		goto out_pm_put;
> +	}
>   
> -	return ret;
> -}
> +	return 0;
>   
> -static void ov2740_stop_streaming(struct ov2740 *ov2740)
> -{
> -	struct i2c_client *client = v4l2_get_subdevdata(&ov2740->sd);
> +out_pm_put:
> +	pm_runtime_put(&client->dev);
>   
> -	if (ov2740_write_reg(ov2740, OV2740_REG_MODE_SELECT, 1,
> -			     OV2740_MODE_STANDBY))
> -		dev_err(&client->dev, "failed to stop streaming\n");
> +	return ret;
>   }
>   
> -static int ov2740_set_stream(struct v4l2_subdev *sd, int enable)
> +static int ov2740_disable_streams(struct v4l2_subdev *sd,
> +				  struct v4l2_subdev_state *state, u32 pad,
> +				  u64 streams_mask)
>   {
> -	struct ov2740 *ov2740 = to_ov2740(sd);
>   	struct i2c_client *client = v4l2_get_subdevdata(sd);
> -	struct v4l2_subdev_state *sd_state;
> -	int ret = 0;
> -
> -	sd_state = v4l2_subdev_lock_and_get_active_state(&ov2740->sd);
> -
> -	if (enable) {
> -		ret = pm_runtime_resume_and_get(&client->dev);
> -		if (ret < 0)
> -			goto out_unlock;
> +	struct ov2740 *ov2740 = to_ov2740(sd);
> +	int ret;
>   
> -		ret = ov2740_start_streaming(ov2740);
> -		if (ret) {
> -			enable = 0;
> -			ov2740_stop_streaming(ov2740);
> -			pm_runtime_put(&client->dev);
> -		}
> -	} else {
> -		ov2740_stop_streaming(ov2740);
> -		pm_runtime_put(&client->dev);
> -	}
> +	ret = ov2740_write_reg(ov2740, OV2740_REG_MODE_SELECT, 1,
> +			       OV2740_MODE_STANDBY);
>   
> -out_unlock:
> -	v4l2_subdev_unlock_state(sd_state);
> +	pm_runtime_put(&client->dev);
>   
>   	return ret;
>   }
> @@ -1089,7 +1079,7 @@ static int ov2740_init_state(struct v4l2_subdev *sd,
>   }
>   
>   static const struct v4l2_subdev_video_ops ov2740_video_ops = {
> -	.s_stream = ov2740_set_stream,
> +	.s_stream = v4l2_subdev_s_stream_helper,
>   };
>   
>   static const struct v4l2_subdev_pad_ops ov2740_pad_ops = {
> @@ -1097,6 +1087,8 @@ static const struct v4l2_subdev_pad_ops ov2740_pad_ops = {
>   	.set_fmt = ov2740_set_format,
>   	.enum_mbus_code = ov2740_enum_mbus_code,
>   	.enum_frame_size = ov2740_enum_frame_size,
> +	.enable_streams = ov2740_enable_streams,
> +	.disable_streams = ov2740_disable_streams,
>   };
>   
>   static const struct v4l2_subdev_ops ov2740_subdev_ops = {

-- 
Julien Massot
Senior Software Engineer
Collabora Ltd.
Platinum Building, St John's Innovation Park, Cambridge CB4 0DS, UK
Registered in England & Wales, no. 5513718

Re: [PATCH v8 34/38] media: ov2740: Track streaming state

[Julien Massot]

On 3/13/24 08:25, Sakari Ailus wrote:
> With enable_streams and disable_streams, the driver for a device where
> streams are not independently started and stopped needs to maintain state
> information on streams that have been requested to be started. Do that
> now.
> 
> In the future, a helper function in the framework is a desirable way to do
> this instead.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Julien Massot <julien.massot@collabora.com>
> ---
>   drivers/media/i2c/ov2740.c | 14 ++++++++++++++
>   1 file changed, 14 insertions(+)
> 
> diff --git a/drivers/media/i2c/ov2740.c b/drivers/media/i2c/ov2740.c
> index 44c6724a102c..df57f0096e98 100644
> --- a/drivers/media/i2c/ov2740.c
> +++ b/drivers/media/i2c/ov2740.c
> @@ -539,6 +539,9 @@ struct ov2740 {
>   
>   	/* True if the device has been identified */
>   	bool identified;
> +
> +	/* Track streaming state */
> +	u8 streaming;
>   };
>   
>   static inline struct ov2740 *to_ov2740(struct v4l2_subdev *subdev)
> @@ -929,6 +932,11 @@ static int ov2740_enable_streams(struct v4l2_subdev *sd,
>   	int link_freq_index;
>   	int ret;
>   
> +	if (ov2740->streaming) {
> +		ov2740->streaming |= streams_mask;
> +		return 0;
> +	}
> +
>   	ret = pm_runtime_resume_and_get(&client->dev);
>   	if (ret < 0)
>   		return ret;
> @@ -975,6 +983,8 @@ static int ov2740_enable_streams(struct v4l2_subdev *sd,
>   		goto out_pm_put;
>   	}
>   
> +	ov2740->streaming |= streams_mask;
> +
>   	return 0;
>   
>   out_pm_put:
> @@ -991,6 +1001,10 @@ static int ov2740_disable_streams(struct v4l2_subdev *sd,
>   	struct ov2740 *ov2740 = to_ov2740(sd);
>   	int ret;
>   
> +	ov2740->streaming &= ~streams_mask;
> +	if (ov2740->streaming)
> +		return 0;
> +
>   	ret = ov2740_write_reg(ov2740, OV2740_REG_MODE_SELECT, 1,
>   			       OV2740_MODE_STANDBY);
>   

-- 
Julien Massot
Senior Software Engineer
Collabora Ltd.
Platinum Building, St John's Innovation Park, Cambridge CB4 0DS, UK
Registered in England & Wales, no. 5513718

Re: [PATCH v8 38/38] media: ov2740: Add IMMUTABLE route flag

[Julien Massot]

On 3/13/24 08:25, Sakari Ailus wrote:
> Add immutable route flag to the routing table.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Julien Massot <julien.massot@collabora.com>
> ---
>   drivers/media/i2c/ov2740.c | 6 ++++--
>   1 file changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/media/i2c/ov2740.c b/drivers/media/i2c/ov2740.c
> index 7488b2535071..3e2585eb6ec9 100644
> --- a/drivers/media/i2c/ov2740.c
> +++ b/drivers/media/i2c/ov2740.c
> @@ -1150,12 +1150,14 @@ static int ov2740_init_state(struct v4l2_subdev *sd,
>   			.sink_pad = OV2740_PAD_PIXEL,
>   			.source_pad = OV2740_PAD_SOURCE,
>   			.source_stream = OV2740_STREAM_PIXEL,
> -			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE |
> +				 V4L2_SUBDEV_ROUTE_FL_IMMUTABLE,
>   		}, {
>   			.sink_pad = OV2740_PAD_META,
>   			.source_pad = OV2740_PAD_SOURCE,
>   			.source_stream = OV2740_STREAM_META,
> -			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE |
> +				 V4L2_SUBDEV_ROUTE_FL_IMMUTABLE,
>   		},
>   	};
>   	struct v4l2_subdev_krouting routing = {

-- 
Julien Massot
Senior Software Engineer
Collabora Ltd.
Platinum Building, St John's Innovation Park, Cambridge CB4 0DS, UK
Registered in England & Wales, no. 5513718

Re: [PATCH v8 35/38] media: ov2740: Add support for embedded data

[Sakari Ailus]

Hi Bingbu,

On Thu, Mar 14, 2024 at 03:00:49PM +0800, Bingbu Cao wrote:
> > +	entry->stream = OV2740_STREAM_PIXEL;
> > +	entry->bus.csi2.dt = MIPI_CSI2_DT_RAW10;
> > +	entry++;
> > +	desc->num_entries++;
> > +
> > +	entry->pixelcode = MEDIA_BUS_FMT_META_8;
> 
> Should we change the pixelcode to META_10 also?

For others to know: Bingbu confirmed the sensor embedded data output is
actually using 10 bits per pixel packing.

> 
> > +	entry->stream = OV2740_STREAM_META;
> > +	entry->bus.csi2.dt = MIPI_CSI2_DT_GENERIC_LONG(1);
> 
> In the register setting array, the 0x4816 is set to 0x52 instead of
> its default value 0x53, so the data type should be
> MIPI_CSI2_DT_GENERIC_LONG(0) or MIPI_CSI2_DT_EMBEDDED_8B.
> 
> Or you can change back to the default settings. :)

I'll use MIPI_CSI2_DT_EMBEDDED_8B.

-- 
Regards,

Sakari Ailus

Re: [PATCH v8 35/38] media: ov2740: Add support for embedded data

[Sakari Ailus]

Hi Julien,

Thanks for the review.

On Thu, Mar 14, 2024 at 09:24:48AM +0100, Julien Massot wrote:
> Hi Sakari,
> 
> On 3/13/24 08:25, Sakari Ailus wrote:
> > Add support for embedded data. This introduces two internal pads for pixel
> > and embedded data streams. As the driver supports a single mode only,
> > there's no need for backward compatibility in mode selection.
> > 
> > The embedded data is configured to be placed before the image data whereas
> > after the image data is the default.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >   drivers/media/i2c/ov2740.c | 150 +++++++++++++++++++++++++++++++++----
> >   1 file changed, 137 insertions(+), 13 deletions(-)
> > 
> > diff --git a/drivers/media/i2c/ov2740.c b/drivers/media/i2c/ov2740.c
> > index df57f0096e98..7488b2535071 100644
> > --- a/drivers/media/i2c/ov2740.c
> > +++ b/drivers/media/i2c/ov2740.c
> > @@ -11,6 +11,7 @@
> >   #include <linux/pm_runtime.h>
> >   #include <linux/nvmem-provider.h>
> >   #include <linux/regmap.h>
> > +#include <media/mipi-csi2.h>
> >   #include <media/v4l2-ctrls.h>
> >   #include <media/v4l2-device.h>
> >   #include <media/v4l2-fwnode.h>
> > @@ -71,11 +72,31 @@
> >   #define OV2740_REG_ISP_CTRL00		0x5000
> >   /* ISP CTRL01 */
> >   #define OV2740_REG_ISP_CTRL01		0x5001
> > +
> > +/* Embedded data line location control */
> > +#define OV2740_REG_EMBEDDED_FLAG	0x5a08
> > +#define OV2740_EMBEDDED_FLAG_FOOTER	BIT(2) /* otherwise it's in header */
> > +#define OV2740_EMBEDDED_FLAG_MYSTERY	BIT(1)
> >   /* Customer Addresses: 0x7010 - 0x710F */
> >   #define CUSTOMER_USE_OTP_SIZE		0x100
> >   /* OTP registers from sensor */
> >   #define OV2740_REG_OTP_CUSTOMER		0x7010
> > +enum {
> > +	OV2740_PAD_SOURCE,
> > +	OV2740_PAD_PIXEL,
> > +	OV2740_PAD_META,
> > +	OV2740_NUM_PADS,
> > +};
> > +
> > +enum {
> > +	OV2740_STREAM_PIXEL,
> > +	OV2740_STREAM_META,
> > +};
> > +
> > +#define OV2740_META_WIDTH		100U /* 97 bytes of actual data */
> > +#define OV2740_META_HEIGHT		1U
> > +
> >   struct nvm_data {
> >   	struct nvmem_device *nvmem;
> >   	struct regmap *regmap;
> > @@ -513,7 +534,7 @@ static const struct ov2740_mode supported_modes_180mhz[] = {
> >   struct ov2740 {
> >   	struct v4l2_subdev sd;
> > -	struct media_pad pad;
> > +	struct media_pad pads[OV2740_NUM_PADS];
> >   	struct v4l2_ctrl_handler ctrl_handler;
> >   	/* V4L2 Controls */
> > @@ -976,6 +997,11 @@ static int ov2740_enable_streams(struct v4l2_subdev *sd,
> >   	if (ret)
> >   		goto out_pm_put;
> > +	ret = ov2740_write_reg(ov2740, OV2740_REG_EMBEDDED_FLAG, 1,
> > +			       OV2740_EMBEDDED_FLAG_MYSTERY);
> > +	if (ret)
> > +		return ret;
> > +
> >   	ret = ov2740_write_reg(ov2740, OV2740_REG_MODE_SELECT, 1,
> >   			       OV2740_MODE_STREAMING);
> >   	if (ret) {
> > @@ -1013,23 +1039,49 @@ static int ov2740_disable_streams(struct v4l2_subdev *sd,
> >   	return ret;
> >   }
> > -static int ov2740_set_format(struct v4l2_subdev *sd,
> > -			     struct v4l2_subdev_state *sd_state,
> > -			     struct v4l2_subdev_format *fmt)
> > +static int __ov2740_set_format(struct v4l2_subdev *sd,
> > +			       struct v4l2_subdev_state *sd_state,
> > +			       struct v4l2_mbus_framefmt *format,
> > +			       enum v4l2_subdev_format_whence which,
> > +			       unsigned int pad, unsigned int stream)
> >   {
> > +	struct v4l2_mbus_framefmt *src_pix_fmt, *src_meta_fmt, *pix_fmt,
> > +		*meta_fmt;
> >   	struct ov2740 *ov2740 = to_ov2740(sd);
> >   	const struct ov2740_mode *mode;
> >   	s32 vblank_def, h_blank;
> > +	/*
> > +	 * Allow setting format on internal pixel pad as well as the source
> > +	 * pad's pixel stream (for compatibility).
> > +	 */
> > +	if (pad == OV2740_PAD_SOURCE || pad == OV2740_PAD_META ||
> > +	    stream == OV2740_STREAM_META) {
> This is equivalent to
> if (pad != OV2740_PAD_PIXEL)
> Correct me if I'm wrong but this code doesn't allow to set the format on the
> source pad.

Good point. I put that to the comment but somehow failed to align the code
with the statement.

> 
> Should it be:
> if ((pad == OV2740_PAD_SOURCE && stream == OV2740_STREAM_META) ||
> 	pad == OV2740_PAD_META) {

This seems better indeed!

> 
> 
> > +		*format = *v4l2_subdev_state_get_format(sd_state, pad, stream);
> > +		return 0;
> > +	}
> > +
> > +	pix_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_PIXEL, 0);
> > +	meta_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_META, 0);
> > +	src_pix_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_SOURCE,
> > +						   OV2740_STREAM_PIXEL);
> > +	src_meta_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_SOURCE,
> > +						    OV2740_STREAM_META);
> > +
> >   	mode = v4l2_find_nearest_size(ov2740->supported_modes,
> >   				      ov2740->supported_modes_count,
> >   				      width, height,
> > -				      fmt->format.width, fmt->format.height);
> > +				      format->width, format->height);
> > +	ov2740_update_pad_format(mode, pix_fmt);
> > +	*format = *src_pix_fmt = *pix_fmt;
> > -	ov2740_update_pad_format(mode, &fmt->format);
> > -	*v4l2_subdev_state_get_format(sd_state, fmt->pad) = fmt->format;
> > +	meta_fmt->code = MEDIA_BUS_FMT_OV2740_EMBEDDED;
> > +	meta_fmt->width = OV2740_META_WIDTH;
> > +	meta_fmt->height = OV2740_META_HEIGHT;
> > +	*src_meta_fmt = *meta_fmt;
> > +	src_meta_fmt->code = MEDIA_BUS_FMT_META_10;
> > -	if (fmt->which == V4L2_SUBDEV_FORMAT_TRY)
> > +	if (which == V4L2_SUBDEV_FORMAT_TRY)
> >   		return 0;
> >   	ov2740->cur_mode = mode;

-- 
Kind regards,

Sakari Ailus

Re: [PATCH v8 01/38] media: mc: Add INTERNAL pad flag

[Sakari Ailus]

On Thu, Mar 14, 2024 at 09:17:08AM +0200, Tomi Valkeinen wrote:
> > @@ -381,6 +382,13 @@ Types and flags used to represent the media graph elements
> >   	  enabled links even when this flag isn't set; the absence of the flag
> >   	  doesn't imply there is none.
> > +    *  -  ``MEDIA_PAD_FL_INTERNAL``
> > +       -  The internal flag indicates an internal pad that has no external
> > +	  connections. Such a pad shall not be connected with a link.
> > +
> > +	  The internal flag may currently be present only in a source pad where
> 
> s/source/sink/
> 
> Reviewed-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>

Thanks, I'll fix that for v9.

-- 
Sakari Ailus

Re: [PATCH v8 03/38] media: uapi: Add generic serial metadata mbus formats

[Sakari Ailus]

Moi,

On Thu, Mar 14, 2024 at 09:30:50AM +0200, Tomi Valkeinen wrote:
> On 13/03/2024 09:24, Sakari Ailus wrote:
> > Add generic serial metadata mbus formats. These formats describe data
> > width and packing but not the content itself. The reason for specifying
> > such formats is that the formats as such are fairly device specific but
> > they are still handled by CSI-2 receiver drivers that should not be aware
> > of device specific formats. What makes generic metadata formats possible
> > is that these formats are parsed by software only, after capturing the
> > data to system memory.
> > 
> > Also add a definition for "Data unit" to cover what is essentially a pixel
> > but is not image data.
> 
> The CCS spec talks about legacy packing and optimized packing for 16+ bit
> formats. You cover only the "legacy" ones here. Did you look at those?

The reason is that the bus data layout of the new packing at higher bit
depth matches with packing at lower bit depths (half to be precise). That's
why there's no need to define formats for the new packing methods at higher
bit depths (the driver simply uses the packing at half of the bit depth).

-- 
Terveisin,

Sakari Ailus

Re: [PATCH v8 09/38] media: Documentation: v4l: Document internal source pads

[Sakari Ailus]

Hi Julien,

On Fri, Mar 15, 2024 at 04:32:59PM +0100, Julien Massot wrote:
> 
> 
> On 3/13/24 08:24, Sakari Ailus wrote:
> > Document internal source pads, pads that have both SINK and INTERNAL flags
> > set. Use the IMX219 camera sensor as an example.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >   .../userspace-api/media/v4l/dev-subdev.rst    | 145 ++++++++++++++++++
> >   1 file changed, 145 insertions(+)
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > index a387e8a15b8d..1808f40f63e3 100644
> > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > @@ -553,6 +553,27 @@ A stream at a specific point in the media pipeline is identified by the
> >   sub-device and a (pad, stream) pair. For sub-devices that do not support
> >   multiplexed streams the 'stream' field is always 0.
> > +.. _v4l2-subdev-internal-source-pads:
> > +
> > +Internal source pads and routing
> > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > +
> > +Cases where a single sub-device source pad is traversed by multiple streams, one
> > +or more of which originate from within the sub-device itself, are special as
> > +there is no external sink pad for such routes. In those cases, the sources of
> > +the internally generated streams are represented by internal source pads, which
> > +are sink pads that have the :ref:`MEDIA_PAD_FL_INTERNAL <MEDIA-PAD-FL-INTERNAL>`
> > +pad flag set.
> > +
> > +Internal pads have all the properties of an external pad, including formats and
> > +selections. The format in this case is the source format of the stream. An
> > +internal pad always has a single stream only (0).
> > +
> > +Routes from an internal source pad to an external source pad are typically not
> > +modifiable but they can be activated and deactivated using the
> > +:ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE <v4l2-subdev-routing-flags>` flag, depending
> > +on driver capabilities.
> > +
> >   Interaction between routes, streams, formats and selections
> >   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > @@ -668,3 +689,127 @@ To configure this pipeline, the userspace must take the following steps:
> >      the configurations along the stream towards the receiver, using
> >      :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each
> >      stream endpoint in each sub-device.
> > +
> > +Internal pads setup example
> > +^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > +
> > +A simple example of a multiplexed stream setup might be as follows:
> > +
> > +- An IMX219 camera sensor source sub-device, with one sink pad (0), one source pad
> > +  (1), an internal sink pad (2) that represents the source of embedded
> The pixel pad is an internal pad as well ?

How about:

- An IMX219 camera sensor source sub-device, with one internal sink pad
  (0) for image data, one source pad (1), an internal sink pad (2) that
  represents the source of embedded
 
> 
> > +  data. There are two routes, one from the sink pad to the source, and another
> > +  from the internal sink pad to the source pad. Both streams are always active,
> > +  i.e. there is no need to separately enable the embedded data stream. The
> > +  sensor uses the CSI-2 bus.
> > +
> > +- A CSI-2 receiver in the SoC (Receiver). The receiver has a single sink pad
> > +  (pad 0), connected to the bridge, and two source pads (pads 1-2), going to the
> > +  DMA engine. The receiver demultiplexes the incoming streams to the source
> > +  pads.
> > +
> > +- DMA Engines in the SoC (DMA Engine), one for each stream. Each DMA engine is
> > +  connected to a single source pad in the receiver.
> > +
> > +The sensor, the bridge and the receiver are modeled as V4L2 sub-devices,
> > +exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are
> > +modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
> > +
> > +To configure this pipeline, the userspace must take the following steps:
> > +
> > +1) Set up media links between entities: connect the sensors to the bridge,
> > +   bridge to the receiver, and the receiver to the DMA engines. This step does
> > +   not differ from normal non-multiplexed media controller setup.
> > +
> > +2) Configure routing
> > +
> > +.. flat-table:: Camera sensor. There are no configurable routes.
> > +    :header-rows: 1
> > +
> > +    * - Sink Pad/Stream
> > +      - Source Pad/Stream
> > +      - Routing Flags
> > +      - Comments
> > +    * - 0/0
> > +      - 1/0
> - 1/0
> - 0/0
> > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > +      - Pixel data stream from the sink pad
> > +    * - 2/0
> > +      - 1/1
> - 0/1
> > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > +      - Metadata stream from the internal sink pad
> 
> In latest patch "[PATCH v6 05/15] media: i2c: imx219: Add embedded data
> support"
> we have:
> - 0 -> source pad
> - 1 -> pixel/image
> - 2 -> EDATA
> 
> This is also what you did for ov2740.

Yes, these were leftovers from the CCS example. I've fixed them.

> 
> With that fixed:
> Reviewed-by Julien Massot <julien.massot@collabora.com>

Thank you!

-- 
Regards,

Sakari Ailus

Re: [PATCH v8 03/38] media: uapi: Add generic serial metadata mbus formats

[Tomi Valkeinen]

On 19/03/2024 15:27, Sakari Ailus wrote:
> Moi,
> 
> On Thu, Mar 14, 2024 at 09:30:50AM +0200, Tomi Valkeinen wrote:
>> On 13/03/2024 09:24, Sakari Ailus wrote:
>>> Add generic serial metadata mbus formats. These formats describe data
>>> width and packing but not the content itself. The reason for specifying
>>> such formats is that the formats as such are fairly device specific but
>>> they are still handled by CSI-2 receiver drivers that should not be aware
>>> of device specific formats. What makes generic metadata formats possible
>>> is that these formats are parsed by software only, after capturing the
>>> data to system memory.
>>>
>>> Also add a definition for "Data unit" to cover what is essentially a pixel
>>> but is not image data.
>>
>> The CCS spec talks about legacy packing and optimized packing for 16+ bit
>> formats. You cover only the "legacy" ones here. Did you look at those?
> 
> The reason is that the bus data layout of the new packing at higher bit
> depth matches with packing at lower bit depths (half to be precise). That's
> why there's no need to define formats for the new packing methods at higher
> bit depths (the driver simply uses the packing at half of the bit depth).

Hmm. If we're capturing 10-bit raw format, say, with the width of 640 
pixels, we'll configure the video stream format according to those. For 
the embedded stream, we'll set it to V4L2_META_FMT_GENERIC_CSI2_10 and 
640 width, right?

If we're capturing 20-bit raw, we'll configure the video stream format 
again accordingly, width to 640, and 20 bit fourcc/mbus code. If the 
embedded stream uses the "legacy" packing, we'll set the format to 
V4L2_META_FMT_GENERIC_CSI2_20 with width of 640, right?

But if it's using packed format for the embedded stream, we set the 
format to V4L2_META_FMT_GENERIC_CSI2_10 and width to 1280?

Considering that the video and (line-based) embedded data come from the 
same source, I'd expect the widths to be the same.

  Tomi

Re: [PATCH v8 09/38] media: Documentation: v4l: Document internal source pads

[Julien Massot]

Hi Sakari,
On 3/19/24 14:47, Sakari Ailus wrote:
> Hi Julien,
> 
> On Fri, Mar 15, 2024 at 04:32:59PM +0100, Julien Massot wrote:
>>
>>
>> On 3/13/24 08:24, Sakari Ailus wrote:
>>> Document internal source pads, pads that have both SINK and INTERNAL flags
>>> set. Use the IMX219 camera sensor as an example.
>>>
>>> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
>>> ---
>>>    .../userspace-api/media/v4l/dev-subdev.rst    | 145 ++++++++++++++++++
>>>    1 file changed, 145 insertions(+)
>>>
>>> diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
>>> index a387e8a15b8d..1808f40f63e3 100644
>>> --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
>>> +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
>>> @@ -553,6 +553,27 @@ A stream at a specific point in the media pipeline is identified by the
>>>    sub-device and a (pad, stream) pair. For sub-devices that do not support
>>>    multiplexed streams the 'stream' field is always 0.
>>> +.. _v4l2-subdev-internal-source-pads:
>>> +
>>> +Internal source pads and routing
>>> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>> +
>>> +Cases where a single sub-device source pad is traversed by multiple streams, one
>>> +or more of which originate from within the sub-device itself, are special as
>>> +there is no external sink pad for such routes. In those cases, the sources of
>>> +the internally generated streams are represented by internal source pads, which
>>> +are sink pads that have the :ref:`MEDIA_PAD_FL_INTERNAL <MEDIA-PAD-FL-INTERNAL>`
>>> +pad flag set.
>>> +
>>> +Internal pads have all the properties of an external pad, including formats and
>>> +selections. The format in this case is the source format of the stream. An
>>> +internal pad always has a single stream only (0).
>>> +
>>> +Routes from an internal source pad to an external source pad are typically not
>>> +modifiable but they can be activated and deactivated using the
>>> +:ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE <v4l2-subdev-routing-flags>` flag, depending
>>> +on driver capabilities.
>>> +
>>>    Interaction between routes, streams, formats and selections
>>>    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>> @@ -668,3 +689,127 @@ To configure this pipeline, the userspace must take the following steps:
>>>       the configurations along the stream towards the receiver, using
>>>       :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each
>>>       stream endpoint in each sub-device.
>>> +
>>> +Internal pads setup example
>>> +^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>> +
>>> +A simple example of a multiplexed stream setup might be as follows:
>>> +
>>> +- An IMX219 camera sensor source sub-device, with one sink pad (0), one source pad
>>> +  (1), an internal sink pad (2) that represents the source of embedded
>> The pixel pad is an internal pad as well ?
> 
> How about:
> 
> - An IMX219 camera sensor source sub-device, with one internal sink pad
>    (0) for image data, one source pad (1), an internal sink pad (2) that
>    represents the source of embedded
sure looks good to me!

>   
>>
>>> +  data. There are two routes, one from the sink pad to the source, and another
>>> +  from the internal sink pad to the source pad. Both streams are always active,
>>> +  i.e. there is no need to separately enable the embedded data stream. The
>>> +  sensor uses the CSI-2 bus.
>>> +
>>> +- A CSI-2 receiver in the SoC (Receiver). The receiver has a single sink pad
>>> +  (pad 0), connected to the bridge, and two source pads (pads 1-2), going to the
>>> +  DMA engine. The receiver demultiplexes the incoming streams to the source
>>> +  pads.
>>> +
>>> +- DMA Engines in the SoC (DMA Engine), one for each stream. Each DMA engine is
>>> +  connected to a single source pad in the receiver.
>>> +
>>> +The sensor, the bridge and the receiver are modeled as V4L2 sub-devices,
>>> +exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are
>>> +modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
>>> +
>>> +To configure this pipeline, the userspace must take the following steps:
>>> +
>>> +1) Set up media links between entities: connect the sensors to the bridge,
>>> +   bridge to the receiver, and the receiver to the DMA engines. This step does
>>> +   not differ from normal non-multiplexed media controller setup.
>>> +
>>> +2) Configure routing
>>> +
>>> +.. flat-table:: Camera sensor. There are no configurable routes.
>>> +    :header-rows: 1
>>> +
>>> +    * - Sink Pad/Stream
>>> +      - Source Pad/Stream
>>> +      - Routing Flags
>>> +      - Comments
>>> +    * - 0/0
>>> +      - 1/0
>> - 1/0
>> - 0/0
>>> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
>>> +      - Pixel data stream from the sink pad
>>> +    * - 2/0
>>> +      - 1/1
>> - 0/1
>>> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
>>> +      - Metadata stream from the internal sink pad
>>
>> In latest patch "[PATCH v6 05/15] media: i2c: imx219: Add embedded data
>> support"
>> we have:
>> - 0 -> source pad
>> - 1 -> pixel/image
>> - 2 -> EDATA
>>
>> This is also what you did for ov2740.
> 
> Yes, these were leftovers from the CCS example. I've fixed them.
> 
>>
>> With that fixed:
>> Reviewed-by Julien Massot <julien.massot@collabora.com>
> 
> Thank you!
> 

Re: [PATCH v8 01/38] media: mc: Add INTERNAL pad flag

[Laurent Pinchart]

On Thu, Mar 14, 2024 at 09:17:08AM +0200, Tomi Valkeinen wrote:
> On 13/03/2024 09:24, Sakari Ailus wrote:
> > Internal source pads will be used as routing endpoints in V4L2
> > [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
> > Internal source pads are pads that have both SINK and INTERNAL flags set.
> > 
> > Also prevent creating links to pads that have been flagged as internal and
> > initialising SOURCE pads with INTERNAL flag set.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >   .../userspace-api/media/mediactl/media-types.rst       |  8 ++++++++
> >   drivers/media/mc/mc-entity.c                           | 10 ++++++++--
> >   include/uapi/linux/media.h                             |  1 +
> >   3 files changed, 17 insertions(+), 2 deletions(-)
> > 
> > diff --git a/Documentation/userspace-api/media/mediactl/media-types.rst b/Documentation/userspace-api/media/mediactl/media-types.rst
> > index 6332e8395263..f55ef055bcf8 100644
> > --- a/Documentation/userspace-api/media/mediactl/media-types.rst
> > +++ b/Documentation/userspace-api/media/mediactl/media-types.rst
> > @@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
> >   .. _MEDIA-PAD-FL-SINK:
> >   .. _MEDIA-PAD-FL-SOURCE:
> >   .. _MEDIA-PAD-FL-MUST-CONNECT:
> > +.. _MEDIA-PAD-FL-INTERNAL:
> >   
> >   .. flat-table:: Media pad flags
> >       :header-rows:  0
> > @@ -381,6 +382,13 @@ Types and flags used to represent the media graph elements
> >   	  enabled links even when this flag isn't set; the absence of the flag
> >   	  doesn't imply there is none.
> >   
> > +    *  -  ``MEDIA_PAD_FL_INTERNAL``
> > +       -  The internal flag indicates an internal pad that has no external
> > +	  connections. Such a pad shall not be connected with a link.

I would expand this slightly, as it's the only source of documentation
regarding internal pads.

       -  The internal flag indicates an internal pad that has no external
	  connections. This can be used to model, for instance, the pixel array
	  internal to an image sensor. As they are internal to entities,
	  internal pads shall not be connected with links.

> > +
> > +	  The internal flag may currently be present only in a source pad where
> 
> s/source/sink/
> 
> Reviewed-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > +	  it indicates that the :ref:``stream <media-glossary-stream>``
> > +	  originates from within the entity.
> >   
> >   One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
> >   must be set for every pad.
> > diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> > index 0e28b9a7936e..1973e9e1013e 100644
> > --- a/drivers/media/mc/mc-entity.c
> > +++ b/drivers/media/mc/mc-entity.c
> > @@ -213,7 +213,9 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
> >   		iter->index = i++;
> >   
> >   		if (hweight32(iter->flags & (MEDIA_PAD_FL_SINK |
> > -					     MEDIA_PAD_FL_SOURCE)) != 1) {
> > +					     MEDIA_PAD_FL_SOURCE)) != 1 ||
> > +		    (iter->flags & MEDIA_PAD_FL_INTERNAL &&
> > +		     !(iter->flags & MEDIA_PAD_FL_SINK))) {
> >   			ret = -EINVAL;
> >   			break;
> >   		}

This may become more readable written as a switch statement:

		const u32 pad_flags_mask = MEDIA_PAD_FL_SINK |
					   MEDIA_PAD_FL_SOURCE |
					   MEDIA_PAD_FL_INTERNAL;

		switch (iter->flags & pad_flags_mask) {
		case MEDIA_PAD_FL_SINK:
		case MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL:
		case MEDIA_PAD_FL_SOURCE:
			break;

		default:
			ret = -EINVAL;
			break;
		}

		if (ret)
			break;

And now that I've written this, I'm not too sure anymore :-) Another
option would be


	const u32 pad_flags = iter->flags & (MEDIA_PAD_FL_SINK |
					     MEDIA_PAD_FL_SOURCE |
					     MEDIA_PAD_FL_INTERNAL);

	if (pad_flags != MEDIA_PAD_FL_SINK &&
	    pad_flags != MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL
	    pad_flags != MEDIA_PAD_FL_SOURCE) {
		ret = -EINVAL;
		break;
	}

Up to you.

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> > @@ -1112,7 +1114,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
> >   
> >   	for (i = 0; i < entity->num_pads; i++) {
> >   		if ((entity->pads[i].flags &
> > -		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
> > +		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
> > +		      MEDIA_PAD_FL_INTERNAL)) != pad_type)
> >   			continue;
> >   
> >   		if (entity->pads[i].sig_type == sig_type)
> > @@ -1142,6 +1145,9 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
> >   		return -EINVAL;
> >   	if (WARN_ON(!(sink->pads[sink_pad].flags & MEDIA_PAD_FL_SINK)))
> >   		return -EINVAL;
> > +	if (WARN_ON(source->pads[source_pad].flags & MEDIA_PAD_FL_INTERNAL) ||
> > +	    WARN_ON(sink->pads[sink_pad].flags & MEDIA_PAD_FL_INTERNAL))
> > +		return -EINVAL;
> >   
> >   	link = media_add_link(&source->links);
> >   	if (link == NULL)
> > diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
> > index 1c80b1d6bbaf..80cfd12a43fc 100644
> > --- a/include/uapi/linux/media.h
> > +++ b/include/uapi/linux/media.h
> > @@ -208,6 +208,7 @@ struct media_entity_desc {
> >   #define MEDIA_PAD_FL_SINK			(1U << 0)
> >   #define MEDIA_PAD_FL_SOURCE			(1U << 1)
> >   #define MEDIA_PAD_FL_MUST_CONNECT		(1U << 2)
> > +#define MEDIA_PAD_FL_INTERNAL			(1U << 3)
> >   
> >   struct media_pad_desc {
> >   	__u32 entity;		/* entity ID */

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 02/38] media: Documentation: Add "stream" into glossary

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Wed, Mar 13, 2024 at 09:24:40AM +0200, Sakari Ailus wrote:
> Add term "stream" to the glossary of the Media subsystem documentation.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  Documentation/userspace-api/media/glossary.rst | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/Documentation/userspace-api/media/glossary.rst b/Documentation/userspace-api/media/glossary.rst
> index 96a360edbf3b..ef0ab601b5bf 100644
> --- a/Documentation/userspace-api/media/glossary.rst
> +++ b/Documentation/userspace-api/media/glossary.rst
> @@ -173,6 +173,12 @@ Glossary
>  	An integrated circuit that integrates all components of a computer
>  	or other electronic systems.
>  
> +_media-glossary-stream:
> +    Stream
> +	A distinct flow of data (image data or metadata) from an initial source
> +	to a final sink. The initial source may be e.g. an image sensor and the
> +	final sink e.g. a memory buffer.

I would add

	A link may carry multiple streams.

or something similar.

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> +
>      V4L2 API
>  	**V4L2 userspace API**
>  

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 03/38] media: uapi: Add generic serial metadata mbus formats

[Laurent Pinchart]

On Tue, Mar 19, 2024 at 04:20:35PM +0200, Tomi Valkeinen wrote:
> On 19/03/2024 15:27, Sakari Ailus wrote:
> > On Thu, Mar 14, 2024 at 09:30:50AM +0200, Tomi Valkeinen wrote:
> >> On 13/03/2024 09:24, Sakari Ailus wrote:
> >>> Add generic serial metadata mbus formats. These formats describe data
> >>> width and packing but not the content itself. The reason for specifying
> >>> such formats is that the formats as such are fairly device specific but
> >>> they are still handled by CSI-2 receiver drivers that should not be aware
> >>> of device specific formats. What makes generic metadata formats possible
> >>> is that these formats are parsed by software only, after capturing the
> >>> data to system memory.
> >>>
> >>> Also add a definition for "Data unit" to cover what is essentially a pixel
> >>> but is not image data.
> >>
> >> The CCS spec talks about legacy packing and optimized packing for 16+ bit
> >> formats. You cover only the "legacy" ones here. Did you look at those?
> > 
> > The reason is that the bus data layout of the new packing at higher bit
> > depth matches with packing at lower bit depths (half to be precise). That's
> > why there's no need to define formats for the new packing methods at higher
> > bit depths (the driver simply uses the packing at half of the bit depth).
> 
> Hmm. If we're capturing 10-bit raw format, say, with the width of 640 
> pixels, we'll configure the video stream format according to those. For 
> the embedded stream, we'll set it to V4L2_META_FMT_GENERIC_CSI2_10 and 
> 640 width, right?
> 
> If we're capturing 20-bit raw, we'll configure the video stream format 
> again accordingly, width to 640, and 20 bit fourcc/mbus code. If the 
> embedded stream uses the "legacy" packing, we'll set the format to 
> V4L2_META_FMT_GENERIC_CSI2_20 with width of 640, right?
> 
> But if it's using packed format for the embedded stream, we set the 
> format to V4L2_META_FMT_GENERIC_CSI2_10 and width to 1280?
> 
> Considering that the video and (line-based) embedded data come from the 
> same source, I'd expect the widths to be the same.

I don't have a strong objection against multiplying the width, but we
need to figure out the impact on other kernel space components, as well
as on userspace. I suppose the media bus code for the embedded data
stream on the sensor source pad when using optimized packing and
capturing RAW20 images would be MEDIA_BUS_FMT_META_10 ? In that case I
think the sensor driver should be able to handle the width calculations
on its own, and the value would just be propagated by userspace.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 03/38] media: uapi: Add generic serial metadata mbus formats

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Wed, Mar 13, 2024 at 09:24:41AM +0200, Sakari Ailus wrote:
> Add generic serial metadata mbus formats. These formats describe data
> width and packing but not the content itself. The reason for specifying
> such formats is that the formats as such are fairly device specific but
> they are still handled by CSI-2 receiver drivers that should not be aware
> of device specific formats. What makes generic metadata formats possible
> is that these formats are parsed by software only, after capturing the
> data to system memory.
> 
> Also add a definition for "Data unit" to cover what is essentially a pixel
> but is not image data.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../userspace-api/media/glossary.rst          |   9 +
>  .../media/v4l/subdev-formats.rst              | 258 ++++++++++++++++++
>  include/uapi/linux/media-bus-format.h         |   9 +
>  3 files changed, 276 insertions(+)
> 
> diff --git a/Documentation/userspace-api/media/glossary.rst b/Documentation/userspace-api/media/glossary.rst
> index ef0ab601b5bf..7078141894c5 100644
> --- a/Documentation/userspace-api/media/glossary.rst
> +++ b/Documentation/userspace-api/media/glossary.rst
> @@ -25,6 +25,15 @@ Glossary
>  
>  	See :ref:`cec`.
>  
> +.. _media-glossary-data-unit:
> +
> +    Data unit
> +
> +	Unit of data transported by a bus. On parallel buses, the data unit
> +	consists of one or more related samples while on serial buses the data
> +	unit is logical. If the data unit is image data, it may also be called a
> +	pixel.

I'm pretty sure nobody will be able to understand what this means, but I
don't have a better proposal at the moment.

> +
>      Device Driver
>  	Part of the Linux Kernel that implements support for a hardware
>  	component.
> diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> index eb3cd20b0cf2..cbd475f7cae9 100644
> --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
> +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> @@ -8306,3 +8306,261 @@ The following table lists the existing metadata formats.
>  	both sides of the link and the bus format is a fixed
>  	metadata format that is not configurable from userspace.
>  	Width and height will be set to 0 for this format.
> +
> +Generic Serial Metadata Formats
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +Generic serial metadata formats are used on serial buses where the actual data
> +content is more or less device specific but the data is transmitted and received
> +by multiple devices that do not process the data in any way, simply writing
> +it to system memory for processing in software at the end of the pipeline.
> +
> +The more specific variant describing the actual data is used on the internal
> +source pad of the originating sub-device.

Maybe this paragraph would be best added in the patch that adds the
specific metadata formats, you could then mention one of them as an
example:

The exact format of the data generated by the device is reported on the
internal source pad of the originating sub-device, using one of the more
specific metadata formats such as MEDIA_BUS_FMT_CCS_EMBEDDED.

> +
> +"b" in an array cell signifies a byte of data, followed by the number of the bit

s/bit$/byte/

> +and finally the bit number in subscript. "X" indicates a padding bit.

We use a lower-case x in pixfmt-rgb.rst, I would do the same here. We
also use single quotes there, turning "b" and "x" into 'b' and 'x'.

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> +
> +.. _media-bus-format-generic-meta:
> +
> +.. cssclass: longtable
> +
> +.. flat-table:: Generic Serial Metadata Formats
> +    :header-rows:  2
> +    :stub-columns: 0
> +
> +    * - Identifier
> +      - Code
> +      -
> +      - :cspan:`23` Data organization within bus ``Data unit
> +	<media-glossary-data-unit>``
> +    * -
> +      -
> +      - Bit
> +      - 23
> +      - 22
> +      - 21
> +      - 20
> +      - 19
> +      - 18
> +      - 17
> +      - 16
> +      - 15
> +      - 14
> +      - 13
> +      - 12
> +      - 11
> +      - 10
> +      - 9
> +      - 8
> +      - 7
> +      - 6
> +      - 5
> +      - 4
> +      - 3
> +      - 2
> +      - 1
> +      - 0
> +    * .. _MEDIA-BUS-FMT-META-8:
> +
> +      - MEDIA_BUS_FMT_META_8
> +      - 0x8001
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      - b0\ :sub:`7`
> +      - b0\ :sub:`6`
> +      - b0\ :sub:`5`
> +      - b0\ :sub:`4`
> +      - b0\ :sub:`3`
> +      - b0\ :sub:`2`
> +      - b0\ :sub:`1`
> +      - b0\ :sub:`0`
> +    * .. _MEDIA-BUS-FMT-META-10:
> +
> +      - MEDIA_BUS_FMT_META_10
> +      - 0x8002
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      - b0\ :sub:`7`
> +      - b0\ :sub:`6`
> +      - b0\ :sub:`5`
> +      - b0\ :sub:`4`
> +      - b0\ :sub:`3`
> +      - b0\ :sub:`2`
> +      - b0\ :sub:`1`
> +      - b0\ :sub:`0`
> +      - X
> +      - X
> +    * .. _MEDIA-BUS-FMT-META-12:
> +
> +      - MEDIA_BUS_FMT_META_12
> +      - 0x8003
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      - b0\ :sub:`7`
> +      - b0\ :sub:`6`
> +      - b0\ :sub:`5`
> +      - b0\ :sub:`4`
> +      - b0\ :sub:`3`
> +      - b0\ :sub:`2`
> +      - b0\ :sub:`1`
> +      - b0\ :sub:`0`
> +      - X
> +      - X
> +      - X
> +      - X
> +    * .. _MEDIA-BUS-FMT-META-14:
> +
> +      - MEDIA_BUS_FMT_META_14
> +      - 0x8004
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      - b0\ :sub:`7`
> +      - b0\ :sub:`6`
> +      - b0\ :sub:`5`
> +      - b0\ :sub:`4`
> +      - b0\ :sub:`3`
> +      - b0\ :sub:`2`
> +      - b0\ :sub:`1`
> +      - b0\ :sub:`0`
> +      - X
> +      - X
> +      - X
> +      - X
> +      - X
> +      - X
> +    * .. _MEDIA-BUS-FMT-META-16:
> +
> +      - MEDIA_BUS_FMT_META_16
> +      - 0x8005
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      - b0\ :sub:`7`
> +      - b0\ :sub:`6`
> +      - b0\ :sub:`5`
> +      - b0\ :sub:`4`
> +      - b0\ :sub:`3`
> +      - b0\ :sub:`2`
> +      - b0\ :sub:`1`
> +      - b0\ :sub:`0`
> +      - X
> +      - X
> +      - X
> +      - X
> +      - X
> +      - X
> +      - X
> +      - X
> +    * .. _MEDIA-BUS-FMT-META-20:
> +
> +      - MEDIA_BUS_FMT_META_20
> +      - 0x8006
> +      -
> +      -
> +      -
> +      -
> +      -
> +      - b0\ :sub:`7`
> +      - b0\ :sub:`6`
> +      - b0\ :sub:`5`
> +      - b0\ :sub:`4`
> +      - b0\ :sub:`3`
> +      - b0\ :sub:`2`
> +      - b0\ :sub:`1`
> +      - b0\ :sub:`0`
> +      - X
> +      - X
> +      - X
> +      - X
> +      - X
> +      - X
> +      - X
> +      - X
> +      - X
> +      - X
> +      - X
> +      - X
> +    * .. _MEDIA-BUS-FMT-META-24:
> +
> +      - MEDIA_BUS_FMT_META_24
> +      - 0x8007
> +      -
> +      - b0\ :sub:`7`
> +      - b0\ :sub:`6`
> +      - b0\ :sub:`5`
> +      - b0\ :sub:`4`
> +      - b0\ :sub:`3`
> +      - b0\ :sub:`2`
> +      - b0\ :sub:`1`
> +      - b0\ :sub:`0`
> +      - X
> +      - X
> +      - X
> +      - X
> +      - X
> +      - X
> +      - X
> +      - X
> +      - X
> +      - X
> +      - X
> +      - X
> +      - X
> +      - X
> +      - X
> +      - X
> diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
> index f05f747e444d..d4c1d991014b 100644
> --- a/include/uapi/linux/media-bus-format.h
> +++ b/include/uapi/linux/media-bus-format.h
> @@ -174,4 +174,13 @@
>   */
>  #define MEDIA_BUS_FMT_METADATA_FIXED		0x7001
>  
> +/* Generic line based metadata formats for serial buses. Next is 0x8008. */
> +#define MEDIA_BUS_FMT_META_8			0x8001
> +#define MEDIA_BUS_FMT_META_10			0x8002
> +#define MEDIA_BUS_FMT_META_12			0x8003
> +#define MEDIA_BUS_FMT_META_14			0x8004
> +#define MEDIA_BUS_FMT_META_16			0x8005
> +#define MEDIA_BUS_FMT_META_20			0x8006
> +#define MEDIA_BUS_FMT_META_24			0x8007
> +
>  #endif /* __LINUX_MEDIA_BUS_FORMAT_H */

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 03/38] media: uapi: Add generic serial metadata mbus formats

[Laurent Pinchart]

On Wed, Mar 20, 2024 at 12:33:48AM +0200, Laurent Pinchart wrote:
> On Tue, Mar 19, 2024 at 04:20:35PM +0200, Tomi Valkeinen wrote:
> > On 19/03/2024 15:27, Sakari Ailus wrote:
> > > On Thu, Mar 14, 2024 at 09:30:50AM +0200, Tomi Valkeinen wrote:
> > >> On 13/03/2024 09:24, Sakari Ailus wrote:
> > >>> Add generic serial metadata mbus formats. These formats describe data
> > >>> width and packing but not the content itself. The reason for specifying
> > >>> such formats is that the formats as such are fairly device specific but
> > >>> they are still handled by CSI-2 receiver drivers that should not be aware
> > >>> of device specific formats. What makes generic metadata formats possible
> > >>> is that these formats are parsed by software only, after capturing the
> > >>> data to system memory.
> > >>>
> > >>> Also add a definition for "Data unit" to cover what is essentially a pixel
> > >>> but is not image data.
> > >>
> > >> The CCS spec talks about legacy packing and optimized packing for 16+ bit
> > >> formats. You cover only the "legacy" ones here. Did you look at those?
> > > 
> > > The reason is that the bus data layout of the new packing at higher bit
> > > depth matches with packing at lower bit depths (half to be precise). That's
> > > why there's no need to define formats for the new packing methods at higher
> > > bit depths (the driver simply uses the packing at half of the bit depth).
> > 
> > Hmm. If we're capturing 10-bit raw format, say, with the width of 640 
> > pixels, we'll configure the video stream format according to those. For 
> > the embedded stream, we'll set it to V4L2_META_FMT_GENERIC_CSI2_10 and 
> > 640 width, right?
> > 
> > If we're capturing 20-bit raw, we'll configure the video stream format 
> > again accordingly, width to 640, and 20 bit fourcc/mbus code. If the 
> > embedded stream uses the "legacy" packing, we'll set the format to 
> > V4L2_META_FMT_GENERIC_CSI2_20 with width of 640, right?
> > 
> > But if it's using packed format for the embedded stream, we set the 
> > format to V4L2_META_FMT_GENERIC_CSI2_10 and width to 1280?
> > 
> > Considering that the video and (line-based) embedded data come from the 
> > same source, I'd expect the widths to be the same.
> 
> I don't have a strong objection against multiplying the width, but we
> need to figure out the impact on other kernel space components, as well
> as on userspace. I suppose the media bus code for the embedded data
> stream on the sensor source pad when using optimized packing and
> capturing RAW20 images would be MEDIA_BUS_FMT_META_10 ? In that case I
> think the sensor driver should be able to handle the width calculations
> on its own, and the value would just be propagated by userspace.

This should be documented somewhere in this series by the way (not in
this patch).

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 04/38] media: uapi: Document which mbus format fields are valid for metadata

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Wed, Mar 13, 2024 at 09:24:42AM +0200, Sakari Ailus wrote:
> Now that metadata mbus formats have been added, it is necessary to define
> which fields in struct v4l2_mbus_format are applicable to them (not many).
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  include/uapi/linux/v4l2-mediabus.h | 18 ++++++++++++------
>  1 file changed, 12 insertions(+), 6 deletions(-)
> 
> diff --git a/include/uapi/linux/v4l2-mediabus.h b/include/uapi/linux/v4l2-mediabus.h
> index 6b07b73473b5..de1d6161bf62 100644
> --- a/include/uapi/linux/v4l2-mediabus.h
> +++ b/include/uapi/linux/v4l2-mediabus.h
> @@ -19,12 +19,18 @@
>   * @width:	image width
>   * @height:	image height
>   * @code:	data format code (from enum v4l2_mbus_pixelcode)
> - * @field:	used interlacing type (from enum v4l2_field)
> - * @colorspace:	colorspace of the data (from enum v4l2_colorspace)
> - * @ycbcr_enc:	YCbCr encoding of the data (from enum v4l2_ycbcr_encoding)
> - * @hsv_enc:	HSV encoding of the data (from enum v4l2_hsv_encoding)
> - * @quantization: quantization of the data (from enum v4l2_quantization)
> - * @xfer_func:  transfer function of the data (from enum v4l2_xfer_func)
> + * @field:	used interlacing type (from enum v4l2_field), zero on metadata
> + *		mbus codes

I would write "zero for metadata formats". Up to you.

Shouldn't you also update
Documentation/userspace-api/media/v4l/subdev-formats.rst ?

> + * @colorspace:	colorspace of the data (from enum v4l2_colorspace), zero on
> + *		metadata mbus codes
> + * @ycbcr_enc:	YCbCr encoding of the data (from enum v4l2_ycbcr_encoding), zero
> + *		on metadata mbus codes
> + * @hsv_enc:	HSV encoding of the data (from enum v4l2_hsv_encoding), zero on
> + *		metadata mbus codes
> + * @quantization: quantization of the data (from enum v4l2_quantization), zero
> + *		on metadata mbus codes
> + * @xfer_func:  transfer function of the data (from enum v4l2_xfer_func), zero
> + *		on metadata mbus codes
>   * @flags:	flags (V4L2_MBUS_FRAMEFMT_*)
>   * @reserved:  reserved bytes that can be later used
>   */

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 05/38] media: uapi: Add generic 8-bit metadata format definitions

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

Mentioning V4L2 on the subject line would help differentiating this from
the media bus codes.

On Wed, Mar 13, 2024 at 09:24:43AM +0200, Sakari Ailus wrote:
> Generic 8-bit metadata formats define the in-memory data layout but not
> the format of the data itself. The reasoning for having such formats is to
> allow CSI-2 receiver drivers to receive and DMA drivers to write the data
> to memory without knowing a large number of device specific formats.

s/device specific/device-specific/

> These formats may be used only in conjunction of a Media controller

s/of a/with a/

> pipeline where the internal pad of the source sub-device defines the
> specific format of the data (using an mbus code).
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../userspace-api/media/v4l/dev-subdev.rst    |   2 +
>  .../userspace-api/media/v4l/meta-formats.rst  |   1 +
>  .../media/v4l/metafmt-generic.rst             | 304 ++++++++++++++++++
>  drivers/media/v4l2-core/v4l2-ioctl.c          |   7 +
>  include/uapi/linux/videodev2.h                |   8 +
>  5 files changed, 322 insertions(+)
>  create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
> 
> diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> index 43988516acdd..f375b820ab68 100644
> --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> @@ -506,6 +506,8 @@ source pads.
>  
>      subdev-formats
>  
> +.. _subdev-routing:
> +
>  Streams, multiplexed media pads and internal routing
>  ----------------------------------------------------
>  
> diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
> index 0bb61fc5bc00..919f595576b9 100644
> --- a/Documentation/userspace-api/media/v4l/meta-formats.rst
> +++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
> @@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only.
>      metafmt-vsp1-hgo
>      metafmt-vsp1-hgt
>      metafmt-vivid
> +    metafmt-generic

I think the intention was to keep those alphabetically sorted, even if
vivid is out of order already.

> diff --git a/Documentation/userspace-api/media/v4l/metafmt-generic.rst b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> new file mode 100644
> index 000000000000..2ebab1f895e0
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> @@ -0,0 +1,304 @@
> +.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
> +
> +**************************************************************************************************************************************************************************************************************************************************************************************************************************
> +V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O')
> +**************************************************************************************************************************************************************************************************************************************************************************************************************************

Aren't you supposed to match the line lengths ?

> +
> +
> +Generic line-based metadata formats
> +
> +
> +Description
> +===========
> +
> +These generic line-based metadata formats define the memory layout of the data
> +without defining the format or meaning of the metadata itself. These formats may
> +only be used with a Media controller pipeline where the more specific format is

"Media Controller" or "media controller", but not a hybrid mix.

> +defined in an :ref:`internal source pad <MEDIA-PAD-FL-INTERNAL>` of the source

s/defined in/reported by/

> +sub-device. See also :ref:`source routes <subdev-routing>`.
> +
> +.. _v4l2-meta-fmt-generic-8:
> +
> +V4L2_META_FMT_GENERIC_8
> +-----------------------
> +
> +The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format.
> +
> +This format is also used on CSI-2 for both 8 bits per ``Data unit

s/Data unit/Data Unit/ (through the whole patch series)

> +<media-glossary-data-unit>`` as well as for 16 bits per Data unit when two bytes
> +of metadata are packed into one 16-bit Data unit.
> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_8.**
> +Each cell is one byte. "M" denotes a byte of metadata.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|
> +
> +.. flat-table::

Let's add a title here

.. flat-table:: Sample 4x2 Metadata Frame

(feel free to adjust). Same below.

> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - M\ :sub:`10`
> +      - M\ :sub:`20`
> +      - M\ :sub:`30`
> +    * - start + 4:
> +      - M\ :sub:`01`
> +      - M\ :sub:`11`
> +      - M\ :sub:`21`
> +      - M\ :sub:`31`
> +
> +.. _v4l2-meta-fmt-generic-csi2-10:
> +
> +V4L2_META_FMT_GENERIC_CSI2_10
> +-----------------------------
> +
> +V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
> +for each 8 bits of data. Every four bytes of metadata is followed by a single
> +byte of padding. The way the data is packed follows the MIPI CSI-2 specification
> +and the padding is defined in the MIPI CCS specification.

Doesn't really sound packed, when you have one byte of padding after
four bytes of data :-) I understand where this "packing" is from, but
the documentation seems confusing for people not familiar with the
topic. How about simplifying it as follows ?

V4L2_META_FMT_GENERIC_CSI2_10 contains 8-bit generic metadata, stored with one
padding byte after every four bytes of metadata. This is usually produced by
CSI-2 receivers with a source that transmits MEDIA_BUS_FMT_META_10, when the
CSI-2 receiver packs the received data as defined in the MIPI CSI-2
specification.

> +
> +This format is also used in conjunction with 20 bits per ``Data unit
> +<media-glossary-data-unit>`` formats that pack two bytes of metadata into one
> +Data unit.
> +
> +This format is little endian.

How is it little endian when it's byte-oriented ?

> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
> +Each cell is one byte. "M" denotes a byte of metadata and "X" a byte of padding.

'M' and 'x'.

These comments apply to the formats below too.

> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
> +
> +.. flat-table::
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - M\ :sub:`10`
> +      - M\ :sub:`20`
> +      - M\ :sub:`30`
> +      - X
> +    * - start + 5:
> +      - M\ :sub:`01`
> +      - M\ :sub:`11`
> +      - M\ :sub:`21`
> +      - M\ :sub:`31`
> +      - X
> +
> +.. _v4l2-meta-fmt-generic-csi2-12:
> +
> +V4L2_META_FMT_GENERIC_CSI2_12
> +-----------------------------
> +
> +V4L2_META_FMT_GENERIC_CSI2_12 contains packed 8-bit generic metadata, 12 bits
> +for each 8 bits of data. Every four bytes of metadata is followed by a single
> +byte of padding. The way the data is packed follows the MIPI CSI-2 specification
> +and the padding is defined in the MIPI CCS specification.
> +
> +This format is also used in conjunction with 24 bits per ``Data unit
> +<media-glossary-data-unit>`` formats that pack two bytes of metadata into one
> +Data unit.
> +
> +This format is little endian.
> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_12.**
> +Each cell is one byte. "M" denotes a byte of metadata and "X" a byte of padding.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> +
> +.. flat-table::
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - M\ :sub:`10`
> +      - X
> +      - M\ :sub:`20`
> +      - M\ :sub:`30`
> +      - X
> +    * - start + 6:
> +      - M\ :sub:`01`
> +      - M\ :sub:`11`
> +      - X
> +      - M\ :sub:`21`
> +      - M\ :sub:`31`
> +      - X
> +
> +.. _v4l2-meta-fmt-generic-csi2-14:
> +
> +V4L2_META_FMT_GENERIC_CSI2_14
> +-----------------------------
> +
> +V4L2_META_FMT_GENERIC_CSI2_14 contains packed 8-bit generic metadata, 14 bits
> +for each 8 bits of data. Every four bytes of metadata is followed by three bytes
> +of padding. The way the data is packed follows the MIPI CSI-2 specification and
> +the padding is defined in the MIPI CCS specification.
> +
> +This format is little endian.
> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_14.**
> +Each cell is one byte. "M" denotes a byte of metadata and "X" a byte of padding.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{.8cm}|
> +
> +.. flat-table::
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - M\ :sub:`10`
> +      - M\ :sub:`20`
> +      - M\ :sub:`30`
> +      - X
> +      - X
> +      - X
> +    * - start + 7:
> +      - M\ :sub:`01`
> +      - M\ :sub:`11`
> +      - M\ :sub:`21`
> +      - M\ :sub:`31`
> +      - X
> +      - X
> +      - X
> +
> +.. _v4l2-meta-fmt-generic-csi2-16:
> +
> +V4L2_META_FMT_GENERIC_CSI2_16
> +-----------------------------
> +
> +V4L2_META_FMT_GENERIC_CSI2_16 contains packed 8-bit generic metadata, 16 bits
> +for each 8 bits of data. Every byte of metadata is followed by one byte of
> +padding. The way the data is packed follows the MIPI CSI-2 specification and the
> +padding is defined in the MIPI CCS specification.
> +
> +This format is little endian.
> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_16.**
> +Each cell is one byte. "M" denotes a byte of metadata and "X" a byte of padding.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|
> +
> +.. flat-table::
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - X
> +      - M\ :sub:`10`
> +      - X
> +      - M\ :sub:`20`
> +      - X
> +      - M\ :sub:`30`
> +      - X
> +    * - start + 8:
> +      - M\ :sub:`01`
> +      - X
> +      - M\ :sub:`11`
> +      - X
> +      - M\ :sub:`21`
> +      - X
> +      - M\ :sub:`31`
> +      - X
> +
> +.. _v4l2-meta-fmt-generic-csi2-20:
> +
> +V4L2_META_FMT_GENERIC_CSI2_20
> +-----------------------------
> +
> +V4L2_META_FMT_GENERIC_CSI2_20 contains packed 8-bit generic metadata, 20 bits
> +for each 8 bits of data. Every byte of metadata is followed by alternating one
> +and two bytes of padding. The way the data is packed follows the MIPI CSI-2
> +specification and the padding is defined in the MIPI CCS specification.
> +
> +This format is little endian.
> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_20.**
> +Each cell is one byte. "M" denotes a byte of metadata and "X" a byte of padding.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> +
> +.. flat-table::
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8 8 8 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - X
> +      - M\ :sub:`10`
> +      - X
> +      - X
> +      - M\ :sub:`20`
> +      - X
> +      - M\ :sub:`30`
> +      - X
> +      - X
> +    * - start + 10:
> +      - M\ :sub:`01`
> +      - X
> +      - M\ :sub:`11`
> +      - X
> +      - X
> +      - M\ :sub:`21`
> +      - X
> +      - M\ :sub:`31`
> +      - X
> +      - X
> +
> +.. _v4l2-meta-fmt-generic-csi2-24:
> +
> +V4L2_META_FMT_GENERIC_CSI2_24
> +-----------------------------
> +
> +V4L2_META_FMT_GENERIC_CSI2_24 contains packed 8-bit generic metadata, 24 bits
> +for each 8 bits of data. Every byte of metadata is followed by two bytes of
> +padding. The way the data is packed follows the MIPI CSI-2 specification and the
> +padding is defined in the MIPI CCS specification.
> +
> +This format is little endian.
> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_24.**
> +Each cell is one byte. "M" denotes a byte of metadata and "X" a byte of padding.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> +
> +.. flat-table::
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8 8 8 8 8 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - X
> +      - X
> +      - M\ :sub:`10`
> +      - X
> +      - X
> +      - M\ :sub:`20`
> +      - X
> +      - X
> +      - M\ :sub:`30`
> +      - X
> +      - X
> +    * - start + 12:
> +      - M\ :sub:`01`
> +      - X
> +      - X
> +      - M\ :sub:`11`
> +      - X
> +      - X
> +      - M\ :sub:`21`
> +      - X
> +      - X
> +      - M\ :sub:`31`
> +      - X
> +      - X
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index 33076af4dfdb..7cb6063f7056 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -1452,6 +1452,13 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
>  	case V4L2_PIX_FMT_Y210:		descr = "10-bit YUYV Packed"; break;
>  	case V4L2_PIX_FMT_Y212:		descr = "12-bit YUYV Packed"; break;
>  	case V4L2_PIX_FMT_Y216:		descr = "16-bit YUYV Packed"; break;
> +	case V4L2_META_FMT_GENERIC_8:	descr = "8-bit Generic Metadata"; break;
> +	case V4L2_META_FMT_GENERIC_CSI2_10:	descr = "8b Generic Meta, 10b CSI-2"; break;

You could write "8-bit Generic Meta, 10b CSI-2" and it would still fit.
Up to you.

> +	case V4L2_META_FMT_GENERIC_CSI2_12:	descr = "8b Generic Meta, 12b CSI-2"; break;
> +	case V4L2_META_FMT_GENERIC_CSI2_14:	descr = "8b Generic Meta, 14b CSI-2"; break;
> +	case V4L2_META_FMT_GENERIC_CSI2_16:	descr = "8b Generic Meta, 16b CSI-2"; break;
> +	case V4L2_META_FMT_GENERIC_CSI2_20:	descr = "8b Generic Meta, 20b CSI-2"; break;
> +	case V4L2_META_FMT_GENERIC_CSI2_24:	descr = "8b Generic Meta, 24b CSI-2"; break;
>  
>  	default:
>  		/* Compressed formats */
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index a8015e5e7fa4..6a4f8ae30186 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -839,6 +839,14 @@ struct v4l2_pix_format {
>  #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
>  #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
>  
> +#define V4L2_META_FMT_GENERIC_8		v4l2_fourcc('M', 'E', 'T', '8') /* Generic 8-bit metadata */
> +#define V4L2_META_FMT_GENERIC_CSI2_10	v4l2_fourcc('M', 'C', '1', 'A') /* 10-bit CSI-2 packed 8-bit metadata */
> +#define V4L2_META_FMT_GENERIC_CSI2_12	v4l2_fourcc('M', 'C', '1', 'C') /* 12-bit CSI-2 packed 8-bit metadata */
> +#define V4L2_META_FMT_GENERIC_CSI2_14	v4l2_fourcc('M', 'C', '1', 'E') /* 14-bit CSI-2 packed 8-bit metadata */
> +#define V4L2_META_FMT_GENERIC_CSI2_16	v4l2_fourcc('M', 'C', '1', 'G') /* 16-bit CSI-2 packed 8-bit metadata */
> +#define V4L2_META_FMT_GENERIC_CSI2_20	v4l2_fourcc('M', 'C', '1', 'K') /* 20-bit CSI-2 packed 8-bit metadata */
> +#define V4L2_META_FMT_GENERIC_CSI2_24	v4l2_fourcc('M', 'C', '1', 'O') /* 24-bit CSI-2 packed 8-bit metadata */
> +
>  /* priv field value to indicates that subsequent fields are valid. */
>  #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
>  

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 06/38] media: v4l: Support line-based metadata capture

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Wed, Mar 13, 2024 at 09:24:44AM +0200, Sakari Ailus wrote:
> many camera sensors, among other devices, transmit embedded data and image

s/many/Many/

> data for each CSI-2 frame. This embedded data typically contains register
> configuration of the sensor that has been used to capture the image data
> of the same frame.
> 
> The embedded data is received by the CSI-2 receiver and has the same
> properties as the image data, including that it is line based: it has
> width, height and bytesperline (stride).
> 
> Add these fields to struct v4l2_meta_format and document them.
> 
> Also add V4L2_FMT_FLAG_META_LINE_BASED to tell a given format is
> line-based i.e. these fields of struct v4l2_meta_format are valid for it.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../userspace-api/media/v4l/dev-meta.rst          | 15 +++++++++++++++
>  .../userspace-api/media/v4l/vidioc-enum-fmt.rst   |  7 +++++++
>  .../media/videodev2.h.rst.exceptions              |  1 +
>  drivers/media/v4l2-core/v4l2-ioctl.c              |  5 +++--
>  include/uapi/linux/videodev2.h                    | 10 ++++++++++
>  5 files changed, 36 insertions(+), 2 deletions(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/dev-meta.rst b/Documentation/userspace-api/media/v4l/dev-meta.rst
> index 0e7e1ee1471a..4dfd79e0a705 100644
> --- a/Documentation/userspace-api/media/v4l/dev-meta.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-meta.rst
> @@ -65,3 +65,18 @@ to 0.
>        - ``buffersize``
>        - Maximum buffer size in bytes required for data. The value is set by the
>          driver.
> +    * - __u32
> +      - ``width``
> +      - Width of a line of metadata in Data units. Valid when
> +	:c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set,
> +	otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
> +    * - __u32
> +      - ``height``
> +      - Number of rows of metadata. Valid when :c:type`v4l2_fmtdesc` flag
> +	``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
> +	:c:func:`VIDIOC_ENUM_FMT`.
> +    * - __u32
> +      - ``bytesperline``
> +      - Offset in bytes between the beginning of two consecutive lines. Valid
> +	when :c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is
> +	set, otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> index 000c154b0f98..a79abf4428c8 100644
> --- a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> +++ b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> @@ -227,6 +227,13 @@ the ``mbus_code`` field is handled differently:
>  	The application can ask to configure the quantization of the capture
>  	device when calling the :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl with
>  	:ref:`V4L2_PIX_FMT_FLAG_SET_CSC <v4l2-pix-fmt-flag-set-csc>` set.
> +    * - ``V4L2_FMT_FLAG_META_LINE_BASED``
> +      - 0x0200
> +      - The metadata format is line-based. In this case the ``width``,
> +	``height`` and ``bytesperline`` fields of :c:type:`v4l2_meta_format` are
> +	valid. The buffer consists of ``height`` lines, each having ``width``
> +	Data units of data and offset (in bytes) between the beginning of each

s/and offset/, and the offset/

> +	two consecutive lines is ``bytesperline``.
>  
>  Return Value
>  ============
> diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b/Documentation/userspace-api/media/videodev2.h.rst.exceptions
> index 3e58aac4ef0b..bdc628e8c1d6 100644
> --- a/Documentation/userspace-api/media/videodev2.h.rst.exceptions
> +++ b/Documentation/userspace-api/media/videodev2.h.rst.exceptions
> @@ -215,6 +215,7 @@ replace define V4L2_FMT_FLAG_CSC_XFER_FUNC fmtdesc-flags
>  replace define V4L2_FMT_FLAG_CSC_YCBCR_ENC fmtdesc-flags
>  replace define V4L2_FMT_FLAG_CSC_HSV_ENC fmtdesc-flags
>  replace define V4L2_FMT_FLAG_CSC_QUANTIZATION fmtdesc-flags
> +replace define V4L2_FMT_FLAG_META_LINE_BASED fmtdesc-flags
>  
>  # V4L2 timecode types
>  replace define V4L2_TC_TYPE_24FPS timecode-type
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index 7cb6063f7056..d125d23e4e61 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -343,8 +343,9 @@ static void v4l_print_format(const void *arg, bool write_only)
>  	case V4L2_BUF_TYPE_META_OUTPUT:
>  		meta = &p->fmt.meta;
>  		pixelformat = meta->dataformat;
> -		pr_cont(", dataformat=%p4cc, buffersize=%u\n",
> -			&pixelformat, meta->buffersize);
> +		pr_cont(", dataformat=%p4cc, buffersize=%u, width=%u, height=%u, bytesperline=%u\n",
> +			&pixelformat, meta->buffersize, meta->width,
> +			meta->height, meta->bytesperline);
>  		break;
>  	}
>  }
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index 6a4f8ae30186..88fdf190a437 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -877,6 +877,7 @@ struct v4l2_fmtdesc {
>  #define V4L2_FMT_FLAG_CSC_YCBCR_ENC		0x0080
>  #define V4L2_FMT_FLAG_CSC_HSV_ENC		V4L2_FMT_FLAG_CSC_YCBCR_ENC
>  #define V4L2_FMT_FLAG_CSC_QUANTIZATION		0x0100
> +#define V4L2_FMT_FLAG_META_LINE_BASED		0x0200

Could the V4L2_FMT_FLAG_META_LINE_BASED flag be set by the V4L2 core for
line-based metadata formats, instead of requiring drivers to set it ? It
would ensure consistency.

>  
>  	/* Frame Size and frame rate enumeration */
>  /*
> @@ -2423,10 +2424,19 @@ struct v4l2_sdr_format {
>   * struct v4l2_meta_format - metadata format definition
>   * @dataformat:		little endian four character code (fourcc)
>   * @buffersize:		maximum size in bytes required for data
> + * @width:		number of data units of data per line (valid for line
> + *			based formats only, see format documentation)
> + * @height:		number of lines of data per buffer (valid for line based
> + *			formats only)
> + * @bytesperline:	offset between the beginnings of two adjacent lines in
> + *			bytes (valid for line based formats only)
>   */
>  struct v4l2_meta_format {
>  	__u32				dataformat;
>  	__u32				buffersize;
> +	__u32				width;
> +	__u32				height;
> +	__u32				bytesperline;
>  } __attribute__ ((packed));
>  
>  /**

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 07/38] media: Documentation: Additional streams generally don't harm capture

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Wed, Mar 13, 2024 at 09:24:45AM +0200, Sakari Ailus wrote:
> Having extra streams on the source end of the link that cannot be captured
> by the sink sub-device generally are not an issue, at least not on CSI-2
> bus. Still document that there may be hardware specific limitations. For

s/hardware specific/hardware-specific/

> example on parallel bus this might not work on all cases.

s/bus/buses/

> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> Reviewed-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  Documentation/userspace-api/media/v4l/dev-subdev.rst | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> index f375b820ab68..a387e8a15b8d 100644
> --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> @@ -529,9 +529,9 @@ the its sink pad and allows to route them individually to one of its source
>  pads.
>  
>  Subdevice drivers that support multiplexed streams are compatible with
> -non-multiplexed subdev drivers, but, of course, require a routing configuration
> -where the link between those two types of drivers contains only a single
> -stream.
> +non-multiplexed subdev drivers. However, if the driver at the sink end of a link
> +does not support streams, then only the stream 0 on source end may be

s/the stream 0 on source end/stream 0 of the source/

> +captured. There may be additional hardware specific limitations.

s/hardware specific/hardware-specific/

Or maybe

There may be additional limitations specific to the sink device.

Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

>  
>  Understanding streams
>  ^^^^^^^^^^^^^^^^^^^^^

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 08/38] media: Documentation: Document embedded data guidelines for camera sensors

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Wed, Mar 13, 2024 at 09:24:46AM +0200, Sakari Ailus wrote:
> Document how embedded data support should be implemented for camera
> sensors, and when and how CCS embedded data format should be referenced.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../media/drivers/camera-sensor.rst           | 29 +++++++++++++++++++
>  1 file changed, 29 insertions(+)
> 
> diff --git a/Documentation/userspace-api/media/drivers/camera-sensor.rst b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> index 919a50e8b9d9..92545538b855 100644
> --- a/Documentation/userspace-api/media/drivers/camera-sensor.rst
> +++ b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> @@ -102,3 +102,32 @@ register programming sequences shall initialize the :ref:`V4L2_CID_HFLIP
>  values programmed by the register sequences. The default values of these
>  controls shall be 0 (disabled). Especially these controls shall not be inverted,
>  independently of the sensor's mounting rotation.
> +
> +Embedded data
> +-------------
> +
> +Many sensors, mostly raw sensors, support embedded data which is used to convey
> +the sensor configuration for the captured frame back to the host. While CSI-2 is
> +the most common bus used by such sensors, embedded data can be available on
> +other bus types as well.
> +
> +Embedded data support shall use an internal source pad (a pad that has

s/source pad/sink pad/

Or do we call those "internal source pad" ?

As this is uAPI documentation, it is mainly targetting (in my opinion)
userspace developers. I would write "Embedded data support uses ..." to
describe the behaviour seen from userspace, instead of using "shall" to
describe the requirements towards drivers.

> +``MEDIA_PAD_FL_SINK <MEDIA-PAD-FL-SINK>`` and ``MEDIA_PAD_FL_INTERNAL
> +<MEDIA-PAD-FL-INTERNAL>`` flags set) and route to the external pad. If embedded
> +data output can be disabled in hardware, it should be possible to disable the
> +embedded data route via ``VIDIOC_SUBDEV_S_ROUTING`` IOCTL.

You should mention the image pad here too:

Such sensors expose two internal sink pads (pads that have both the
``MEDIA_PAD_FL_SINK <MEDIA-PAD-FL-SINK>`` and ``MEDIA_PAD_FL_INTERNAL
<MEDIA-PAD-FL-INTERNAL>`` flags set) to model the source of the image and
embedded data streams. Each of those pads produces a single stream, and the
sub-device routes those streams to the external (source) pad. If embedded data
output can be disabled in hardware, the sub-device allows disabling the
embedded data route via the ``VIDIOC_SUBDEV_S_ROUTING`` IOCTL.

> +
> +In general, changing the embedded data format from the driver-configured values
> +is not supported. The height of the metadata is hardware specific and the width

s/hardware specific/device-specific/

> +is that (or less of that) of the image width, as configured on the pixel data
> +stream.
> +
> +CCS and non-CCS embedded data
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Embedded data which is compliant with CCS definitions shall use ``CCS embedded
> +data format <MEDIA-BUS-FMT-CCS-EMBEDDED>``. Device specific embedded data which

You should clarify here that you're talking about the internal sink pad.

s/Device specific/Only device-specific/

> +is compliant up to MIPI CCS embedded data levels 1 or 2 only shall refer to CCS

s/up to/with the/
s/only shall/may/

> +embedded data formats and document the level of conformance. The rest of the
> +device specific embedded data format shall be documented in the context of the

s/device specific/device-specific/

> +data format itself.

Not sure what you mean by the context in the last sentence.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 09/38] media: Documentation: v4l: Document internal source pads

[Laurent Pinchart]

On Tue, Mar 19, 2024 at 01:47:07PM +0000, Sakari Ailus wrote:
> On Fri, Mar 15, 2024 at 04:32:59PM +0100, Julien Massot wrote:
> > On 3/13/24 08:24, Sakari Ailus wrote:
> > > Document internal source pads, pads that have both SINK and INTERNAL flags
> > > set. Use the IMX219 camera sensor as an example.
> > > 
> > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > ---
> > >   .../userspace-api/media/v4l/dev-subdev.rst    | 145 ++++++++++++++++++
> > >   1 file changed, 145 insertions(+)
> > > 
> > > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > index a387e8a15b8d..1808f40f63e3 100644
> > > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > @@ -553,6 +553,27 @@ A stream at a specific point in the media pipeline is identified by the
> > >   sub-device and a (pad, stream) pair. For sub-devices that do not support
> > >   multiplexed streams the 'stream' field is always 0.
> > > +.. _v4l2-subdev-internal-source-pads:
> > > +
> > > +Internal source pads and routing
> > > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > +
> > > +Cases where a single sub-device source pad is traversed by multiple streams, one
> > > +or more of which originate from within the sub-device itself, are special as
> > > +there is no external sink pad for such routes. In those cases, the sources of
> > > +the internally generated streams are represented by internal source pads, which
> > > +are sink pads that have the :ref:`MEDIA_PAD_FL_INTERNAL <MEDIA-PAD-FL-INTERNAL>`
> > > +pad flag set.

This seems to answer a question in a previous patch, where I wondered if
you meant "internal sink pad" instead of "internal source pad".

Given that this is will most likely be a source of confusion, I think
you should explain this very clearly.

.. note::

   Internal source pads model the source of data *within* a sub-device, and are
   therefore conceptually located on the "sink" side of the sub-device from the
   point of view of sub-device internal routes. For this reason, and despite
   their name, they use the :ref:`MEDIA_PAD_FL_SINK <MEDIA-PAD-FL-SINK>` flag.
   When used in the routing API, they are addressed through the ``sink_pad``
   and ``sink_stream`` fields of the :c:type:`v4l2_subdev_route` structure.

I hope this will be enough... We'll need to be extra careful during
review in the beginning, making sure to always use the right terms.

> > > +
> > > +Internal pads have all the properties of an external pad, including formats and
> > > +selections. The format in this case is the source format of the stream. An
> > > +internal pad always has a single stream only (0).
> > > +
> > > +Routes from an internal source pad to an external source pad are typically not
> > > +modifiable but they can be activated and deactivated using the
> > > +:ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE <v4l2-subdev-routing-flags>` flag, depending
> > > +on driver capabilities.
> > > +
> > >   Interaction between routes, streams, formats and selections
> > >   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > @@ -668,3 +689,127 @@ To configure this pipeline, the userspace must take the following steps:
> > >      the configurations along the stream towards the receiver, using
> > >      :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each
> > >      stream endpoint in each sub-device.
> > > +
> > > +Internal pads setup example
> > > +^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > +
> > > +A simple example of a multiplexed stream setup might be as follows:
> > > +
> > > +- An IMX219 camera sensor source sub-device, with one sink pad (0), one source pad
> > > +  (1), an internal sink pad (2) that represents the source of embedded
> > 
> > The pixel pad is an internal pad as well ?
> 
> How about:
> 
> - An IMX219 camera sensor source sub-device, with one internal sink pad
>   (0) for image data, one source pad (1), an internal sink pad (2) that
>   represents the source of embedded

Sounds better, except you're now talking about internal sink pads, not
internal source pads :-) We have to pick one option and stick to it.
Same below.

> > > +  data. There are two routes, one from the sink pad to the source, and another

"from the internal image source pad to the source" sounds weird, so you
may need to write "from the internal image source pad to the external
source pad". If we instead talked about "internal sink pads" to model
the source of the streams, you could write "from the internal image sink
pad to the source pad".

> > > +  from the internal sink pad to the source pad. Both streams are always active,
> > > +  i.e. there is no need to separately enable the embedded data stream. The
> > > +  sensor uses the CSI-2 bus.
> > > +
> > > +- A CSI-2 receiver in the SoC (Receiver). The receiver has a single sink pad
> > > +  (pad 0), connected to the bridge, and two source pads (pads 1-2), going to the

I think you meant s/bridge/sensor/

> > > +  DMA engine. The receiver demultiplexes the incoming streams to the source

And "DMA engines" here ?

> > > +  pads.
> > > +
> > > +- DMA Engines in the SoC (DMA Engine), one for each stream. Each DMA engine is

s/Engines/engine/

> > > +  connected to a single source pad in the receiver.

s/in the/of the/

> > > +
> > > +The sensor, the bridge and the receiver are modeled as V4L2 sub-devices,

What bridge ? Same below.

> > > +exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are
> > > +modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
> > > +
> > > +To configure this pipeline, the userspace must take the following steps:
> > > +
> > > +1) Set up media links between entities: connect the sensors to the bridge,
> > > +   bridge to the receiver, and the receiver to the DMA engines. This step does
> > > +   not differ from normal non-multiplexed media controller setup.
> > > +
> > > +2) Configure routing
> > > +
> > > +.. flat-table:: Camera sensor. There are no configurable routes.
> > > +    :header-rows: 1
> > > +
> > > +    * - Sink Pad/Stream
> > > +      - Source Pad/Stream
> > > +      - Routing Flags
> > > +      - Comments
> > > +    * - 0/0
> > > +      - 1/0
> >
> > - 1/0
> > - 0/0
> >
> > > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > > +      - Pixel data stream from the sink pad
> > > +    * - 2/0
> > > +      - 1/1
> >
> > - 0/1
> >
> > > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > > +      - Metadata stream from the internal sink pad
> > 
> > In latest patch "[PATCH v6 05/15] media: i2c: imx219: Add embedded data
> > support"
> > we have:
> > - 0 -> source pad
> > - 1 -> pixel/image
> > - 2 -> EDATA
> > 
> > This is also what you did for ov2740.
> 
> Yes, these were leftovers from the CCS example. I've fixed them.
> 
> > With that fixed:
> > Reviewed-by Julien Massot <julien.massot@collabora.com>
> 
> Thank you!

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 10/38] media: Documentation: Document S_ROUTING behaviour

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Wed, Mar 13, 2024 at 09:24:48AM +0200, Sakari Ailus wrote:
> Document S_ROUTING behaviour for different devices.
> 
> Generally in devices that produce streams the streams are static and some
> can be enabled and disabled, whereas in devices that just transport them
> or write them to memory, more configurability is allowed. Document this.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../userspace-api/media/v4l/dev-subdev.rst    | 24 +++++++++++++++++++
>  1 file changed, 24 insertions(+)
> 
> diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> index 1808f40f63e3..08495cc6f4a6 100644
> --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> @@ -593,6 +593,30 @@ Any configurations of a stream within a pad, such as format or selections,
>  are independent of similar configurations on other streams. This is
>  subject to change in the future.
>  
> +Device types and routing setup
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +Different kinds of sub-devices have differing behaviour for route activation,
> +depending on the hardware. In all cases, however, only routes that have the
> +``V4L2_SUBDEV_STREAM_FL_ACTIVE`` flag set are active.
> +
> +Devices generating the streams may allow enabling and disabling some of the
> +routes or the configuration is fixed. If the routes can be disabled, not
> +declaring the routes (or declaring them without
> +``VIDIOC_SUBDEV_STREAM_FL_ACTIVE`` flag set) in ``VIDIOC_SUBDEV_S_ROUTING`` will
> +disable the routes while the sub-device driver retains the streams and their

No idea what you mean by "retains the streams and their configuration".

> +configuration. The ``VIDIOC_SUBDEV_S_ROUTING`` will still return such routes
> +back to the user in the routes array, with the ``V4L2_SUBDEV_STREAM_FL_ACTIVE``
> +flag unset.

I find this quite confusing. There's no explanation as to why the API
behaves this way. I think you should rewrite this documentation, trying
to evaluate it from the point of view of someone who doesn't know about
the routing API.

> +
> +Devices transporting the streams almost always have more configurability with
> +respect to routing. Typically any route between the sub-device's sink and source
> +pads is possible, and multiple routes (usually up to certain limited number) may
> +be active simultaneously. For such devices, no routes are created by the driver
> +and user-created routes are fully replaced when ``VIDIOC_SUBDEV_S_ROUTING`` is
> +called on the sub-device. Such newly created routes have the device's default
> +configuration for format and selection rectangles.
> +
>  Configuring streams
>  ^^^^^^^^^^^^^^^^^^^
>  

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 11/38] media: v4l: subdev: Add a function to lock two sub-device states, use it

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Wed, Mar 13, 2024 at 09:24:49AM +0200, Sakari Ailus wrote:
> Add two new functions, v4l2_subdev_lock_states() and
> v4l2_subdev_unclock_states(), to acquire and release the state of two
> sub-devices. They differ from calling v4l2_subdev_{un,}lock_state() so
> that if the two states share the same lock, the lock is acquired only
> once.

A bit of a hack, but I think we can live with it for now. A better long
term implementation will be to acquire all state locks from
media_pipeline_start() before doing any validation, and always using
different locks for different subdevs.

Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

> Also use the new functions in v4l2_subdev_link_validate().
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 12 +++-----
>  include/media/v4l2-subdev.h           | 40 +++++++++++++++++++++++++++
>  2 files changed, 44 insertions(+), 8 deletions(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 4c6198c48dd6..6c34df19ea2f 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -1400,17 +1400,13 @@ int v4l2_subdev_link_validate(struct media_link *link)
>  
>  	states_locked = sink_state && source_state;
>  
> -	if (states_locked) {
> -		v4l2_subdev_lock_state(sink_state);
> -		v4l2_subdev_lock_state(source_state);
> -	}
> +	if (states_locked)
> +		v4l2_subdev_lock_states(sink_state, source_state);
>  
>  	ret = v4l2_subdev_link_validate_locked(link, states_locked);
>  
> -	if (states_locked) {
> -		v4l2_subdev_unlock_state(sink_state);
> -		v4l2_subdev_unlock_state(source_state);
> -	}
> +	if (states_locked)
> +		v4l2_subdev_unlock_states(sink_state, source_state);
>  
>  	return ret;
>  }
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index a9e6b8146279..9cce48365975 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -1724,6 +1724,46 @@ static inline void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state)
>  	mutex_unlock(state->lock);
>  }
>  
> +/**
> + * v4l2_subdev_lock_states - Lock two sub-device states
> + * @state1: One subdevice state
> + * @state2: The other subdevice state
> + *
> + * Locks the state of two sub-devices.
> + *
> + * The states must be unlocked with v4l2_subdev_unlock_states() after use.
> + *
> + * This differs from calling v4l2_subdev_lock_state() on both states so that if
> + * the states share the same lock, the lock is acquired only once (so no
> + * deadlock occurs). The caller is responsible for ensuring the locks will
> + * always be acquired in the same order.
> + */
> +static inline void v4l2_subdev_lock_states(struct v4l2_subdev_state *state1,
> +					   struct v4l2_subdev_state *state2)
> +{
> +	mutex_lock(state1->lock);
> +	if (state1->lock != state2->lock)
> +		mutex_lock(state2->lock);
> +}
> +
> +/**
> + * v4l2_subdev_unlock_states() - Unlock two sub-device states
> + * @state1: One subdevice state
> + * @state2: The other subdevice state
> + *
> + * Unlocks the state of two sub-devices.
> + *
> + * This differs from calling v4l2_subdev_unlock_state() on both states so that
> + * if the states share the same lock, the lock is released only once.
> + */
> +static inline void v4l2_subdev_unlock_states(struct v4l2_subdev_state *state1,
> +					     struct v4l2_subdev_state *state2)
> +{
> +	mutex_unlock(state1->lock);
> +	if (state1->lock != state2->lock)
> +		mutex_unlock(state2->lock);
> +}
> +
>  /**
>   * v4l2_subdev_get_unlocked_active_state() - Checks that the active subdev state
>   *					     is unlocked and returns it

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 12/38] media: v4l: subdev: Move G_ROUTING handling below S_ROUTING

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Wed, Mar 13, 2024 at 09:24:50AM +0200, Sakari Ailus wrote:
> Move G_ROUTING IOCTL handling below that of S_ROUTING. G_ROUTING
> implementation will soon needed in handling S_ROUTING as well.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>

Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 54 +++++++++++++--------------
>  1 file changed, 27 insertions(+), 27 deletions(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 6c34df19ea2f..ec8de3f872b9 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -909,33 +909,6 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>  	case VIDIOC_SUBDEV_QUERYSTD:
>  		return v4l2_subdev_call(sd, video, querystd, arg);
>  
> -	case VIDIOC_SUBDEV_G_ROUTING: {
> -		struct v4l2_subdev_routing *routing = arg;
> -		struct v4l2_subdev_krouting *krouting;
> -
> -		if (!v4l2_subdev_enable_streams_api)
> -			return -ENOIOCTLCMD;
> -
> -		if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS))
> -			return -ENOIOCTLCMD;
> -
> -		memset(routing->reserved, 0, sizeof(routing->reserved));
> -
> -		krouting = &state->routing;
> -
> -		if (routing->num_routes < krouting->num_routes) {
> -			routing->num_routes = krouting->num_routes;
> -			return -ENOSPC;
> -		}
> -
> -		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
> -		       krouting->routes,
> -		       krouting->num_routes * sizeof(*krouting->routes));
> -		routing->num_routes = krouting->num_routes;
> -
> -		return 0;
> -	}
> -
>  	case VIDIOC_SUBDEV_S_ROUTING: {
>  		struct v4l2_subdev_routing *routing = arg;
>  		struct v4l2_subdev_route *routes =
> @@ -984,6 +957,33 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>  					routing->which, &krouting);
>  	}
>  
> +	case VIDIOC_SUBDEV_G_ROUTING: {
> +		struct v4l2_subdev_routing *routing = arg;
> +		struct v4l2_subdev_krouting *krouting;
> +
> +		if (!v4l2_subdev_enable_streams_api)
> +			return -ENOIOCTLCMD;
> +
> +		if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS))
> +			return -ENOIOCTLCMD;
> +
> +		memset(routing->reserved, 0, sizeof(routing->reserved));
> +
> +		krouting = &state->routing;
> +
> +		if (routing->num_routes < krouting->num_routes) {
> +			routing->num_routes = krouting->num_routes;
> +			return -ENOSPC;
> +		}
> +
> +		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
> +		       krouting->routes,
> +		       krouting->num_routes * sizeof(*krouting->routes));
> +		routing->num_routes = krouting->num_routes;
> +
> +		return 0;
> +	}
> +
>  	case VIDIOC_SUBDEV_G_CLIENT_CAP: {
>  		struct v4l2_subdev_client_capability *client_cap = arg;
>  

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 13/38] media: v4l: subdev: Copy argument back to user also for S_ROUTING

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Wed, Mar 13, 2024 at 09:24:51AM +0200, Sakari Ailus wrote:
> As the user needs to know what went wrong for S_ROUTING, copy array
> arguments back to the user.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  drivers/media/v4l2-core/v4l2-ioctl.c | 9 ++++++---
>  1 file changed, 6 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index d125d23e4e61..95bd56145d38 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -3415,11 +3415,14 @@ video_usercopy(struct file *file, unsigned int orig_cmd, unsigned long arg,
>  	 * FIXME: subdev IOCTLS are partially handled here and partially in
>  	 * v4l2-subdev.c and the 'always_copy' flag can only be set for IOCTLS
>  	 * defined here as part of the 'v4l2_ioctls' array. As
> -	 * VIDIOC_SUBDEV_G_ROUTING needs to return results to applications even
> -	 * in case of failure, but it is not defined here as part of the
> +	 * VIDIOC_SUBDEV_[GS]_ROUTING needs to return results to applications

s/needs/need/

Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

> +	 * even in case of failure, but it is not defined here as part of the
>  	 * 'v4l2_ioctls' array, insert an ad-hoc check to address that.
>  	 */
> -	if (err < 0 && !always_copy && cmd != VIDIOC_SUBDEV_G_ROUTING)
> +	if (cmd == VIDIOC_SUBDEV_G_ROUTING || cmd == VIDIOC_SUBDEV_S_ROUTING)
> +		always_copy = true;
> +
> +	if (err < 0 && !always_copy)
>  		goto out;
>  
>  	if (has_array_args) {

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 14/38] media: v4l: subdev: Add len_routes field to struct v4l2_subdev_routing

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Wed, Mar 13, 2024 at 09:24:52AM +0200, Sakari Ailus wrote:
> The len_routes field is used to tell the size of the routes array in
> struct v4l2_subdev_routing. This way the number of routes returned from
> S_ROUTING IOCTL may be larger than the number of routes provided, in case
> there are more routes returned by the driver.
> 
> Note that this uAPI is still disabled in the code, so this change can
> safely be done. Anyone who manually patched the code to enable this uAPI
> must update their code.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../media/v4l/vidioc-subdev-g-routing.rst     | 33 ++++++++++++-------
>  drivers/media/v4l2-core/v4l2-ioctl.c          |  4 +--
>  drivers/media/v4l2-core/v4l2-subdev.c         |  6 +++-
>  include/media/v4l2-subdev.h                   |  2 ++
>  include/uapi/linux/v4l2-subdev.h              |  9 +++--
>  5 files changed, 37 insertions(+), 17 deletions(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> index 26b5004bfe6d..8f9aa83275c5 100644
> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> @@ -46,20 +46,28 @@ with the ``VIDIOC_SUBDEV_S_ROUTING`` ioctl, by adding or removing routes and
>  setting or clearing flags of the  ``flags`` field of a
>  struct :c:type:`v4l2_subdev_route`.
>  
> -All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called. This
> -means that the userspace must reconfigure all streams after calling the ioctl
> -with e.g. ``VIDIOC_SUBDEV_S_FMT``.
> +All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is
> +called. This means that the userspace must reconfigure all streams after calling
> +the ioctl with e.g. ``VIDIOC_SUBDEV_S_FMT``.

Unrelated. If you really want to reflow, the first line is too short,
'called.' should be moved there.

>  
>  Only subdevices which have both sink and source pads can support routing.
>  
> -When inspecting routes through ``VIDIOC_SUBDEV_G_ROUTING`` and the application
> -provided ``num_routes`` is not big enough to contain all the available routes
> -the subdevice exposes, drivers return the ENOSPC error code and adjust the
> -value of the ``num_routes`` field. Application should then reserve enough memory
> -for all the route entries and call ``VIDIOC_SUBDEV_G_ROUTING`` again.
> +The ``num_routes`` field is used to denote the number of routes set (set by user
> +space on ``VIDIOC_SUBDEV_S_ROUTING`` argument) on the routing table as well as
> +the number of routes returned back from both IOCTLs. ``len_routes`` signifies
> +the number of routes that can fit into the ``routes`` array. The userspace shall
> +set ``len_routes`` for both IOCTLs and ``num_routes`` for
> +``VIDIOC_SUBDEV_S_ROUTING``.
>  
> -On a successful ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the
> -``num_routes`` field to reflect the actual number of routes returned.
> +On a ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the ``num_routes``
> +field to reflect the actual number of routes known by the driver.
> +``num_routes`` larger than ``len_routes`` may be returned by both IOCTLs to
> +indicate there are more routes than fits to the ``routes`` array. In this
> +case first ``len_routes`` were returned back to the userspace in the
> +``routes`` array. This is not considered as an error.
> +
> +Also ``VIDIOC_SUBDEV_S_ROUTING`` may return more route than the user provided in
> +``num_routes`` field due to e.g. hardware properties.
>  

An attempt at making this clearer (in my opinion, feel free to pick the
pieces you like, or reject it outright).


The ``len_routes`` field indicates the number of routes that can fit in the
``routes`` array allocated by userspace. It is set by applications for both
ioctls to indicate how many routes the kernel can return, and is never modified
by the kernel.

The ``num_routes`` field indicates the number of routes in the subdevice routing
table. For ``VIDIOC_SUBDEV_S_ROUTING``, it is set by userspace to the number of
routes that the application stored in the ``routes`` array. For both ioctls, it
is returned by the kernel and indicates how many routes are stored in the
subdevice routing table. This may be smaller or larger than the value of
``num_routes`` set by the application for ``VIDIOC_SUBDEV_S_ROUTING``, as
drivers may adjust the requested routing table.

The kernel can return a ``num_routes`` value larger than ``len_routes`` from
both ioctls. This indicates thare are more routes in the routing table than fits
the ``routes`` array. In this case, the ``routes`` array is filled by the kernel
with the first ``len_routes`` entries of the subdevice routing table. This is
not considered an array, and the ioctl call succeeds. If the applications wants
to retrieve the missing routes, it can issue a new ``VIDIOC_SUBDEV_G_ROUTING``
call with a large enough ``routes`` array.

>  .. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
>  
> @@ -74,6 +82,9 @@ On a successful ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the
>        - ``which``
>        - Routing table to be accessed, from enum
>          :ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
> +    * - __u32
> +      - ``len_routes``
> +      - The length of the array (as in memory reserved for the array)
>      * - struct :c:type:`v4l2_subdev_route`
>        - ``routes[]``
>        - Array of struct :c:type:`v4l2_subdev_route` entries
> @@ -81,7 +92,7 @@ On a successful ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the
>        - ``num_routes``
>        - Number of entries of the routes array
>      * - __u32
> -      - ``reserved``\ [5]
> +      - ``reserved``\ [11]
>        - Reserved for future extensions. Applications and drivers must set
>  	the array to zero.
>  
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index 95bd56145d38..018e7efc21ca 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -3155,13 +3155,13 @@ static int check_array_args(unsigned int cmd, void *parg, size_t *array_size,
>  	case VIDIOC_SUBDEV_S_ROUTING: {
>  		struct v4l2_subdev_routing *routing = parg;
>  
> -		if (routing->num_routes > 256)
> +		if (routing->len_routes > 256)
>  			return -E2BIG;
>  
>  		*user_ptr = u64_to_user_ptr(routing->routes);
>  		*kernel_ptr = (void **)&routing->routes;
>  		*array_size = sizeof(struct v4l2_subdev_route)
> -			    * routing->num_routes;
> +			    * routing->len_routes;
>  		ret = 1;
>  		break;
>  	}
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index ec8de3f872b9..486a5c3016de 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -925,6 +925,9 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>  		if (routing->which != V4L2_SUBDEV_FORMAT_TRY && ro_subdev)
>  			return -EPERM;
>  
> +		if (routing->num_routes > routing->len_routes)
> +			return -EINVAL;
> +
>  		memset(routing->reserved, 0, sizeof(routing->reserved));
>  
>  		for (i = 0; i < routing->num_routes; ++i) {
> @@ -951,6 +954,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>  		}
>  
>  		krouting.num_routes = routing->num_routes;
> +		krouting.len_routes = routing->len_routes;
>  		krouting.routes = routes;
>  
>  		return v4l2_subdev_call(sd, pad, set_routing, state,
> @@ -971,7 +975,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>  
>  		krouting = &state->routing;
>  
> -		if (routing->num_routes < krouting->num_routes) {
> +		if (routing->len_routes < krouting->num_routes) {
>  			routing->num_routes = krouting->num_routes;
>  			return -ENOSPC;

Do we still want to return -ENOSPC here ?

>  		}
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 9cce48365975..1df6b963a1c9 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -728,12 +728,14 @@ struct v4l2_subdev_stream_configs {
>  /**
>   * struct v4l2_subdev_krouting - subdev routing table
>   *
> + * @len_routes: length of routes array, in routes
>   * @num_routes: number of routes
>   * @routes: &struct v4l2_subdev_route
>   *
>   * This structure contains the routing table for a subdev.
>   */
>  struct v4l2_subdev_krouting {
> +	unsigned int len_routes;
>  	unsigned int num_routes;
>  	struct v4l2_subdev_route *routes;
>  };
> diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
> index 7048c51581c6..ca543982460c 100644
> --- a/include/uapi/linux/v4l2-subdev.h
> +++ b/include/uapi/linux/v4l2-subdev.h
> @@ -224,15 +224,18 @@ struct v4l2_subdev_route {
>   * struct v4l2_subdev_routing - Subdev routing information
>   *
>   * @which: configuration type (from enum v4l2_subdev_format_whence)
> - * @num_routes: the total number of routes in the routes array
> + * @len_routes: the length of the routes array, in routes
>   * @routes: pointer to the routes array
> + * @num_routes: the total number of routes, possibly more than fits in the
> + *		routes array
>   * @reserved: drivers and applications must zero this array
>   */
>  struct v4l2_subdev_routing {
>  	__u32 which;
> -	__u32 num_routes;
> +	__u32 len_routes;
>  	__u64 routes;
> -	__u32 reserved[6];
> +	__u32 num_routes;
> +	__u32 reserved[11];
>  };
>  
>  /*

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 15/38] media: v4l: subdev: Return routes set using S_ROUTING

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Wed, Mar 13, 2024 at 09:24:53AM +0200, Sakari Ailus wrote:
> Return the routes set using S_ROUTING back to the user. Also reflect this
> in documentation.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../media/v4l/vidioc-subdev-g-routing.rst            |  5 +++--
>  drivers/media/v4l2-core/v4l2-subdev.c                | 12 +++++++++++-
>  2 files changed, 14 insertions(+), 3 deletions(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> index 8f9aa83275c5..6eb6a59570dc 100644
> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> @@ -43,8 +43,9 @@ The routing configuration determines the flows of data inside an entity.
>  Drivers report their current routing tables using the
>  ``VIDIOC_SUBDEV_G_ROUTING`` ioctl and application may enable or disable routes
>  with the ``VIDIOC_SUBDEV_S_ROUTING`` ioctl, by adding or removing routes and
> -setting or clearing flags of the  ``flags`` field of a
> -struct :c:type:`v4l2_subdev_route`.
> +setting or clearing flags of the ``flags`` field of a struct
> +:c:type:`v4l2_subdev_route`. Similarly to ``VIDIOC_SUBDEV_G_ROUTING``, also
> +``VIDIOC_SUBDEV_S_ROUTING`` returns the routes back to the user.
>  
>  All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is
>  called. This means that the userspace must reconfigure all streams after calling
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 486a5c3016de..a357ce318192 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -957,8 +957,18 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>  		krouting.len_routes = routing->len_routes;
>  		krouting.routes = routes;
>  
> -		return v4l2_subdev_call(sd, pad, set_routing, state,
> +		rval = v4l2_subdev_call(sd, pad, set_routing, state,
>  					routing->which, &krouting);
> +		if (rval < 0)
> +			return rval;
> +
> +		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
> +		       state->routing.routes,
> +		       state->routing.num_routes *

I think this should be capped at len_routes.

> +		       sizeof(*state->routing.routes));
> +		routing->num_routes = state->routing.num_routes;
> +
> +		return 0;
>  	}
>  
>  	case VIDIOC_SUBDEV_G_ROUTING: {
> -- 
> 2.39.2
> 

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 16/38] media: v4l: subdev: Allow a larger number of routes than there's room for

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Wed, Mar 13, 2024 at 09:24:54AM +0200, Sakari Ailus wrote:
> On VIDIOC_SUBDEV_[GS]_ROUTING, only return as many routes back to the user
> as there's room. Do not consider it an error if more routes existed.
> Simply inform the user there are more routes.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../media/v4l/vidioc-subdev-g-routing.rst              |  4 ----
>  drivers/media/v4l2-core/v4l2-subdev.c                  | 10 +++-------
>  2 files changed, 3 insertions(+), 11 deletions(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> index 6eb6a59570dc..08b8d17cef3f 100644
> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> @@ -147,10 +147,6 @@ On success 0 is returned, on error -1 and the ``errno`` variable is set
>  appropriately. The generic error codes are described at the
>  :ref:`Generic Error Codes <gen-errors>` chapter.
>  
> -ENOSPC
> -   The application provided ``num_routes`` is not big enough to contain
> -   all the available routes the subdevice exposes.
> -

I wonder if this patch should be squashed with the previous two. You
update the documentation in 14/38 already.

>  EINVAL
>     The sink or source pad identifiers reference a non-existing pad or reference
>     pads of different types (ie. the sink_pad identifiers refers to a source
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index a357ce318192..a6107e440ef0 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -964,7 +964,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>  
>  		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
>  		       state->routing.routes,
> -		       state->routing.num_routes *
> +		       min(state->routing.num_routes, routing->len_routes)  *
>  		       sizeof(*state->routing.routes));
>  		routing->num_routes = state->routing.num_routes;
>  
> @@ -985,14 +985,10 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>  
>  		krouting = &state->routing;
>  
> -		if (routing->len_routes < krouting->num_routes) {
> -			routing->num_routes = krouting->num_routes;
> -			return -ENOSPC;
> -		}
> -
>  		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
>  		       krouting->routes,
> -		       krouting->num_routes * sizeof(*krouting->routes));
> +		       min(krouting->num_routes, routing->len_routes) *
> +		       sizeof(*krouting->routes));
>  		routing->num_routes = krouting->num_routes;
>  
>  		return 0;

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 17/38] media: v4l: subdev: Add trivial set_routing support

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Wed, Mar 13, 2024 at 09:24:55AM +0200, Sakari Ailus wrote:
> Add trivial S_ROUTING IOCTL support for drivers where routing is static.
> Essentially this means returning the same information G_ROUTING call would
> have done.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>

Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 14 ++++++++++++++
>  1 file changed, 14 insertions(+)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index a6107e440ef0..c8c435df92c8 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -930,6 +930,20 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>  
>  		memset(routing->reserved, 0, sizeof(routing->reserved));
>  
> +		/*
> +		 * If the driver doesn't support setting routing, just return
> +		 * the routing table here.
> +		 */
> +		if (!v4l2_subdev_has_op(sd, pad, set_routing)) {
> +			memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
> +			       state->routing.routes,
> +			       min(state->routing.num_routes, routing->len_routes) *
> +			       sizeof(*state->routing.routes));
> +			routing->num_routes = state->routing.num_routes;
> +
> +			return 0;
> +		}
> +
>  		for (i = 0; i < routing->num_routes; ++i) {
>  			const struct v4l2_subdev_route *route = &routes[i];
>  			const struct media_pad *pads = sd->entity.pads;

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 01/38] media: mc: Add INTERNAL pad flag

[Sakari Ailus]

Hi Laurent,

On Wed, Mar 20, 2024 at 12:17:07AM +0200, Laurent Pinchart wrote:
> On Thu, Mar 14, 2024 at 09:17:08AM +0200, Tomi Valkeinen wrote:
> > On 13/03/2024 09:24, Sakari Ailus wrote:
> > > Internal source pads will be used as routing endpoints in V4L2
> > > [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
> > > Internal source pads are pads that have both SINK and INTERNAL flags set.
> > > 
> > > Also prevent creating links to pads that have been flagged as internal and
> > > initialising SOURCE pads with INTERNAL flag set.
> > > 
> > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > ---
> > >   .../userspace-api/media/mediactl/media-types.rst       |  8 ++++++++
> > >   drivers/media/mc/mc-entity.c                           | 10 ++++++++--
> > >   include/uapi/linux/media.h                             |  1 +
> > >   3 files changed, 17 insertions(+), 2 deletions(-)
> > > 
> > > diff --git a/Documentation/userspace-api/media/mediactl/media-types.rst b/Documentation/userspace-api/media/mediactl/media-types.rst
> > > index 6332e8395263..f55ef055bcf8 100644
> > > --- a/Documentation/userspace-api/media/mediactl/media-types.rst
> > > +++ b/Documentation/userspace-api/media/mediactl/media-types.rst
> > > @@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
> > >   .. _MEDIA-PAD-FL-SINK:
> > >   .. _MEDIA-PAD-FL-SOURCE:
> > >   .. _MEDIA-PAD-FL-MUST-CONNECT:
> > > +.. _MEDIA-PAD-FL-INTERNAL:
> > >   
> > >   .. flat-table:: Media pad flags
> > >       :header-rows:  0
> > > @@ -381,6 +382,13 @@ Types and flags used to represent the media graph elements
> > >   	  enabled links even when this flag isn't set; the absence of the flag
> > >   	  doesn't imply there is none.
> > >   
> > > +    *  -  ``MEDIA_PAD_FL_INTERNAL``
> > > +       -  The internal flag indicates an internal pad that has no external
> > > +	  connections. Such a pad shall not be connected with a link.
> 
> I would expand this slightly, as it's the only source of documentation
> regarding internal pads.

Patch 9 adds more documentation, this patch is for MC only.

> 
>        -  The internal flag indicates an internal pad that has no external
> 	  connections. This can be used to model, for instance, the pixel array
> 	  internal to an image sensor. As they are internal to entities,
> 	  internal pads shall not be connected with links.

I'd drop the sentence related to sensors.

> 
> > > +
> > > +	  The internal flag may currently be present only in a source pad where
> > 
> > s/source/sink/
> > 
> > Reviewed-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > > +	  it indicates that the :ref:``stream <media-glossary-stream>``
> > > +	  originates from within the entity.
> > >   
> > >   One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
> > >   must be set for every pad.
> > > diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> > > index 0e28b9a7936e..1973e9e1013e 100644
> > > --- a/drivers/media/mc/mc-entity.c
> > > +++ b/drivers/media/mc/mc-entity.c
> > > @@ -213,7 +213,9 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
> > >   		iter->index = i++;
> > >   
> > >   		if (hweight32(iter->flags & (MEDIA_PAD_FL_SINK |
> > > -					     MEDIA_PAD_FL_SOURCE)) != 1) {
> > > +					     MEDIA_PAD_FL_SOURCE)) != 1 ||
> > > +		    (iter->flags & MEDIA_PAD_FL_INTERNAL &&
> > > +		     !(iter->flags & MEDIA_PAD_FL_SINK))) {
> > >   			ret = -EINVAL;
> > >   			break;
> > >   		}
> 
> This may become more readable written as a switch statement:
> 
> 		const u32 pad_flags_mask = MEDIA_PAD_FL_SINK |
> 					   MEDIA_PAD_FL_SOURCE |
> 					   MEDIA_PAD_FL_INTERNAL;
> 
> 		switch (iter->flags & pad_flags_mask) {
> 		case MEDIA_PAD_FL_SINK:
> 		case MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL:
> 		case MEDIA_PAD_FL_SOURCE:
> 			break;
> 
> 		default:
> 			ret = -EINVAL;
> 			break;
> 		}
> 
> 		if (ret)
> 			break;
> 
> And now that I've written this, I'm not too sure anymore :-) Another
> option would be
> 
> 
> 	const u32 pad_flags = iter->flags & (MEDIA_PAD_FL_SINK |
> 					     MEDIA_PAD_FL_SOURCE |
> 					     MEDIA_PAD_FL_INTERNAL);
> 
> 	if (pad_flags != MEDIA_PAD_FL_SINK &&
> 	    pad_flags != MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL
> 	    pad_flags != MEDIA_PAD_FL_SOURCE) {
> 		ret = -EINVAL;
> 		break;
> 	}
> 
> Up to you.

I'd prefer to keep it as-is since the original check is testing two
independent things instead of merging them: that only either SINK or SOURCE
is set, and then separately that if INTERNAL is set, then SINK is set, too.

Of the two options you suggested, I prefer the latter.

> 
> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Thank you!

> 
> > > @@ -1112,7 +1114,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
> > >   
> > >   	for (i = 0; i < entity->num_pads; i++) {
> > >   		if ((entity->pads[i].flags &
> > > -		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
> > > +		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
> > > +		      MEDIA_PAD_FL_INTERNAL)) != pad_type)
> > >   			continue;
> > >   
> > >   		if (entity->pads[i].sig_type == sig_type)
> > > @@ -1142,6 +1145,9 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
> > >   		return -EINVAL;
> > >   	if (WARN_ON(!(sink->pads[sink_pad].flags & MEDIA_PAD_FL_SINK)))
> > >   		return -EINVAL;
> > > +	if (WARN_ON(source->pads[source_pad].flags & MEDIA_PAD_FL_INTERNAL) ||
> > > +	    WARN_ON(sink->pads[sink_pad].flags & MEDIA_PAD_FL_INTERNAL))
> > > +		return -EINVAL;
> > >   
> > >   	link = media_add_link(&source->links);
> > >   	if (link == NULL)
> > > diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
> > > index 1c80b1d6bbaf..80cfd12a43fc 100644
> > > --- a/include/uapi/linux/media.h
> > > +++ b/include/uapi/linux/media.h
> > > @@ -208,6 +208,7 @@ struct media_entity_desc {
> > >   #define MEDIA_PAD_FL_SINK			(1U << 0)
> > >   #define MEDIA_PAD_FL_SOURCE			(1U << 1)
> > >   #define MEDIA_PAD_FL_MUST_CONNECT		(1U << 2)
> > > +#define MEDIA_PAD_FL_INTERNAL			(1U << 3)
> > >   
> > >   struct media_pad_desc {
> > >   	__u32 entity;		/* entity ID */
> 

-- 
Regards,

Sakari Ailus

Re: [PATCH v8 03/38] media: uapi: Add generic serial metadata mbus formats

[Sakari Ailus]

Moi,

On Tue, Mar 19, 2024 at 04:20:35PM +0200, Tomi Valkeinen wrote:
> On 19/03/2024 15:27, Sakari Ailus wrote:
> > Moi,
> > 
> > On Thu, Mar 14, 2024 at 09:30:50AM +0200, Tomi Valkeinen wrote:
> > > On 13/03/2024 09:24, Sakari Ailus wrote:
> > > > Add generic serial metadata mbus formats. These formats describe data
> > > > width and packing but not the content itself. The reason for specifying
> > > > such formats is that the formats as such are fairly device specific but
> > > > they are still handled by CSI-2 receiver drivers that should not be aware
> > > > of device specific formats. What makes generic metadata formats possible
> > > > is that these formats are parsed by software only, after capturing the
> > > > data to system memory.
> > > > 
> > > > Also add a definition for "Data unit" to cover what is essentially a pixel
> > > > but is not image data.
> > > 
> > > The CCS spec talks about legacy packing and optimized packing for 16+ bit
> > > formats. You cover only the "legacy" ones here. Did you look at those?
> > 
> > The reason is that the bus data layout of the new packing at higher bit
> > depth matches with packing at lower bit depths (half to be precise). That's
> > why there's no need to define formats for the new packing methods at higher
> > bit depths (the driver simply uses the packing at half of the bit depth).
> 
> Hmm. If we're capturing 10-bit raw format, say, with the width of 640
> pixels, we'll configure the video stream format according to those. For the
> embedded stream, we'll set it to V4L2_META_FMT_GENERIC_CSI2_10 and 640
> width, right?
> 
> If we're capturing 20-bit raw, we'll configure the video stream format again
> accordingly, width to 640, and 20 bit fourcc/mbus code. If the embedded
> stream uses the "legacy" packing, we'll set the format to
> V4L2_META_FMT_GENERIC_CSI2_20 with width of 640, right?
> 
> But if it's using packed format for the embedded stream, we set the format
> to V4L2_META_FMT_GENERIC_CSI2_10 and width to 1280?

The driver sets the embedded data sub-device format apart from the mbus
code. The width would be 1280 in this case, yes, and fewer embedded data
lines could be needed.

> 
> Considering that the video and (line-based) embedded data come from the same
> source, I'd expect the widths to be the same.

The width in bytes (with the packing) of embedded data is still the same as
for the image data. Pixels don't matter in this case.

-- 
Terveisin,

Sakari Ailus

Re: [PATCH v8 03/38] media: uapi: Add generic serial metadata mbus formats

[Sakari Ailus]

Hi Laurent,

On Wed, Mar 20, 2024 at 01:00:48AM +0200, Laurent Pinchart wrote:
> On Wed, Mar 20, 2024 at 12:33:48AM +0200, Laurent Pinchart wrote:
> > On Tue, Mar 19, 2024 at 04:20:35PM +0200, Tomi Valkeinen wrote:
> > > On 19/03/2024 15:27, Sakari Ailus wrote:
> > > > On Thu, Mar 14, 2024 at 09:30:50AM +0200, Tomi Valkeinen wrote:
> > > >> On 13/03/2024 09:24, Sakari Ailus wrote:
> > > >>> Add generic serial metadata mbus formats. These formats describe data
> > > >>> width and packing but not the content itself. The reason for specifying
> > > >>> such formats is that the formats as such are fairly device specific but
> > > >>> they are still handled by CSI-2 receiver drivers that should not be aware
> > > >>> of device specific formats. What makes generic metadata formats possible
> > > >>> is that these formats are parsed by software only, after capturing the
> > > >>> data to system memory.
> > > >>>
> > > >>> Also add a definition for "Data unit" to cover what is essentially a pixel
> > > >>> but is not image data.
> > > >>
> > > >> The CCS spec talks about legacy packing and optimized packing for 16+ bit
> > > >> formats. You cover only the "legacy" ones here. Did you look at those?
> > > > 
> > > > The reason is that the bus data layout of the new packing at higher bit
> > > > depth matches with packing at lower bit depths (half to be precise). That's
> > > > why there's no need to define formats for the new packing methods at higher
> > > > bit depths (the driver simply uses the packing at half of the bit depth).
> > > 
> > > Hmm. If we're capturing 10-bit raw format, say, with the width of 640 
> > > pixels, we'll configure the video stream format according to those. For 
> > > the embedded stream, we'll set it to V4L2_META_FMT_GENERIC_CSI2_10 and 
> > > 640 width, right?
> > > 
> > > If we're capturing 20-bit raw, we'll configure the video stream format 
> > > again accordingly, width to 640, and 20 bit fourcc/mbus code. If the 
> > > embedded stream uses the "legacy" packing, we'll set the format to 
> > > V4L2_META_FMT_GENERIC_CSI2_20 with width of 640, right?
> > > 
> > > But if it's using packed format for the embedded stream, we set the 
> > > format to V4L2_META_FMT_GENERIC_CSI2_10 and width to 1280?
> > > 
> > > Considering that the video and (line-based) embedded data come from the 
> > > same source, I'd expect the widths to be the same.
> > 
> > I don't have a strong objection against multiplying the width, but we
> > need to figure out the impact on other kernel space components, as well
> > as on userspace. I suppose the media bus code for the embedded data
> > stream on the sensor source pad when using optimized packing and
> > capturing RAW20 images would be MEDIA_BUS_FMT_META_10 ? In that case I
> > think the sensor driver should be able to handle the width calculations
> > on its own, and the value would just be propagated by userspace.
> 
> This should be documented somewhere in this series by the way (not in
> this patch).

This could go to the CCS driver documentation. I modified the last
paragraph and added a new one:

------8<-----------
Devices supporting embedded data for bit depths greater than or equal to 16 may
support more dense packing or legacy single metadata byte per data unit, or both
of these. The supported embedded data formats can be enumerated and configured
on stream 1 of the source pad (1) of the CCS source sub-device.

The use of the denser packing results in embedded data lines being longer than
the pixel data in data units since the data units are smaller. In bytes the
embedded data lines are still not longer than the image data lines.
------8<-----------

I believe the reason for the specs requiring embedded data lines not being
longer (in bytes) is most likely that some hardware may have issues
reciving the data otherwise for various reasons.

-- 
Regards,

Sakari Ailus

Re: [PATCH v8 03/38] media: uapi: Add generic serial metadata mbus formats

[Sakari Ailus]

Hi Laurent,

On Wed, Mar 20, 2024 at 12:59:48AM +0200, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.
> 
> On Wed, Mar 13, 2024 at 09:24:41AM +0200, Sakari Ailus wrote:
> > Add generic serial metadata mbus formats. These formats describe data
> > width and packing but not the content itself. The reason for specifying
> > such formats is that the formats as such are fairly device specific but
> > they are still handled by CSI-2 receiver drivers that should not be aware
> > of device specific formats. What makes generic metadata formats possible
> > is that these formats are parsed by software only, after capturing the
> > data to system memory.
> > 
> > Also add a definition for "Data unit" to cover what is essentially a pixel
> > but is not image data.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  .../userspace-api/media/glossary.rst          |   9 +
> >  .../media/v4l/subdev-formats.rst              | 258 ++++++++++++++++++
> >  include/uapi/linux/media-bus-format.h         |   9 +
> >  3 files changed, 276 insertions(+)
> > 
> > diff --git a/Documentation/userspace-api/media/glossary.rst b/Documentation/userspace-api/media/glossary.rst
> > index ef0ab601b5bf..7078141894c5 100644
> > --- a/Documentation/userspace-api/media/glossary.rst
> > +++ b/Documentation/userspace-api/media/glossary.rst
> > @@ -25,6 +25,15 @@ Glossary
> >  
> >  	See :ref:`cec`.
> >  
> > +.. _media-glossary-data-unit:
> > +
> > +    Data unit
> > +
> > +	Unit of data transported by a bus. On parallel buses, the data unit
> > +	consists of one or more related samples while on serial buses the data
> > +	unit is logical. If the data unit is image data, it may also be called a
> > +	pixel.
> 
> I'm pretty sure nobody will be able to understand what this means, but I
> don't have a better proposal at the moment.
> 
> > +
> >      Device Driver
> >  	Part of the Linux Kernel that implements support for a hardware
> >  	component.
> > diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > index eb3cd20b0cf2..cbd475f7cae9 100644
> > --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > @@ -8306,3 +8306,261 @@ The following table lists the existing metadata formats.
> >  	both sides of the link and the bus format is a fixed
> >  	metadata format that is not configurable from userspace.
> >  	Width and height will be set to 0 for this format.
> > +
> > +Generic Serial Metadata Formats
> > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > +
> > +Generic serial metadata formats are used on serial buses where the actual data
> > +content is more or less device specific but the data is transmitted and received
> > +by multiple devices that do not process the data in any way, simply writing
> > +it to system memory for processing in software at the end of the pipeline.
> > +
> > +The more specific variant describing the actual data is used on the internal
> > +source pad of the originating sub-device.
> 
> Maybe this paragraph would be best added in the patch that adds the
> specific metadata formats, you could then mention one of them as an
> example:
> 
> The exact format of the data generated by the device is reported on the
> internal source pad of the originating sub-device, using one of the more
> specific metadata formats such as MEDIA_BUS_FMT_CCS_EMBEDDED.

I'd really like to get rid of the "internal source" pads as the naming is
really confusing (it's present still in this version but not in many
locations). They're sink pads after all, so I'd call them such. In a few
locations there's text that explains they do represent sources of data
within the sub-device itself.

> 
> > +
> > +"b" in an array cell signifies a byte of data, followed by the number of the bit
> 
> s/bit$/byte/

Uh, yes.

> 
> > +and finally the bit number in subscript. "X" indicates a padding bit.
> 
> We use a lower-case x in pixfmt-rgb.rst, I would do the same here. We
> also use single quotes there, turning "b" and "x" into 'b' and 'x'.

This is documentation, not C source code where we'd want to denote a single
character. Double quotes should thus be used instead.

I'll switch to lower case X.

> 
> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Thank you!

-- 
Regards,

Sakari Ailus

Re: [PATCH v8 04/38] media: uapi: Document which mbus format fields are valid for metadata

[Sakari Ailus]

Hi Laurent,

On Wed, Mar 20, 2024 at 01:14:40AM +0200, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.
> 
> On Wed, Mar 13, 2024 at 09:24:42AM +0200, Sakari Ailus wrote:
> > Now that metadata mbus formats have been added, it is necessary to define
> > which fields in struct v4l2_mbus_format are applicable to them (not many).
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  include/uapi/linux/v4l2-mediabus.h | 18 ++++++++++++------
> >  1 file changed, 12 insertions(+), 6 deletions(-)
> > 
> > diff --git a/include/uapi/linux/v4l2-mediabus.h b/include/uapi/linux/v4l2-mediabus.h
> > index 6b07b73473b5..de1d6161bf62 100644
> > --- a/include/uapi/linux/v4l2-mediabus.h
> > +++ b/include/uapi/linux/v4l2-mediabus.h
> > @@ -19,12 +19,18 @@
> >   * @width:	image width
> >   * @height:	image height
> >   * @code:	data format code (from enum v4l2_mbus_pixelcode)
> > - * @field:	used interlacing type (from enum v4l2_field)
> > - * @colorspace:	colorspace of the data (from enum v4l2_colorspace)
> > - * @ycbcr_enc:	YCbCr encoding of the data (from enum v4l2_ycbcr_encoding)
> > - * @hsv_enc:	HSV encoding of the data (from enum v4l2_hsv_encoding)
> > - * @quantization: quantization of the data (from enum v4l2_quantization)
> > - * @xfer_func:  transfer function of the data (from enum v4l2_xfer_func)
> > + * @field:	used interlacing type (from enum v4l2_field), zero on metadata
> > + *		mbus codes
> 
> I would write "zero for metadata formats". Up to you.

The code field determines this, and elsewhere it's called mbus code. I
prefer thus "mbus code".

> 
> Shouldn't you also update
> Documentation/userspace-api/media/v4l/subdev-formats.rst ?

Good point. I'll do that for v9, with similar additions than here.

> 
> > + * @colorspace:	colorspace of the data (from enum v4l2_colorspace), zero on
> > + *		metadata mbus codes
> > + * @ycbcr_enc:	YCbCr encoding of the data (from enum v4l2_ycbcr_encoding), zero
> > + *		on metadata mbus codes
> > + * @hsv_enc:	HSV encoding of the data (from enum v4l2_hsv_encoding), zero on
> > + *		metadata mbus codes
> > + * @quantization: quantization of the data (from enum v4l2_quantization), zero
> > + *		on metadata mbus codes
> > + * @xfer_func:  transfer function of the data (from enum v4l2_xfer_func), zero
> > + *		on metadata mbus codes
> >   * @flags:	flags (V4L2_MBUS_FRAMEFMT_*)
> >   * @reserved:  reserved bytes that can be later used
> >   */

-- 
Kind regards,

Sakari Ailus

Re: [PATCH v8 19/38] media: ccs: Use {enable,disable}_streams operations

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Wed, Mar 13, 2024 at 09:24:57AM +0200, Sakari Ailus wrote:
> Switch from s_stream() video op to enable_streams() and disable_streams()
> pad operations. They are preferred and required for streams support.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  drivers/media/i2c/ccs/ccs-core.c | 187 +++++++++++++++----------------
>  1 file changed, 88 insertions(+), 99 deletions(-)
> 
> diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
> index 671540a8ab6a..9e70946653e9 100644
> --- a/drivers/media/i2c/ccs/ccs-core.c
> +++ b/drivers/media/i2c/ccs/ccs-core.c
> @@ -1715,22 +1715,66 @@ static int ccs_power_off(struct device *dev)
>  }
>  
>  /* -----------------------------------------------------------------------------
> - * Video stream management
> + * V4L2 subdev video operations
>   */
>  
> -static int ccs_start_streaming(struct ccs_sensor *sensor)
> +static int ccs_pm_get_init(struct ccs_sensor *sensor)

This patch would have been easier to review if you had moved the
function in a separate patch.

> +{
> +	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
> +	int rval;
> +
> +	/*
> +	 * It can't use pm_runtime_resume_and_get() here, as the driver
> +	 * relies at the returned value to detect if the device was already
> +	 * active or not.
> +	 */
> +	rval = pm_runtime_get_sync(&client->dev);
> +	if (rval < 0)
> +		goto error;
> +
> +	/* Device was already active, so don't set controls */
> +	if (rval == 1 && !sensor->handler_setup_needed)
> +		return 0;
> +
> +	sensor->handler_setup_needed = false;
> +
> +	/* Restore V4L2 controls to the previously suspended device */
> +	rval = __v4l2_ctrl_handler_setup(&sensor->pixel_array->ctrl_handler);
> +	if (rval)
> +		goto error;
> +
> +	rval = __v4l2_ctrl_handler_setup(&sensor->src->ctrl_handler);
> +	if (rval)
> +		goto error;
> +
> +	/* Keep PM runtime usage_count incremented on success */
> +	return 0;

Missing blank line.

> +error:
> +	pm_runtime_put(&client->dev);
> +	return rval;
> +}
> +
> +static int ccs_enable_streams(struct v4l2_subdev *subdev,
> +			      struct v4l2_subdev_state *state, u32 pad,
> +			      u64 streams_mask)
>  {
> +	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
>  	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
>  	unsigned int binning_mode;
>  	int rval;
>  
> -	mutex_lock(&sensor->mutex);
> +	if (pad != CCS_PAD_SRC)
> +		return -EINVAL;
> +
> +	rval = ccs_pm_get_init(sensor);
> +	if (rval)
> +		return rval;
>  
>  	rval = ccs_write(sensor, CSI_DATA_FORMAT,
>  			 (sensor->csi_format->width << 8) |
>  			 sensor->csi_format->compressed);
>  	if (rval)
> -		goto out;
> +		goto err_pm_put;

You could also have renamed the error label separately :-)

>  
>  	/* Binning configuration */
>  	if (sensor->binning_horizontal == 1 &&
> @@ -1743,38 +1787,38 @@ static int ccs_start_streaming(struct ccs_sensor *sensor)
>  
>  		rval = ccs_write(sensor, BINNING_TYPE, binning_type);
>  		if (rval < 0)
> -			goto out;
> +			goto err_pm_put;
>  
>  		binning_mode = 1;
>  	}
>  	rval = ccs_write(sensor, BINNING_MODE, binning_mode);
>  	if (rval < 0)
> -		goto out;
> +		goto err_pm_put;
>  
>  	/* Set up PLL */
>  	rval = ccs_pll_configure(sensor);
>  	if (rval)
> -		goto out;
> +		goto err_pm_put;
>  
>  	/* Analog crop start coordinates */
>  	rval = ccs_write(sensor, X_ADDR_START, sensor->pa_src.left);
>  	if (rval < 0)
> -		goto out;
> +		goto err_pm_put;
>  
>  	rval = ccs_write(sensor, Y_ADDR_START, sensor->pa_src.top);
>  	if (rval < 0)
> -		goto out;
> +		goto err_pm_put;
>  
>  	/* Analog crop end coordinates */
>  	rval = ccs_write(sensor, X_ADDR_END,
>  			 sensor->pa_src.left + sensor->pa_src.width - 1);
>  	if (rval < 0)
> -		goto out;
> +		goto err_pm_put;
>  
>  	rval = ccs_write(sensor, Y_ADDR_END,
>  			 sensor->pa_src.top + sensor->pa_src.height - 1);
>  	if (rval < 0)
> -		goto out;
> +		goto err_pm_put;
>  
>  	/*
>  	 * Output from pixel array, including blanking, is set using
> @@ -1787,22 +1831,22 @@ static int ccs_start_streaming(struct ccs_sensor *sensor)
>  		rval = ccs_write(sensor, DIGITAL_CROP_X_OFFSET,
>  				 sensor->scaler_sink.left);
>  		if (rval < 0)
> -			goto out;
> +			goto err_pm_put;
>  
>  		rval = ccs_write(sensor, DIGITAL_CROP_Y_OFFSET,
>  				 sensor->scaler_sink.top);
>  		if (rval < 0)
> -			goto out;
> +			goto err_pm_put;
>  
>  		rval = ccs_write(sensor, DIGITAL_CROP_IMAGE_WIDTH,
>  				 sensor->scaler_sink.width);
>  		if (rval < 0)
> -			goto out;
> +			goto err_pm_put;
>  
>  		rval = ccs_write(sensor, DIGITAL_CROP_IMAGE_HEIGHT,
>  				 sensor->scaler_sink.height);
>  		if (rval < 0)
> -			goto out;
> +			goto err_pm_put;
>  	}
>  
>  	/* Scaling */
> @@ -1810,20 +1854,20 @@ static int ccs_start_streaming(struct ccs_sensor *sensor)
>  	    != CCS_SCALING_CAPABILITY_NONE) {
>  		rval = ccs_write(sensor, SCALING_MODE, sensor->scaling_mode);
>  		if (rval < 0)
> -			goto out;
> +			goto err_pm_put;
>  
>  		rval = ccs_write(sensor, SCALE_M, sensor->scale_m);
>  		if (rval < 0)
> -			goto out;
> +			goto err_pm_put;
>  	}
>  
>  	/* Output size from sensor */
>  	rval = ccs_write(sensor, X_OUTPUT_SIZE, sensor->src_src.width);
>  	if (rval < 0)
> -		goto out;
> +		goto err_pm_put;
>  	rval = ccs_write(sensor, Y_OUTPUT_SIZE, sensor->src_src.height);
>  	if (rval < 0)
> -		goto out;
> +		goto err_pm_put;
>  
>  	if (CCS_LIM(sensor, FLASH_MODE_CAPABILITY) &
>  	    (CCS_FLASH_MODE_CAPABILITY_SINGLE_STROBE |
> @@ -1832,111 +1876,52 @@ static int ccs_start_streaming(struct ccs_sensor *sensor)
>  	    sensor->hwcfg.strobe_setup->trigger != 0) {
>  		rval = ccs_setup_flash_strobe(sensor);
>  		if (rval)
> -			goto out;
> +			goto err_pm_put;
>  	}
>  
>  	rval = ccs_call_quirk(sensor, pre_streamon);
>  	if (rval) {
>  		dev_err(&client->dev, "pre_streamon quirks failed\n");
> -		goto out;
> +		goto err_pm_put;
>  	}
>  
>  	rval = ccs_write(sensor, MODE_SELECT, CCS_MODE_SELECT_STREAMING);
>  
> -out:
> -	mutex_unlock(&sensor->mutex);
> -
> -	return rval;
> -}
> -
> -static int ccs_stop_streaming(struct ccs_sensor *sensor)
> -{
> -	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
> -	int rval;
> -
> -	mutex_lock(&sensor->mutex);
> -	rval = ccs_write(sensor, MODE_SELECT, CCS_MODE_SELECT_SOFTWARE_STANDBY);
> -	if (rval)
> -		goto out;
> -
> -	rval = ccs_call_quirk(sensor, post_streamoff);
> -	if (rval)
> -		dev_err(&client->dev, "post_streamoff quirks failed\n");
> -
> -out:
> -	mutex_unlock(&sensor->mutex);
> -	return rval;
> -}
> -
> -/* -----------------------------------------------------------------------------
> - * V4L2 subdev video operations
> - */
> -
> -static int ccs_pm_get_init(struct ccs_sensor *sensor)
> -{
> -	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
> -	int rval;
> -
> -	/*
> -	 * It can't use pm_runtime_resume_and_get() here, as the driver
> -	 * relies at the returned value to detect if the device was already
> -	 * active or not.
> -	 */
> -	rval = pm_runtime_get_sync(&client->dev);
> -	if (rval < 0)
> -		goto error;
> -
> -	/* Device was already active, so don't set controls */
> -	if (rval == 1 && !sensor->handler_setup_needed)
> -		return 0;
> -
> -	sensor->handler_setup_needed = false;
> +	sensor->streaming = true;
>  
> -	/* Restore V4L2 controls to the previously suspended device */
> -	rval = v4l2_ctrl_handler_setup(&sensor->pixel_array->ctrl_handler);
> -	if (rval)
> -		goto error;
> +	return 0;
>  
> -	rval = v4l2_ctrl_handler_setup(&sensor->src->ctrl_handler);
> -	if (rval)
> -		goto error;
> +err_pm_put:
> +	pm_runtime_mark_last_busy(&client->dev);
> +	pm_runtime_put_autosuspend(&client->dev);

On failures I tend not to use autosuspend, based on the idea that it may
be easier to recover from a failure if the sensor got powered off. I'm
not sure that's really relevant though.

Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

>  
> -	/* Keep PM runtime usage_count incremented on success */
> -	return 0;
> -error:
> -	pm_runtime_put(&client->dev);
>  	return rval;
>  }
>  
> -static int ccs_set_stream(struct v4l2_subdev *subdev, int enable)
> +static int ccs_disable_streams(struct v4l2_subdev *subdev,
> +			       struct v4l2_subdev_state *state, u32 pad,
> +			       u64 streams_mask)
>  {
>  	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
>  	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
>  	int rval;
>  
> -	if (!enable) {
> -		ccs_stop_streaming(sensor);
> -		sensor->streaming = false;
> -		pm_runtime_mark_last_busy(&client->dev);
> -		pm_runtime_put_autosuspend(&client->dev);
> -
> -		return 0;
> -	}
> +	if (pad != CCS_PAD_SRC)
> +		return -EINVAL;
>  
> -	rval = ccs_pm_get_init(sensor);
> +	rval = ccs_write(sensor, MODE_SELECT, CCS_MODE_SELECT_SOFTWARE_STANDBY);
>  	if (rval)
>  		return rval;
>  
> -	sensor->streaming = true;
> +	rval = ccs_call_quirk(sensor, post_streamoff);
> +	if (rval)
> +		dev_err(&client->dev, "post_streamoff quirks failed\n");
>  
> -	rval = ccs_start_streaming(sensor);
> -	if (rval < 0) {
> -		sensor->streaming = false;
> -		pm_runtime_mark_last_busy(&client->dev);
> -		pm_runtime_put_autosuspend(&client->dev);
> -	}
> +	sensor->streaming = false;
> +	pm_runtime_mark_last_busy(&client->dev);
> +	pm_runtime_put_autosuspend(&client->dev);
>  
> -	return rval;
> +	return 0;
>  }
>  
>  static int ccs_pre_streamon(struct v4l2_subdev *subdev, u32 flags)
> @@ -1962,7 +1947,9 @@ static int ccs_pre_streamon(struct v4l2_subdev *subdev, u32 flags)
>  		}
>  	}
>  
> +	mutex_lock(&sensor->mutex);
>  	rval = ccs_pm_get_init(sensor);
> +	mutex_unlock(&sensor->mutex);
>  	if (rval)
>  		return rval;
>  
> @@ -3046,7 +3033,7 @@ static int ccs_init_state(struct v4l2_subdev *sd,
>  }
>  
>  static const struct v4l2_subdev_video_ops ccs_video_ops = {
> -	.s_stream = ccs_set_stream,
> +	.s_stream = v4l2_subdev_s_stream_helper,
>  	.pre_streamon = ccs_pre_streamon,
>  	.post_streamoff = ccs_post_streamoff,
>  };
> @@ -3057,6 +3044,8 @@ static const struct v4l2_subdev_pad_ops ccs_pad_ops = {
>  	.set_fmt = ccs_set_format,
>  	.get_selection = ccs_get_selection,
>  	.set_selection = ccs_set_selection,
> +	.enable_streams = ccs_enable_streams,
> +	.disable_streams = ccs_disable_streams,
>  };
>  
>  static const struct v4l2_subdev_sensor_ops ccs_sensor_ops = {

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 18/38] media: ccs: No need to set streaming to false in power off

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Wed, Mar 13, 2024 at 09:24:56AM +0200, Sakari Ailus wrote:
> Streaming will have been stopped by the sensor is powered off, and so
> sensor->streaming is also false already. Do not set it as part of the
> runtime suspend callback.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>

Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

> ---
>  drivers/media/i2c/ccs/ccs-core.c | 1 -
>  1 file changed, 1 deletion(-)
> 
> diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
> index e21287d50c15..671540a8ab6a 100644
> --- a/drivers/media/i2c/ccs/ccs-core.c
> +++ b/drivers/media/i2c/ccs/ccs-core.c
> @@ -1710,7 +1710,6 @@ static int ccs_power_off(struct device *dev)
>  	usleep_range(5000, 5000);
>  	regulator_bulk_disable(ARRAY_SIZE(ccs_regulators),
>  			       sensor->regulators);
> -	sensor->streaming = false;
>  
>  	return 0;
>  }

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 20/38] media: ccs: Track streaming state

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Wed, Mar 13, 2024 at 09:24:58AM +0200, Sakari Ailus wrote:
> With enable_streams and disable_streams, the driver for a device where
> streams are not independently started and stopped needs to maintain state
> information on streams that have been requested to be started. Do that
> now.
> 
> In the future, a helper function in the framework is a desirable way to do
> this instead.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>

Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

> ---
>  drivers/media/i2c/ccs/ccs-core.c | 13 ++++++++++---
>  drivers/media/i2c/ccs/ccs.h      |  2 +-
>  2 files changed, 11 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
> index 9e70946653e9..0cd8ee957655 100644
> --- a/drivers/media/i2c/ccs/ccs-core.c
> +++ b/drivers/media/i2c/ccs/ccs-core.c
> @@ -1766,6 +1766,11 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
>  	if (pad != CCS_PAD_SRC)
>  		return -EINVAL;
>  
> +	if (sensor->streaming) {
> +		sensor->streaming |= streams_mask;
> +		return 0;
> +	}
> +
>  	rval = ccs_pm_get_init(sensor);
>  	if (rval)
>  		return rval;
> @@ -1887,7 +1892,7 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
>  
>  	rval = ccs_write(sensor, MODE_SELECT, CCS_MODE_SELECT_STREAMING);
>  
> -	sensor->streaming = true;
> +	sensor->streaming |= streams_mask;
>  
>  	return 0;
>  
> @@ -1909,6 +1914,10 @@ static int ccs_disable_streams(struct v4l2_subdev *subdev,
>  	if (pad != CCS_PAD_SRC)
>  		return -EINVAL;
>  
> +	sensor->streaming &= ~streams_mask;
> +	if (sensor->streaming)
> +		return 0;
> +
>  	rval = ccs_write(sensor, MODE_SELECT, CCS_MODE_SELECT_SOFTWARE_STANDBY);
>  	if (rval)
>  		return rval;
> @@ -1917,7 +1926,6 @@ static int ccs_disable_streams(struct v4l2_subdev *subdev,
>  	if (rval)
>  		dev_err(&client->dev, "post_streamoff quirks failed\n");
>  
> -	sensor->streaming = false;
>  	pm_runtime_mark_last_busy(&client->dev);
>  	pm_runtime_put_autosuspend(&client->dev);
>  
> @@ -3525,7 +3533,6 @@ static int ccs_probe(struct i2c_client *client)
>  		goto out_cleanup;
>  	}
>  
> -	sensor->streaming = false;
>  	sensor->dev_init_done = true;
>  	sensor->handler_setup_needed = true;
>  
> diff --git a/drivers/media/i2c/ccs/ccs.h b/drivers/media/i2c/ccs/ccs.h
> index 096573845a10..4725e6eca8d0 100644
> --- a/drivers/media/i2c/ccs/ccs.h
> +++ b/drivers/media/i2c/ccs/ccs.h
> @@ -236,7 +236,7 @@ struct ccs_sensor {
>  	u16 image_start; /* image data start line */
>  	u16 visible_pixel_start; /* start pixel of the visible image */
>  
> -	bool streaming;
> +	u8 streaming;
>  	bool dev_init_done;
>  	bool handler_setup_needed;
>  	u8 compressed_min_bpp;

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 21/38] media: ccs: Move ccs_validate_csi_data_format up

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Wed, Mar 13, 2024 at 09:24:59AM +0200, Sakari Ailus wrote:
> ccs_validate_csi_data_format() will soon be needed elsewhere, above its
> current location. Move it up.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>

Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

> ---
>  drivers/media/i2c/ccs/ccs-core.c | 28 ++++++++++++++--------------
>  1 file changed, 14 insertions(+), 14 deletions(-)
> 
> diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
> index 0cd8ee957655..0efbc63534bc 100644
> --- a/drivers/media/i2c/ccs/ccs-core.c
> +++ b/drivers/media/i2c/ccs/ccs-core.c
> @@ -1979,6 +1979,20 @@ static int ccs_post_streamoff(struct v4l2_subdev *subdev)
>  	return pm_runtime_put(&client->dev);
>  }
>  
> +static const struct ccs_csi_data_format
> +*ccs_validate_csi_data_format(struct ccs_sensor *sensor, u32 code)
> +{
> +	unsigned int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(ccs_csi_data_formats); i++) {
> +		if (sensor->mbus_frame_fmts & (1 << i) &&
> +		    ccs_csi_data_formats[i].code == code)
> +			return &ccs_csi_data_formats[i];
> +	}
> +
> +	return sensor->csi_format;
> +}
> +
>  static int ccs_enum_mbus_code(struct v4l2_subdev *subdev,
>  			      struct v4l2_subdev_state *sd_state,
>  			      struct v4l2_subdev_mbus_code_enum *code)
> @@ -2114,20 +2128,6 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
>  	}
>  }
>  
> -static const struct ccs_csi_data_format
> -*ccs_validate_csi_data_format(struct ccs_sensor *sensor, u32 code)
> -{
> -	unsigned int i;
> -
> -	for (i = 0; i < ARRAY_SIZE(ccs_csi_data_formats); i++) {
> -		if (sensor->mbus_frame_fmts & (1 << i) &&
> -		    ccs_csi_data_formats[i].code == code)
> -			return &ccs_csi_data_formats[i];
> -	}
> -
> -	return sensor->csi_format;
> -}
> -
>  static int ccs_set_format_source(struct v4l2_subdev *subdev,
>  				 struct v4l2_subdev_state *sd_state,
>  				 struct v4l2_subdev_format *fmt)

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 22/38] media: ccs: Support frame descriptors

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Wed, Mar 13, 2024 at 09:25:00AM +0200, Sakari Ailus wrote:
> Provide information on the frame layout using frame descriptors.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  drivers/media/i2c/ccs/ccs-core.c  | 60 +++++++++++++++++++++++++++++++
>  drivers/media/i2c/ccs/ccs-quirk.h |  7 ++++
>  drivers/media/i2c/ccs/ccs.h       |  4 +++
>  3 files changed, 71 insertions(+)
> 
> diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
> index 0efbc63534bc..9cc2080b73ec 100644
> --- a/drivers/media/i2c/ccs/ccs-core.c
> +++ b/drivers/media/i2c/ccs/ccs-core.c
> @@ -25,6 +25,7 @@
>  #include <linux/slab.h>
>  #include <linux/smiapp.h>
>  #include <linux/v4l2-mediabus.h>
> +#include <media/mipi-csi2.h>
>  #include <media/v4l2-cci.h>
>  #include <media/v4l2-device.h>
>  #include <media/v4l2-fwnode.h>
> @@ -245,6 +246,33 @@ static int ccs_read_all_limits(struct ccs_sensor *sensor)
>  	return ret;
>  }
>  
> +static u8 ccs_mipi_csi2_data_type(unsigned int bpp)
> +{
> +	switch (bpp) {
> +	case 6:
> +		return MIPI_CSI2_DT_RAW6;
> +	case 7:
> +		return MIPI_CSI2_DT_RAW7;
> +	case 8:
> +		return MIPI_CSI2_DT_RAW8;
> +	case 10:
> +		return MIPI_CSI2_DT_RAW10;
> +	case 12:
> +		return MIPI_CSI2_DT_RAW12;
> +	case 14:
> +		return MIPI_CSI2_DT_RAW14;
> +	case 16:
> +		return MIPI_CSI2_DT_RAW16;
> +	case 20:
> +		return MIPI_CSI2_DT_RAW20;
> +	case 24:
> +		return MIPI_CSI2_DT_RAW24;
> +	default:
> +		WARN_ON(1);
> +		return 0;
> +	}
> +}
> +
>  static int ccs_read_frame_fmt(struct ccs_sensor *sensor)
>  {
>  	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
> @@ -2632,6 +2660,37 @@ static int ccs_set_selection(struct v4l2_subdev *subdev,
>  	return ret;
>  }
>  
> +static int ccs_get_frame_desc(struct v4l2_subdev *subdev, unsigned int pad,
> +				 struct v4l2_mbus_frame_desc *desc)
> +{
> +	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
> +	struct v4l2_mbus_frame_desc_entry *entry = desc->entry;
> +
> +	if (ccs_has_quirk(sensor, frame_desc))
> +		return ccs_call_quirk(sensor, frame_desc, desc);

I would introduce the quirk later, along with the patch that will use
it.

> +
> +	switch (sensor->hwcfg.csi_signalling_mode) {
> +	case CCS_CSI_SIGNALING_MODE_CSI_2_DPHY:
> +	case CCS_CSI_SIGNALING_MODE_CSI_2_CPHY:
> +		desc->type = V4L2_MBUS_FRAME_DESC_TYPE_CSI2;
> +		break;
> +	default:
> +		/* FIXME: CCP2 support */
> +		return -EINVAL;
> +	}
> +
> +	entry->pixelcode = sensor->csi_format->code;
> +	entry->stream = CCS_STREAM_PIXEL;
> +	entry->bus.csi2.dt =
> +		sensor->csi_format->width == sensor->csi_format->compressed ?
> +		ccs_mipi_csi2_data_type(sensor->csi_format->compressed) :

Functionally equivalent,

		ccs_mipi_csi2_data_type(sensor->csi_format->width) :

would be clearer I think. The way it's written today made me wonder why
you want the DT for the compressed format, which is not what you're
doing.

> +		CCS_DEFAULT_COMPRESSED_DT;
> +	entry++;
> +	desc->num_entries++;
> +
> +	return 0;
> +}
> +
>  static int ccs_get_skip_frames(struct v4l2_subdev *subdev, u32 *frames)
>  {
>  	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
> @@ -3054,6 +3113,7 @@ static const struct v4l2_subdev_pad_ops ccs_pad_ops = {
>  	.set_selection = ccs_set_selection,
>  	.enable_streams = ccs_enable_streams,
>  	.disable_streams = ccs_disable_streams,
> +	.get_frame_desc = ccs_get_frame_desc,
>  };
>  
>  static const struct v4l2_subdev_sensor_ops ccs_sensor_ops = {
> diff --git a/drivers/media/i2c/ccs/ccs-quirk.h b/drivers/media/i2c/ccs/ccs-quirk.h
> index 392c97109617..3e1d9eaa33fa 100644
> --- a/drivers/media/i2c/ccs/ccs-quirk.h
> +++ b/drivers/media/i2c/ccs/ccs-quirk.h
> @@ -36,6 +36,7 @@ struct ccs_sensor;
>   *			 access may be done by the caller (default read
>   *			 value is zero), else negative error code on error
>   * @flags: Quirk flags
> + * @frame_desc: Obtain the frame descriptor
>   */
>  struct ccs_quirk {
>  	int (*limits)(struct ccs_sensor *sensor);
> @@ -46,6 +47,8 @@ struct ccs_quirk {
>  	int (*init)(struct ccs_sensor *sensor);
>  	int (*reg_access)(struct ccs_sensor *sensor, bool write, u32 *reg,
>  			  u32 *val);
> +	int (*frame_desc)(struct ccs_sensor *sensor,
> +			  struct v4l2_mbus_frame_desc *desc);
>  	unsigned long flags;
>  };
>  
> @@ -62,6 +65,10 @@ struct ccs_reg_8 {
>  		.val = _val,		\
>  	}
>  
> +#define ccs_has_quirk(sensor, _quirk)					\
> +	((sensor)->minfo.quirk &&					\
> +	 (sensor)->minfo.quirk->_quirk)
> +
>  #define ccs_call_quirk(sensor, _quirk, ...)				\
>  	((sensor)->minfo.quirk &&					\
>  	 (sensor)->minfo.quirk->_quirk ?				\
> diff --git a/drivers/media/i2c/ccs/ccs.h b/drivers/media/i2c/ccs/ccs.h
> index 4725e6eca8d0..adb152366ea2 100644
> --- a/drivers/media/i2c/ccs/ccs.h
> +++ b/drivers/media/i2c/ccs/ccs.h
> @@ -46,6 +46,8 @@
>  
>  #define CCS_COLOUR_COMPONENTS		4
>  
> +#define CCS_DEFAULT_COMPRESSED_DT	0x30

I'd write

#define CCS_DEFAULT_COMPRESSED_DT	MIPI_CSI2_DT_USER_DEFINED(0)

> +
>  #define SMIAPP_NAME			"smiapp"
>  #define CCS_NAME			"ccs"
>  
> @@ -175,6 +177,8 @@ struct ccs_csi_data_format {
>  #define CCS_PAD_SRC			1
>  #define CCS_PADS			2
>  
> +#define CCS_STREAM_PIXEL		0
> +
>  struct ccs_binning_subtype {
>  	u8 horizontal:4;
>  	u8 vertical:4;

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 23/38] media: uapi: ccs: Add media bus code for MIPI CCS embedded data

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Wed, Mar 13, 2024 at 09:25:01AM +0200, Sakari Ailus wrote:
> Add new MIPI CCS embedded data media bus code
> (MEDIA_BUS_FMT_CCS_EMBEDDED).
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../media/v4l/subdev-formats.rst              | 28 +++++++++++++++++++
>  include/uapi/linux/media-bus-format.h         |  3 ++
>  2 files changed, 31 insertions(+)
> 
> diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> index cbd475f7cae9..c8f982411e70 100644
> --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
> +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> @@ -8564,3 +8564,31 @@ and finally the bit number in subscript. "X" indicates a padding bit.
>        - X
>        - X
>        - X
> +
> +.. _MEDIA-BUS-FMT-CCS-EMBEDDED:
> +
> +MIPI CCS Embedded Data Formats
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +`MIPI CCS <https://www.mipi.org/specifications/camera-command-set>`_ defines a
> +metadata format for sensor embedded data, which is used to store the register
> +configuration used for capturing a given frame. The format is defined in the CCS
> +specification. The media bus code for this format is
> +``MEDIA_BUS_FMT_CCS_EMBEDDED``.
> +
> +The CCS embedded data format definition includes three levels:
> +
> +1. Padding within CSI-2 bus :ref:`Data unit <media-glossary-data-unit>` as

s/Data unit/Data Unit/

> +   documented in the MIPI CCS specification.
> +
> +2. The tagged data format as documented in the MIPI CCS specification.
> +
> +3. Register addresses and register documentation as documented in the MIPI CCS
> +   specification.
> +
> +The format definition shall be used only by devices that fulfill all three
> +levels above.

Hmmmm... Do we need to mandate level 3 ? There are lots of sensors that
comply with the first two levels but have their own register set. Would
you like a sensor-specific embedded data format for each of them ? If
so, how would we document it without essentially copying the datasheet ?

> +
> +This mbus code are only used for "2-byte simplified tagged data format" (code
> +0xa) but their use may be extended further in the future, to cover other CCS
> +embedded data format codes.
> diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
> index d4c1d991014b..03f7e9ab517b 100644
> --- a/include/uapi/linux/media-bus-format.h
> +++ b/include/uapi/linux/media-bus-format.h
> @@ -183,4 +183,7 @@
>  #define MEDIA_BUS_FMT_META_20			0x8006
>  #define MEDIA_BUS_FMT_META_24			0x8007
>  
> +/* Specific metadata formats. Next is 0x9002. */
> +#define MEDIA_BUS_FMT_CCS_EMBEDDED		0x9001
> +
>  #endif /* __LINUX_MEDIA_BUS_FORMAT_H */

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 31/38] media: uapi: v4l: subdev: Enable streams API

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Wed, Mar 13, 2024 at 09:25:09AM +0200, Sakari Ailus wrote:
> Remove v4l2_subdev_enable_streams_api variable that was used to easily
> enable streams API for development, and conditions that use the variable.
> 
> This patch enables the streams API for V4L2 sub-device interface which
> allows transporting multiple streams on a single MC link.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>

The patch itself looks good, but I'll only send my Rb when I'll be
confident the API is stable :-)

> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 30 ---------------------------
>  1 file changed, 30 deletions(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index c8c435df92c8..b3a697df4b66 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -26,15 +26,6 @@
>  #include <media/v4l2-fh.h>
>  #include <media/v4l2-ioctl.h>
>  
> -#if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
> -/*
> - * The Streams API is an experimental feature. To use the Streams API, set
> - * 'v4l2_subdev_enable_streams_api' to 1 below.
> - */
> -
> -static bool v4l2_subdev_enable_streams_api;
> -#endif
> -
>  /*
>   * Maximum stream ID is 63 for now, as we use u64 bitmask to represent a set
>   * of streams.
> @@ -570,13 +561,6 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>  				       V4L2_SUBDEV_CLIENT_CAP_STREAMS;
>  	int rval;
>  
> -	/*
> -	 * If the streams API is not enabled, remove V4L2_SUBDEV_CAP_STREAMS.
> -	 * Remove this when the API is no longer experimental.
> -	 */
> -	if (!v4l2_subdev_enable_streams_api)
> -		streams_subdev = false;
> -
>  	switch (cmd) {
>  	case VIDIOC_SUBDEV_QUERYCAP: {
>  		struct v4l2_subdev_capability *cap = arg;
> @@ -916,9 +900,6 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>  		struct v4l2_subdev_krouting krouting = {};
>  		unsigned int i;
>  
> -		if (!v4l2_subdev_enable_streams_api)
> -			return -ENOIOCTLCMD;
> -
>  		if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS))
>  			return -ENOIOCTLCMD;
>  
> @@ -989,9 +970,6 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>  		struct v4l2_subdev_routing *routing = arg;
>  		struct v4l2_subdev_krouting *krouting;
>  
> -		if (!v4l2_subdev_enable_streams_api)
> -			return -ENOIOCTLCMD;
> -
>  		if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS))
>  			return -ENOIOCTLCMD;
>  
> @@ -1019,14 +997,6 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>  	case VIDIOC_SUBDEV_S_CLIENT_CAP: {
>  		struct v4l2_subdev_client_capability *client_cap = arg;
>  
> -		/*
> -		 * Clear V4L2_SUBDEV_CLIENT_CAP_STREAMS if streams API is not
> -		 * enabled. Remove this when streams API is no longer
> -		 * experimental.
> -		 */
> -		if (!v4l2_subdev_enable_streams_api)
> -			client_cap->capabilities &= ~V4L2_SUBDEV_CLIENT_CAP_STREAMS;
> -
>  		/* Filter out unsupported capabilities */
>  		client_cap->capabilities &= (V4L2_SUBDEV_CLIENT_CAP_STREAMS |
>  					     V4L2_SUBDEV_CLIENT_CAP_INTERVAL_USES_WHICH);

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 32/38] media: uapi: Add media bus code for ov2740 embedded data

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Wed, Mar 13, 2024 at 09:25:10AM +0200, Sakari Ailus wrote:
> Add a media bus code for ov2740 camera sensor embedded data and document
> it.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../media/v4l/subdev-formats.rst              | 66 +++++++++++++++++++
>  include/uapi/linux/media-bus-format.h         |  3 +-
>  2 files changed, 68 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> index ca4da6a400ff..a875868ed951 100644
> --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
> +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> @@ -8594,3 +8594,69 @@ This mbus code are only used for "2-byte simplified tagged data format" (code
>  embedded data format codes.
>  
>  Also see :ref:`CCS driver documentation <media-ccs-routes>`.
> +
> +Omnivision OV2740 Embedded Data Format
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +The Omnivision OV2740 camera sensor produces the following embedded data format.
> +
> +.. flat-table:: Omnivision OV2740 Embedded Data Format. Octets at indices marked
> +                reserved or unused have been omitted from the table.
> +    :header-rows: 1
> +
> +    * - Byte
> +      - Concent description

s/Concent/Content/

> +    * - 0
> +      - Sensor info

What does this contain ?

> +    * - 4
> +      - Analogue gain (bits 10--8)
> +    * - 5
> +      - Analogue gain (bits 7--0)

I wonder if you could simplify the table by writing

    * - 4
      - Analogue gain (bits 10--0)
    * - 6
      - Coarse integration time (bits 15--0)

...

and documenting that all fields are in big-endian format. A size columns
could then be useful I suppose.

> +    * - 6
> +      - Coarse integration time (bits 15--8)
> +    * - 7
> +      - Coarse integration time (bits 7--0)
> +    * - 10
> +      - Dpc correction threshold (bits 9--2)
> +    * - 15
> +      - Output image width (bits 15--8)
> +    * - 16
> +      - Output image width (bits 7--0)
> +    * - 17
> +      - Output image height (bits 15--8)
> +    * - 18
> +      - Output image height (bits 7--0)
> +    * - 23
> +      - MIPI header revision number

What does this contain ?

> +    * - 31
> +      - Vertical (bit 1) and horizontal flip (bit 0)
> +    * - 32
> +      - Frame duration A

What does this contain ? Same for frame duration B, context count,
context select.

> +    * - 33
> +      - Frame duration B
> +    * - 34
> +      - Context count
> +    * - 35
> +      - Context select
> +    * - 54
> +      - Data pedestal (bits 9--2)
> +    * - 63
> +      - Frame average (bits 9--2)
> +    * - 64
> +      - Digital gain red
> +    * - 65
> +      - Digital gain red

Bit numbers are missing.

> +    * - 66
> +      - Digital gain greenr
> +    * - 67
> +      - Digital gain greenr
> +    * - 68
> +      - Digital gain blue
> +    * - 69
> +      - Digital gain blue
> +    * - 70
> +      - Digital gain greenb
> +    * - 71
> +      - Digital gain greenb
> +    * - 89
> +      - Frame counter

Does this wrap at 255 ?

> diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
> index 03f7e9ab517b..13e68c2ccb61 100644
> --- a/include/uapi/linux/media-bus-format.h
> +++ b/include/uapi/linux/media-bus-format.h
> @@ -183,7 +183,8 @@
>  #define MEDIA_BUS_FMT_META_20			0x8006
>  #define MEDIA_BUS_FMT_META_24			0x8007
>  
> -/* Specific metadata formats. Next is 0x9002. */
> +/* Specific metadata formats. Next is 0x9003. */
>  #define MEDIA_BUS_FMT_CCS_EMBEDDED		0x9001
> +#define MEDIA_BUS_FMT_OV2740_EMBEDDED		0x9002
>  
>  #endif /* __LINUX_MEDIA_BUS_FORMAT_H */

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 33/38] media: ov2740: Switch to {enable,disable}_streams

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Wed, Mar 13, 2024 at 09:25:11AM +0200, Sakari Ailus wrote:
> Switch from s_stream to enable_streams and disable_streams callbacks.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>

Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

> ---
>  drivers/media/i2c/ov2740.c | 72 +++++++++++++++++---------------------
>  1 file changed, 32 insertions(+), 40 deletions(-)
> 
> diff --git a/drivers/media/i2c/ov2740.c b/drivers/media/i2c/ov2740.c
> index 552935ccb4a9..44c6724a102c 100644
> --- a/drivers/media/i2c/ov2740.c
> +++ b/drivers/media/i2c/ov2740.c
> @@ -919,16 +919,23 @@ static int ov2740_load_otp_data(struct nvm_data *nvm)
>  	return ret;
>  }
>  
> -static int ov2740_start_streaming(struct ov2740 *ov2740)
> +static int ov2740_enable_streams(struct v4l2_subdev *sd,
> +				 struct v4l2_subdev_state *state, u32 pad,
> +				 u64 streams_mask)
>  {
> -	struct i2c_client *client = v4l2_get_subdevdata(&ov2740->sd);
> +	struct i2c_client *client = v4l2_get_subdevdata(sd);
> +	struct ov2740 *ov2740 = to_ov2740(sd);
>  	const struct ov2740_reg_list *reg_list;
>  	int link_freq_index;
>  	int ret;
>  
> +	ret = pm_runtime_resume_and_get(&client->dev);
> +	if (ret < 0)
> +		return ret;
> +
>  	ret = ov2740_identify_module(ov2740);
>  	if (ret)
> -		return ret;
> +		goto out_pm_put;
>  
>  	if (ov2740->nvm)
>  		ov2740_load_otp_data(ov2740->nvm);
> @@ -937,7 +944,7 @@ static int ov2740_start_streaming(struct ov2740 *ov2740)
>  	ret = ov2740_write_reg(ov2740, 0x0103, 1, 0x01);
>  	if (ret) {
>  		dev_err(&client->dev, "failed to reset\n");
> -		return ret;
> +		goto out_pm_put;
>  	}
>  
>  	usleep_range(10000, 15000);
> @@ -947,64 +954,47 @@ static int ov2740_start_streaming(struct ov2740 *ov2740)
>  	ret = ov2740_write_reg_list(ov2740, reg_list);
>  	if (ret) {
>  		dev_err(&client->dev, "failed to set plls\n");
> -		return ret;
> +		goto out_pm_put;
>  	}
>  
>  	reg_list = &ov2740->cur_mode->reg_list;
>  	ret = ov2740_write_reg_list(ov2740, reg_list);
>  	if (ret) {
>  		dev_err(&client->dev, "failed to set mode\n");
> -		return ret;
> +		goto out_pm_put;
>  	}
>  
>  	ret = __v4l2_ctrl_handler_setup(ov2740->sd.ctrl_handler);
>  	if (ret)
> -		return ret;
> +		goto out_pm_put;
>  
>  	ret = ov2740_write_reg(ov2740, OV2740_REG_MODE_SELECT, 1,
>  			       OV2740_MODE_STREAMING);
> -	if (ret)
> +	if (ret) {
>  		dev_err(&client->dev, "failed to start streaming\n");
> +		goto out_pm_put;
> +	}
>  
> -	return ret;
> -}
> +	return 0;
>  
> -static void ov2740_stop_streaming(struct ov2740 *ov2740)
> -{
> -	struct i2c_client *client = v4l2_get_subdevdata(&ov2740->sd);
> +out_pm_put:
> +	pm_runtime_put(&client->dev);
>  
> -	if (ov2740_write_reg(ov2740, OV2740_REG_MODE_SELECT, 1,
> -			     OV2740_MODE_STANDBY))
> -		dev_err(&client->dev, "failed to stop streaming\n");
> +	return ret;
>  }
>  
> -static int ov2740_set_stream(struct v4l2_subdev *sd, int enable)
> +static int ov2740_disable_streams(struct v4l2_subdev *sd,
> +				  struct v4l2_subdev_state *state, u32 pad,
> +				  u64 streams_mask)
>  {
> -	struct ov2740 *ov2740 = to_ov2740(sd);
>  	struct i2c_client *client = v4l2_get_subdevdata(sd);
> -	struct v4l2_subdev_state *sd_state;
> -	int ret = 0;
> -
> -	sd_state = v4l2_subdev_lock_and_get_active_state(&ov2740->sd);
> -
> -	if (enable) {
> -		ret = pm_runtime_resume_and_get(&client->dev);
> -		if (ret < 0)
> -			goto out_unlock;
> +	struct ov2740 *ov2740 = to_ov2740(sd);
> +	int ret;
>  
> -		ret = ov2740_start_streaming(ov2740);
> -		if (ret) {
> -			enable = 0;
> -			ov2740_stop_streaming(ov2740);
> -			pm_runtime_put(&client->dev);
> -		}
> -	} else {
> -		ov2740_stop_streaming(ov2740);
> -		pm_runtime_put(&client->dev);
> -	}
> +	ret = ov2740_write_reg(ov2740, OV2740_REG_MODE_SELECT, 1,
> +			       OV2740_MODE_STANDBY);
>  
> -out_unlock:
> -	v4l2_subdev_unlock_state(sd_state);
> +	pm_runtime_put(&client->dev);
>  
>  	return ret;
>  }
> @@ -1089,7 +1079,7 @@ static int ov2740_init_state(struct v4l2_subdev *sd,
>  }
>  
>  static const struct v4l2_subdev_video_ops ov2740_video_ops = {
> -	.s_stream = ov2740_set_stream,
> +	.s_stream = v4l2_subdev_s_stream_helper,
>  };
>  
>  static const struct v4l2_subdev_pad_ops ov2740_pad_ops = {
> @@ -1097,6 +1087,8 @@ static const struct v4l2_subdev_pad_ops ov2740_pad_ops = {
>  	.set_fmt = ov2740_set_format,
>  	.enum_mbus_code = ov2740_enum_mbus_code,
>  	.enum_frame_size = ov2740_enum_frame_size,
> +	.enable_streams = ov2740_enable_streams,
> +	.disable_streams = ov2740_disable_streams,
>  };
>  
>  static const struct v4l2_subdev_ops ov2740_subdev_ops = {

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 34/38] media: ov2740: Track streaming state

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Wed, Mar 13, 2024 at 09:25:12AM +0200, Sakari Ailus wrote:
> With enable_streams and disable_streams, the driver for a device where
> streams are not independently started and stopped needs to maintain state
> information on streams that have been requested to be started. Do that
> now.
> 
> In the future, a helper function in the framework is a desirable way to do
> this instead.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>

Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

> ---
>  drivers/media/i2c/ov2740.c | 14 ++++++++++++++
>  1 file changed, 14 insertions(+)
> 
> diff --git a/drivers/media/i2c/ov2740.c b/drivers/media/i2c/ov2740.c
> index 44c6724a102c..df57f0096e98 100644
> --- a/drivers/media/i2c/ov2740.c
> +++ b/drivers/media/i2c/ov2740.c
> @@ -539,6 +539,9 @@ struct ov2740 {
>  
>  	/* True if the device has been identified */
>  	bool identified;
> +
> +	/* Track streaming state */
> +	u8 streaming;
>  };
>  
>  static inline struct ov2740 *to_ov2740(struct v4l2_subdev *subdev)
> @@ -929,6 +932,11 @@ static int ov2740_enable_streams(struct v4l2_subdev *sd,
>  	int link_freq_index;
>  	int ret;
>  
> +	if (ov2740->streaming) {
> +		ov2740->streaming |= streams_mask;
> +		return 0;
> +	}
> +
>  	ret = pm_runtime_resume_and_get(&client->dev);
>  	if (ret < 0)
>  		return ret;
> @@ -975,6 +983,8 @@ static int ov2740_enable_streams(struct v4l2_subdev *sd,
>  		goto out_pm_put;
>  	}
>  
> +	ov2740->streaming |= streams_mask;
> +
>  	return 0;
>  
>  out_pm_put:
> @@ -991,6 +1001,10 @@ static int ov2740_disable_streams(struct v4l2_subdev *sd,
>  	struct ov2740 *ov2740 = to_ov2740(sd);
>  	int ret;
>  
> +	ov2740->streaming &= ~streams_mask;
> +	if (ov2740->streaming)
> +		return 0;
> +
>  	ret = ov2740_write_reg(ov2740, OV2740_REG_MODE_SELECT, 1,
>  			       OV2740_MODE_STANDBY);
>  

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 37/38] media: ccs: Add IMMUTABLE route flag

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Wed, Mar 13, 2024 at 09:25:15AM +0200, Sakari Ailus wrote:
> Add immutable route flag to the routing table.

I would write here that the metadata stream route is immutable as CCS
sensors don't offer a way to disable embedded data. Or if they do, that
it's currently unimplemented :-) I would then also possibly add a todo
comment to the code.

> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>

Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

> ---
>  drivers/media/i2c/ccs/ccs-core.c | 6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
> index 838daab212f2..5205d1ecf8f2 100644
> --- a/drivers/media/i2c/ccs/ccs-core.c
> +++ b/drivers/media/i2c/ccs/ccs-core.c
> @@ -3368,12 +3368,14 @@ static int ccs_src_init_state(struct v4l2_subdev *sd,
>  			.sink_pad = CCS_PAD_SINK,
>  			.source_pad = CCS_PAD_SRC,
>  			.source_stream = CCS_STREAM_PIXEL,
> -			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE |
> +				 V4L2_SUBDEV_ROUTE_FL_IMMUTABLE,
>  		}, {
>  			.sink_pad = CCS_PAD_META,
>  			.source_pad = CCS_PAD_SRC,
>  			.source_stream = CCS_STREAM_META,
> -			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE |
> +				 V4L2_SUBDEV_ROUTE_FL_IMMUTABLE,
>  		}
>  	};
>  	struct v4l2_subdev_krouting routing = {

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 38/38] media: ov2740: Add IMMUTABLE route flag

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Wed, Mar 13, 2024 at 09:25:16AM +0200, Sakari Ailus wrote:
> Add immutable route flag to the routing table.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>

Same comment as for patch 37/38.

Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

> ---
>  drivers/media/i2c/ov2740.c | 6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/media/i2c/ov2740.c b/drivers/media/i2c/ov2740.c
> index 7488b2535071..3e2585eb6ec9 100644
> --- a/drivers/media/i2c/ov2740.c
> +++ b/drivers/media/i2c/ov2740.c
> @@ -1150,12 +1150,14 @@ static int ov2740_init_state(struct v4l2_subdev *sd,
>  			.sink_pad = OV2740_PAD_PIXEL,
>  			.source_pad = OV2740_PAD_SOURCE,
>  			.source_stream = OV2740_STREAM_PIXEL,
> -			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE |
> +				 V4L2_SUBDEV_ROUTE_FL_IMMUTABLE,
>  		}, {
>  			.sink_pad = OV2740_PAD_META,
>  			.source_pad = OV2740_PAD_SOURCE,
>  			.source_stream = OV2740_STREAM_META,
> -			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE |
> +				 V4L2_SUBDEV_ROUTE_FL_IMMUTABLE,
>  		},
>  	};
>  	struct v4l2_subdev_krouting routing = {

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 36/38] media: v4l: Add V4L2_SUBDEV_ROUTE_FL_IMMUTABLE sub-device routing flag

[Laurent Pinchart]

Hello,

On Wed, Mar 13, 2024 at 07:39:17AM +0000, Sakari Ailus wrote:
> On Wed, Mar 13, 2024 at 09:34:13AM +0200, Tomi Valkeinen wrote:
> > On 13/03/2024 09:25, Sakari Ailus wrote:
> > > Add a flag to denote immutable routes, V4L2_SUBDEV_ROUTE_FL_IMMUTABLE.
> > > Such routes cannot be changed and they're always active.
> > > 
> > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > ---
> > >   Documentation/userspace-api/media/v4l/dev-subdev.rst         | 3 ++-
> > >   .../userspace-api/media/v4l/vidioc-subdev-g-routing.rst      | 5 +++++
> > >   include/uapi/linux/v4l2-subdev.h                             | 5 +++++
> > >   3 files changed, 12 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > index 08495cc6f4a6..2f2423f676cf 100644
> > > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > @@ -572,7 +572,8 @@ internal pad always has a single stream only (0).
> > >   Routes from an internal source pad to an external source pad are typically not
> > >   modifiable but they can be activated and deactivated using the
> > >   :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE <v4l2-subdev-routing-flags>` flag, depending
> > > -on driver capabilities.
> > > +on driver capabilities. This capatibility is indicated by the
> > > +:ref:`V4L2_SUBDEV_ROUTE_FL_IMMUTABLE <v4l2-subdev-routing-flags>` flag.

That's not very clear, it sounds like V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
indicates that the route can be enabled/disabled. I'd rewrite this.

> > >   Interaction between routes, streams, formats and selections
> > >   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > > index 08b8d17cef3f..cd7735f9104e 100644
> > > --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > > +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > > @@ -139,6 +139,11 @@ Also ``VIDIOC_SUBDEV_S_ROUTING`` may return more route than the user provided in
> > >       * - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > >         - 0x0001
> > >         - The route is enabled. Set by applications.
> > > +    * - V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
> > > +      - 0x0002
> > > +      - The route is immutable. Set by the driver. The
> > > +	``V4L2_SUBDEV_ROUTE_FL_ACTIVE`` flag of an immutable route may not be
> > > +	changed.

May not be changed and will always be set ?

> > >   Return Value
> > >   ============
> > > diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
> > > index ca543982460c..7e501cb45e4e 100644
> > > --- a/include/uapi/linux/v4l2-subdev.h
> > > +++ b/include/uapi/linux/v4l2-subdev.h
> > > @@ -200,6 +200,11 @@ struct v4l2_subdev_capability {
> > >    * on a video node.
> > >    */
> > >   #define V4L2_SUBDEV_ROUTE_FL_ACTIVE		(1U << 0)
> > > +/*
> > > + * Is the route immutable. The ACTIVE flag of an immutable route may not be
> > > + * changed.
> > > + */
> > > +#define V4L2_SUBDEV_ROUTE_FL_IMMUTABLE		(1U << 1)
> > >   /**
> > >    * struct v4l2_subdev_route - A route inside a subdev
> > 
> > Is the route fully immutable? The sink/source stream ID cannot be changed
> > (or any new fields we might come up with in the future)?
> 
> I think the new fields should be considered separately when they're added.
> This also applies to the stream IDs, I'll add this to the documentation.
> 
> The naming of the flag is aligned with MC link flag with a similar purpose.
> 
> > Hmm, or would a route with different stream IDs be a, well, different
> > route...
> > 
> > The docs here only talk about the ACTIVE flag. Would
> > V4L2_SUBDEV_ROUTE_FL_ALWAYS_ACTIVE be a better name, to be more explicit on
> > the meaning?
> 
> I prefer immutable. I wonder what Laurent and Hans think.

I'm fine with either. IMMUTABLE is shorter, if that makes a difference.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 35/38] media: ov2740: Add support for embedded data

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Wed, Mar 13, 2024 at 09:25:13AM +0200, Sakari Ailus wrote:
> Add support for embedded data. This introduces two internal pads for pixel
> and embedded data streams. As the driver supports a single mode only,
> there's no need for backward compatibility in mode selection.
> 
> The embedded data is configured to be placed before the image data whereas
> after the image data is the default.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  drivers/media/i2c/ov2740.c | 150 +++++++++++++++++++++++++++++++++----
>  1 file changed, 137 insertions(+), 13 deletions(-)
> 
> diff --git a/drivers/media/i2c/ov2740.c b/drivers/media/i2c/ov2740.c
> index df57f0096e98..7488b2535071 100644
> --- a/drivers/media/i2c/ov2740.c
> +++ b/drivers/media/i2c/ov2740.c
> @@ -11,6 +11,7 @@
>  #include <linux/pm_runtime.h>
>  #include <linux/nvmem-provider.h>
>  #include <linux/regmap.h>
> +#include <media/mipi-csi2.h>
>  #include <media/v4l2-ctrls.h>
>  #include <media/v4l2-device.h>
>  #include <media/v4l2-fwnode.h>
> @@ -71,11 +72,31 @@
>  #define OV2740_REG_ISP_CTRL00		0x5000
>  /* ISP CTRL01 */
>  #define OV2740_REG_ISP_CTRL01		0x5001
> +
> +/* Embedded data line location control */
> +#define OV2740_REG_EMBEDDED_FLAG	0x5a08
> +#define OV2740_EMBEDDED_FLAG_FOOTER	BIT(2) /* otherwise it's in header */
> +#define OV2740_EMBEDDED_FLAG_MYSTERY	BIT(1)
>  /* Customer Addresses: 0x7010 - 0x710F */
>  #define CUSTOMER_USE_OTP_SIZE		0x100
>  /* OTP registers from sensor */
>  #define OV2740_REG_OTP_CUSTOMER		0x7010
>  
> +enum {
> +	OV2740_PAD_SOURCE,
> +	OV2740_PAD_PIXEL,
> +	OV2740_PAD_META,
> +	OV2740_NUM_PADS,
> +};
> +
> +enum {
> +	OV2740_STREAM_PIXEL,
> +	OV2740_STREAM_META,
> +};
> +
> +#define OV2740_META_WIDTH		100U /* 97 bytes of actual data */
> +#define OV2740_META_HEIGHT		1U
> +
>  struct nvm_data {
>  	struct nvmem_device *nvmem;
>  	struct regmap *regmap;
> @@ -513,7 +534,7 @@ static const struct ov2740_mode supported_modes_180mhz[] = {
>  
>  struct ov2740 {
>  	struct v4l2_subdev sd;
> -	struct media_pad pad;
> +	struct media_pad pads[OV2740_NUM_PADS];
>  	struct v4l2_ctrl_handler ctrl_handler;
>  
>  	/* V4L2 Controls */
> @@ -976,6 +997,11 @@ static int ov2740_enable_streams(struct v4l2_subdev *sd,
>  	if (ret)
>  		goto out_pm_put;
>  
> +	ret = ov2740_write_reg(ov2740, OV2740_REG_EMBEDDED_FLAG, 1,
> +			       OV2740_EMBEDDED_FLAG_MYSTERY);
> +	if (ret)
> +		return ret;
> +
>  	ret = ov2740_write_reg(ov2740, OV2740_REG_MODE_SELECT, 1,
>  			       OV2740_MODE_STREAMING);
>  	if (ret) {
> @@ -1013,23 +1039,49 @@ static int ov2740_disable_streams(struct v4l2_subdev *sd,
>  	return ret;
>  }
>  
> -static int ov2740_set_format(struct v4l2_subdev *sd,
> -			     struct v4l2_subdev_state *sd_state,
> -			     struct v4l2_subdev_format *fmt)
> +static int __ov2740_set_format(struct v4l2_subdev *sd,
> +			       struct v4l2_subdev_state *sd_state,
> +			       struct v4l2_mbus_framefmt *format,
> +			       enum v4l2_subdev_format_whence which,
> +			       unsigned int pad, unsigned int stream)
>  {
> +	struct v4l2_mbus_framefmt *src_pix_fmt, *src_meta_fmt, *pix_fmt,
> +		*meta_fmt;
>  	struct ov2740 *ov2740 = to_ov2740(sd);
>  	const struct ov2740_mode *mode;
>  	s32 vblank_def, h_blank;
>  
> +	/*
> +	 * Allow setting format on internal pixel pad as well as the source
> +	 * pad's pixel stream (for compatibility).

The internal image pad represents the pixel array, it should thus report
the full pixel array size, and be unaffected by the selected mode.

> +	 */
> +	if (pad == OV2740_PAD_SOURCE || pad == OV2740_PAD_META ||
> +	    stream == OV2740_STREAM_META) {

As Julien pointed out, this isn't right.

> +		*format = *v4l2_subdev_state_get_format(sd_state, pad, stream);
> +		return 0;
> +	}
> +
> +	pix_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_PIXEL, 0);
> +	meta_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_META, 0);
> +	src_pix_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_SOURCE,
> +						   OV2740_STREAM_PIXEL);
> +	src_meta_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_SOURCE,
> +						    OV2740_STREAM_META);
> +
>  	mode = v4l2_find_nearest_size(ov2740->supported_modes,
>  				      ov2740->supported_modes_count,
>  				      width, height,
> -				      fmt->format.width, fmt->format.height);
> +				      format->width, format->height);
> +	ov2740_update_pad_format(mode, pix_fmt);
> +	*format = *src_pix_fmt = *pix_fmt;
>  
> -	ov2740_update_pad_format(mode, &fmt->format);
> -	*v4l2_subdev_state_get_format(sd_state, fmt->pad) = fmt->format;
> +	meta_fmt->code = MEDIA_BUS_FMT_OV2740_EMBEDDED;
> +	meta_fmt->width = OV2740_META_WIDTH;
> +	meta_fmt->height = OV2740_META_HEIGHT;
> +	*src_meta_fmt = *meta_fmt;
> +	src_meta_fmt->code = MEDIA_BUS_FMT_META_10;
>  
> -	if (fmt->which == V4L2_SUBDEV_FORMAT_TRY)
> +	if (which == V4L2_SUBDEV_FORMAT_TRY)
>  		return 0;
>  
>  	ov2740->cur_mode = mode;
> @@ -1049,6 +1101,14 @@ static int ov2740_set_format(struct v4l2_subdev *sd,
>  	return 0;
>  }
>  
> +static int ov2740_set_format(struct v4l2_subdev *sd,
> +			     struct v4l2_subdev_state *sd_state,
> +			     struct v4l2_subdev_format *fmt)
> +{
> +	return __ov2740_set_format(sd, sd_state, &fmt->format, fmt->which,
> +				   fmt->pad, fmt->stream);
> +}
> +
>  static int ov2740_enum_mbus_code(struct v4l2_subdev *sd,
>  				 struct v4l2_subdev_state *sd_state,
>  				 struct v4l2_subdev_mbus_code_enum *code)

You need to update this too, to enumerate the correct format on the
different pads and streams.

> @@ -1085,10 +1145,68 @@ static int ov2740_enum_frame_size(struct v4l2_subdev *sd,
>  static int ov2740_init_state(struct v4l2_subdev *sd,
>  			     struct v4l2_subdev_state *sd_state)
>  {
> +	struct v4l2_subdev_route routes[] = {
> +		{
> +			.sink_pad = OV2740_PAD_PIXEL,
> +			.source_pad = OV2740_PAD_SOURCE,
> +			.source_stream = OV2740_STREAM_PIXEL,
> +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> +		}, {
> +			.sink_pad = OV2740_PAD_META,
> +			.source_pad = OV2740_PAD_SOURCE,
> +			.source_stream = OV2740_STREAM_META,
> +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> +		},
> +	};
> +	struct v4l2_subdev_krouting routing = {
> +		.routes = routes,
> +		.num_routes = ARRAY_SIZE(routes),
> +	};
> +	struct v4l2_subdev_state *active_state;
> +	struct v4l2_mbus_framefmt format = { 0 };
>  	struct ov2740 *ov2740 = to_ov2740(sd);
> +	int ret;
> +
> +	ret = v4l2_subdev_set_routing(sd, sd_state, &routing);
> +	if (ret)
> +		return ret;
> +
> +	active_state = v4l2_subdev_get_locked_active_state(sd);

There's a lockdep assertion that will trip when initializing any try
state.

> +
> +	ov2740_update_pad_format(&ov2740->supported_modes[0], &format);
> +
> +	return __ov2740_set_format(sd, sd_state, &format,
> +				   active_state == sd_state ?
> +				   V4L2_SUBDEV_FORMAT_ACTIVE :
> +				   V4L2_SUBDEV_FORMAT_TRY, OV2740_PAD_PIXEL, 0);

Move the code specific to the active state from __ov2740_set_format() to
ov2740_set_format(), and drop the which parameter to
__ov2740_set_format(). You'll simplify the code here.

> +}
> +
> +static int ov2740_get_frame_desc(struct v4l2_subdev *sd, unsigned int pad,
> +				 struct v4l2_mbus_frame_desc *desc)
> +{
> +	struct v4l2_mbus_frame_desc_entry *entry = desc->entry;
> +	struct v4l2_subdev_state *sd_state;
> +	struct v4l2_mbus_framefmt *fmt;
> +
> +	desc->type = V4L2_MBUS_FRAME_DESC_TYPE_CSI2;
> +
> +	sd_state = v4l2_subdev_lock_and_get_active_state(sd);
> +	fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_SOURCE,
> +					   OV2740_STREAM_PIXEL);
> +	entry->pixelcode = fmt->code;
> +	v4l2_subdev_unlock_state(sd_state);
> +
> +	entry->stream = OV2740_STREAM_PIXEL;
> +	entry->bus.csi2.dt = MIPI_CSI2_DT_RAW10;
> +	entry++;
> +	desc->num_entries++;

I think addressing entries explicitly would be clearer.

	entry[0].stream = ...;
	entry[0].bus.csi2.dt = ...;

	...

	desc->num_entries = 2;

> +
> +	entry->pixelcode = MEDIA_BUS_FMT_META_8;
> +	entry->stream = OV2740_STREAM_META;
> +	entry->bus.csi2.dt = MIPI_CSI2_DT_GENERIC_LONG(1);

As Bingbu mentioned, this is not correct.

> +	entry++;
> +	desc->num_entries++;
>  
> -	ov2740_update_pad_format(&ov2740->supported_modes[0],
> -				 v4l2_subdev_state_get_format(sd_state, 0));
>  	return 0;
>  }
>  
> @@ -1103,6 +1221,7 @@ static const struct v4l2_subdev_pad_ops ov2740_pad_ops = {
>  	.enum_frame_size = ov2740_enum_frame_size,
>  	.enable_streams = ov2740_enable_streams,
>  	.disable_streams = ov2740_disable_streams,
> +	.get_frame_desc = ov2740_get_frame_desc,
>  };
>  
>  static const struct v4l2_subdev_ops ov2740_subdev_ops = {
> @@ -1369,11 +1488,16 @@ static int ov2740_probe(struct i2c_client *client)
>  	}
>  
>  	ov2740->sd.state_lock = ov2740->ctrl_handler.lock;
> -	ov2740->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
> +	ov2740->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_STREAMS;
>  	ov2740->sd.entity.ops = &ov2740_subdev_entity_ops;
>  	ov2740->sd.entity.function = MEDIA_ENT_F_CAM_SENSOR;
> -	ov2740->pad.flags = MEDIA_PAD_FL_SOURCE;
> -	ret = media_entity_pads_init(&ov2740->sd.entity, 1, &ov2740->pad);
> +	ov2740->pads[OV2740_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
> +	ov2740->pads[OV2740_PAD_PIXEL].flags =
> +		MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL;
> +	ov2740->pads[OV2740_PAD_META].flags =
> +		MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL;
> +	ret = media_entity_pads_init(&ov2740->sd.entity,
> +				     ARRAY_SIZE(ov2740->pads), ov2740->pads);
>  	if (ret) {
>  		dev_err_probe(dev, ret, "failed to init entity pads\n");
>  		goto probe_error_v4l2_ctrl_handler_free;

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 01/38] media: mc: Add INTERNAL pad flag

[Laurent Pinchart]

On Wed, Mar 20, 2024 at 07:49:27AM +0000, Sakari Ailus wrote:
> On Wed, Mar 20, 2024 at 12:17:07AM +0200, Laurent Pinchart wrote:
> > On Thu, Mar 14, 2024 at 09:17:08AM +0200, Tomi Valkeinen wrote:
> > > On 13/03/2024 09:24, Sakari Ailus wrote:
> > > > Internal source pads will be used as routing endpoints in V4L2
> > > > [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
> > > > Internal source pads are pads that have both SINK and INTERNAL flags set.
> > > > 
> > > > Also prevent creating links to pads that have been flagged as internal and
> > > > initialising SOURCE pads with INTERNAL flag set.
> > > > 
> > > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > > ---
> > > >   .../userspace-api/media/mediactl/media-types.rst       |  8 ++++++++
> > > >   drivers/media/mc/mc-entity.c                           | 10 ++++++++--
> > > >   include/uapi/linux/media.h                             |  1 +
> > > >   3 files changed, 17 insertions(+), 2 deletions(-)
> > > > 
> > > > diff --git a/Documentation/userspace-api/media/mediactl/media-types.rst b/Documentation/userspace-api/media/mediactl/media-types.rst
> > > > index 6332e8395263..f55ef055bcf8 100644
> > > > --- a/Documentation/userspace-api/media/mediactl/media-types.rst
> > > > +++ b/Documentation/userspace-api/media/mediactl/media-types.rst
> > > > @@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
> > > >   .. _MEDIA-PAD-FL-SINK:
> > > >   .. _MEDIA-PAD-FL-SOURCE:
> > > >   .. _MEDIA-PAD-FL-MUST-CONNECT:
> > > > +.. _MEDIA-PAD-FL-INTERNAL:
> > > >   
> > > >   .. flat-table:: Media pad flags
> > > >       :header-rows:  0
> > > > @@ -381,6 +382,13 @@ Types and flags used to represent the media graph elements
> > > >   	  enabled links even when this flag isn't set; the absence of the flag
> > > >   	  doesn't imply there is none.
> > > >   
> > > > +    *  -  ``MEDIA_PAD_FL_INTERNAL``
> > > > +       -  The internal flag indicates an internal pad that has no external
> > > > +	  connections. Such a pad shall not be connected with a link.
> > 
> > I would expand this slightly, as it's the only source of documentation
> > regarding internal pads.
> 
> Patch 9 adds more documentation, this patch is for MC only.

That's my point, it's the only source of documentation for internal
pads from an MC point of view, so expanding the documentation would be
good :-)

> > 
> >        -  The internal flag indicates an internal pad that has no external
> > 	  connections. This can be used to model, for instance, the pixel array
> > 	  internal to an image sensor. As they are internal to entities,
> > 	  internal pads shall not be connected with links.
> 
> I'd drop the sentence related to sensors.

I'm fine with another example, or a more generic explanation, but with
that sentence dropped, I think this will leave the reader wondering what
an internal pad is and what it's used for.

> > > > +
> > > > +	  The internal flag may currently be present only in a source pad where
> > > 
> > > s/source/sink/
> > > 
> > > Reviewed-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > > > +	  it indicates that the :ref:``stream <media-glossary-stream>``
> > > > +	  originates from within the entity.
> > > >   
> > > >   One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
> > > >   must be set for every pad.
> > > > diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> > > > index 0e28b9a7936e..1973e9e1013e 100644
> > > > --- a/drivers/media/mc/mc-entity.c
> > > > +++ b/drivers/media/mc/mc-entity.c
> > > > @@ -213,7 +213,9 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
> > > >   		iter->index = i++;
> > > >   
> > > >   		if (hweight32(iter->flags & (MEDIA_PAD_FL_SINK |
> > > > -					     MEDIA_PAD_FL_SOURCE)) != 1) {
> > > > +					     MEDIA_PAD_FL_SOURCE)) != 1 ||
> > > > +		    (iter->flags & MEDIA_PAD_FL_INTERNAL &&
> > > > +		     !(iter->flags & MEDIA_PAD_FL_SINK))) {
> > > >   			ret = -EINVAL;
> > > >   			break;
> > > >   		}
> > 
> > This may become more readable written as a switch statement:
> > 
> > 		const u32 pad_flags_mask = MEDIA_PAD_FL_SINK |
> > 					   MEDIA_PAD_FL_SOURCE |
> > 					   MEDIA_PAD_FL_INTERNAL;
> > 
> > 		switch (iter->flags & pad_flags_mask) {
> > 		case MEDIA_PAD_FL_SINK:
> > 		case MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL:
> > 		case MEDIA_PAD_FL_SOURCE:
> > 			break;
> > 
> > 		default:
> > 			ret = -EINVAL;
> > 			break;
> > 		}
> > 
> > 		if (ret)
> > 			break;
> > 
> > And now that I've written this, I'm not too sure anymore :-) Another
> > option would be
> > 
> > 
> > 	const u32 pad_flags = iter->flags & (MEDIA_PAD_FL_SINK |
> > 					     MEDIA_PAD_FL_SOURCE |
> > 					     MEDIA_PAD_FL_INTERNAL);
> > 
> > 	if (pad_flags != MEDIA_PAD_FL_SINK &&
> > 	    pad_flags != MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL
> > 	    pad_flags != MEDIA_PAD_FL_SOURCE) {
> > 		ret = -EINVAL;
> > 		break;
> > 	}
> > 
> > Up to you.
> 
> I'd prefer to keep it as-is since the original check is testing two
> independent things instead of merging them: that only either SINK or SOURCE
> is set, and then separately that if INTERNAL is set, then SINK is set, too.
> 
> Of the two options you suggested, I prefer the latter.

I prefer the latter too, and I think it's more readable than the current
code. If we later end up having to test for more rules, we can separate
the checks.

> > 
> > Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> 
> Thank you!
> 
> > > > @@ -1112,7 +1114,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
> > > >   
> > > >   	for (i = 0; i < entity->num_pads; i++) {
> > > >   		if ((entity->pads[i].flags &
> > > > -		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
> > > > +		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
> > > > +		      MEDIA_PAD_FL_INTERNAL)) != pad_type)
> > > >   			continue;
> > > >   
> > > >   		if (entity->pads[i].sig_type == sig_type)
> > > > @@ -1142,6 +1145,9 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
> > > >   		return -EINVAL;
> > > >   	if (WARN_ON(!(sink->pads[sink_pad].flags & MEDIA_PAD_FL_SINK)))
> > > >   		return -EINVAL;
> > > > +	if (WARN_ON(source->pads[source_pad].flags & MEDIA_PAD_FL_INTERNAL) ||
> > > > +	    WARN_ON(sink->pads[sink_pad].flags & MEDIA_PAD_FL_INTERNAL))
> > > > +		return -EINVAL;
> > > >   
> > > >   	link = media_add_link(&source->links);
> > > >   	if (link == NULL)
> > > > diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
> > > > index 1c80b1d6bbaf..80cfd12a43fc 100644
> > > > --- a/include/uapi/linux/media.h
> > > > +++ b/include/uapi/linux/media.h
> > > > @@ -208,6 +208,7 @@ struct media_entity_desc {
> > > >   #define MEDIA_PAD_FL_SINK			(1U << 0)
> > > >   #define MEDIA_PAD_FL_SOURCE			(1U << 1)
> > > >   #define MEDIA_PAD_FL_MUST_CONNECT		(1U << 2)
> > > > +#define MEDIA_PAD_FL_INTERNAL			(1U << 3)
> > > >   
> > > >   struct media_pad_desc {
> > > >   	__u32 entity;		/* entity ID */

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 03/38] media: uapi: Add generic serial metadata mbus formats

[Laurent Pinchart]

On Wed, Mar 20, 2024 at 08:48:17AM +0000, Sakari Ailus wrote:
> On Wed, Mar 20, 2024 at 01:00:48AM +0200, Laurent Pinchart wrote:
> > On Wed, Mar 20, 2024 at 12:33:48AM +0200, Laurent Pinchart wrote:
> > > On Tue, Mar 19, 2024 at 04:20:35PM +0200, Tomi Valkeinen wrote:
> > > > On 19/03/2024 15:27, Sakari Ailus wrote:
> > > > > On Thu, Mar 14, 2024 at 09:30:50AM +0200, Tomi Valkeinen wrote:
> > > > >> On 13/03/2024 09:24, Sakari Ailus wrote:
> > > > >>> Add generic serial metadata mbus formats. These formats describe data
> > > > >>> width and packing but not the content itself. The reason for specifying
> > > > >>> such formats is that the formats as such are fairly device specific but
> > > > >>> they are still handled by CSI-2 receiver drivers that should not be aware
> > > > >>> of device specific formats. What makes generic metadata formats possible
> > > > >>> is that these formats are parsed by software only, after capturing the
> > > > >>> data to system memory.
> > > > >>>
> > > > >>> Also add a definition for "Data unit" to cover what is essentially a pixel
> > > > >>> but is not image data.
> > > > >>
> > > > >> The CCS spec talks about legacy packing and optimized packing for 16+ bit
> > > > >> formats. You cover only the "legacy" ones here. Did you look at those?
> > > > > 
> > > > > The reason is that the bus data layout of the new packing at higher bit
> > > > > depth matches with packing at lower bit depths (half to be precise). That's
> > > > > why there's no need to define formats for the new packing methods at higher
> > > > > bit depths (the driver simply uses the packing at half of the bit depth).
> > > > 
> > > > Hmm. If we're capturing 10-bit raw format, say, with the width of 640 
> > > > pixels, we'll configure the video stream format according to those. For 
> > > > the embedded stream, we'll set it to V4L2_META_FMT_GENERIC_CSI2_10 and 
> > > > 640 width, right?
> > > > 
> > > > If we're capturing 20-bit raw, we'll configure the video stream format 
> > > > again accordingly, width to 640, and 20 bit fourcc/mbus code. If the 
> > > > embedded stream uses the "legacy" packing, we'll set the format to 
> > > > V4L2_META_FMT_GENERIC_CSI2_20 with width of 640, right?
> > > > 
> > > > But if it's using packed format for the embedded stream, we set the 
> > > > format to V4L2_META_FMT_GENERIC_CSI2_10 and width to 1280?
> > > > 
> > > > Considering that the video and (line-based) embedded data come from the 
> > > > same source, I'd expect the widths to be the same.
> > > 
> > > I don't have a strong objection against multiplying the width, but we
> > > need to figure out the impact on other kernel space components, as well
> > > as on userspace. I suppose the media bus code for the embedded data
> > > stream on the sensor source pad when using optimized packing and
> > > capturing RAW20 images would be MEDIA_BUS_FMT_META_10 ? In that case I
> > > think the sensor driver should be able to handle the width calculations
> > > on its own, and the value would just be propagated by userspace.
> > 
> > This should be documented somewhere in this series by the way (not in
> > this patch).
> 
> This could go to the CCS driver documentation. I modified the last
> paragraph and added a new one:
> 
> ------8<-----------
> Devices supporting embedded data for bit depths greater than or equal to 16 may
> support more dense packing or legacy single metadata byte per data unit, or both
> of these. The supported embedded data formats can be enumerated and configured
> on stream 1 of the source pad (1) of the CCS source sub-device.
> 
> The use of the denser packing results in embedded data lines being longer than
> the pixel data in data units since the data units are smaller. In bytes the
> embedded data lines are still not longer than the image data lines.

Please document explicitly that e.g. V4L2_META_FMT_GENERIC_CSI2_10 is
used for the RAW20 denser packing (in a general way that covers the
other formats too). You should also explain more explicitly that the
width is doubled in the relevant uAPI data structures.

This is not limited to CCS but is applicable to other sensors too, so
I'd like that documentation to be in a more generic place.

> ------8<-----------
> 
> I believe the reason for the specs requiring embedded data lines not being
> longer (in bytes) is most likely that some hardware may have issues
> reciving the data otherwise for various reasons.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 03/38] media: uapi: Add generic serial metadata mbus formats

[Laurent Pinchart]

Hi Sakari,

On Wed, Mar 20, 2024 at 04:23:01PM +0000, Sakari Ailus wrote:
> On Wed, Mar 20, 2024 at 12:59:48AM +0200, Laurent Pinchart wrote:
> > On Wed, Mar 13, 2024 at 09:24:41AM +0200, Sakari Ailus wrote:
> > > Add generic serial metadata mbus formats. These formats describe data
> > > width and packing but not the content itself. The reason for specifying
> > > such formats is that the formats as such are fairly device specific but
> > > they are still handled by CSI-2 receiver drivers that should not be aware
> > > of device specific formats. What makes generic metadata formats possible
> > > is that these formats are parsed by software only, after capturing the
> > > data to system memory.
> > > 
> > > Also add a definition for "Data unit" to cover what is essentially a pixel
> > > but is not image data.
> > > 
> > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > ---
> > >  .../userspace-api/media/glossary.rst          |   9 +
> > >  .../media/v4l/subdev-formats.rst              | 258 ++++++++++++++++++
> > >  include/uapi/linux/media-bus-format.h         |   9 +
> > >  3 files changed, 276 insertions(+)
> > > 
> > > diff --git a/Documentation/userspace-api/media/glossary.rst b/Documentation/userspace-api/media/glossary.rst
> > > index ef0ab601b5bf..7078141894c5 100644
> > > --- a/Documentation/userspace-api/media/glossary.rst
> > > +++ b/Documentation/userspace-api/media/glossary.rst
> > > @@ -25,6 +25,15 @@ Glossary
> > >  
> > >  	See :ref:`cec`.
> > >  
> > > +.. _media-glossary-data-unit:
> > > +
> > > +    Data unit
> > > +
> > > +	Unit of data transported by a bus. On parallel buses, the data unit
> > > +	consists of one or more related samples while on serial buses the data
> > > +	unit is logical. If the data unit is image data, it may also be called a
> > > +	pixel.
> > 
> > I'm pretty sure nobody will be able to understand what this means, but I
> > don't have a better proposal at the moment.
> > 
> > > +
> > >      Device Driver
> > >  	Part of the Linux Kernel that implements support for a hardware
> > >  	component.
> > > diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > > index eb3cd20b0cf2..cbd475f7cae9 100644
> > > --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > > +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > > @@ -8306,3 +8306,261 @@ The following table lists the existing metadata formats.
> > >  	both sides of the link and the bus format is a fixed
> > >  	metadata format that is not configurable from userspace.
> > >  	Width and height will be set to 0 for this format.
> > > +
> > > +Generic Serial Metadata Formats
> > > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > +
> > > +Generic serial metadata formats are used on serial buses where the actual data
> > > +content is more or less device specific but the data is transmitted and received
> > > +by multiple devices that do not process the data in any way, simply writing
> > > +it to system memory for processing in software at the end of the pipeline.
> > > +
> > > +The more specific variant describing the actual data is used on the internal
> > > +source pad of the originating sub-device.
> > 
> > Maybe this paragraph would be best added in the patch that adds the
> > specific metadata formats, you could then mention one of them as an
> > example:
> > 
> > The exact format of the data generated by the device is reported on the
> > internal source pad of the originating sub-device, using one of the more
> > specific metadata formats such as MEDIA_BUS_FMT_CCS_EMBEDDED.
> 
> I'd really like to get rid of the "internal source" pads as the naming is
> really confusing (it's present still in this version but not in many
> locations). They're sink pads after all, so I'd call them such. In a few
> locations there's text that explains they do represent sources of data
> within the sub-device itself.

I thought we had agreed to use the term "internal sink pad", but I saw a
patch in this series that consistently makes use of "internal source
pad". Both are confusing for different reasons, but regardless of which
one we pick (that discussion belongs to the other patch that documents
the internal source pads), please make sure to use the same terms
consistently through the series in v9.

> > > +
> > > +"b" in an array cell signifies a byte of data, followed by the number of the bit
> > 
> > s/bit$/byte/
> 
> Uh, yes.
> 
> > > +and finally the bit number in subscript. "X" indicates a padding bit.
> > 
> > We use a lower-case x in pixfmt-rgb.rst, I would do the same here. We
> > also use single quotes there, turning "b" and "x" into 'b' and 'x'.
> 
> This is documentation, not C source code where we'd want to denote a single
> character. Double quotes should thus be used instead.

We seem to use single quotes in format documentations, I'd continue
doing so for consistency.

> I'll switch to lower case X.
> 
> > Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> 
> Thank you!

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 29/38] media: ccs: Remove which parameter from ccs_propagate

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Wed, Mar 13, 2024 at 09:25:07AM +0200, Sakari Ailus wrote:
> ccs_propagate() no longer stores information in the driver's context
> struct. The which parameter can thus be removed.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>

Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

> ---
>  drivers/media/i2c/ccs/ccs-core.c | 12 +++++-------
>  1 file changed, 5 insertions(+), 7 deletions(-)
> 
> diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
> index a147dbb9f362..838daab212f2 100644
> --- a/drivers/media/i2c/ccs/ccs-core.c
> +++ b/drivers/media/i2c/ccs/ccs-core.c
> @@ -2304,8 +2304,7 @@ static int ccs_get_format(struct v4l2_subdev *subdev,
>  
>  /* Changes require propagation only on sink pad. */
>  static void ccs_propagate(struct v4l2_subdev *subdev,
> -			  struct v4l2_subdev_state *sd_state, int which,
> -			  int target)
> +			  struct v4l2_subdev_state *sd_state, int target)
>  {
>  	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
>  	struct v4l2_rect *comp, *crop;
> @@ -2512,7 +2511,7 @@ static int ccs_set_format(struct v4l2_subdev *subdev,
>  	crop->top = 0;
>  	crop->width = fmt->format.width;
>  	crop->height = fmt->format.height;
> -	ccs_propagate(subdev, sd_state, fmt->which, V4L2_SEL_TGT_CROP);
> +	ccs_propagate(subdev, sd_state, V4L2_SEL_TGT_CROP);
>  
>  	return 0;
>  }
> @@ -2727,7 +2726,7 @@ static int ccs_set_compose(struct v4l2_subdev *subdev,
>  		ccs_set_compose_scaler(subdev, sd_state, sel, sink_crop, comp);
>  
>  	*comp = sel->r;
> -	ccs_propagate(subdev, sd_state, sel->which, V4L2_SEL_TGT_COMPOSE);
> +	ccs_propagate(subdev, sd_state, V4L2_SEL_TGT_COMPOSE);
>  
>  	if (sel->which == V4L2_SUBDEV_FORMAT_ACTIVE)
>  		return ccs_pll_blanking_update(sensor);
> @@ -2817,7 +2816,7 @@ static int ccs_set_crop(struct v4l2_subdev *subdev,
>  	*crop = sel->r;
>  
>  	if (ssd != sensor->pixel_array && sel->pad == CCS_PAD_SINK)
> -		ccs_propagate(subdev, sd_state, sel->which, V4L2_SEL_TGT_CROP);
> +		ccs_propagate(subdev, sd_state, V4L2_SEL_TGT_CROP);
>  
>  	return 0;
>  }
> @@ -3339,7 +3338,6 @@ static int ccs_init_state(struct v4l2_subdev *sd,
>  		v4l2_subdev_state_get_format(sd_state, pad, CCS_STREAM_PIXEL);
>  	struct v4l2_rect *crop =
>  		v4l2_subdev_state_get_crop(sd_state, pad, CCS_STREAM_PIXEL);
> -	bool is_active = !sd->active_state || sd->active_state == sd_state;
>  
>  	ccs_get_native_size(ssd, crop);
>  
> @@ -3357,7 +3355,7 @@ static int ccs_init_state(struct v4l2_subdev *sd,
>  		sensor->csi_format->code : sensor->internal_csi_format->code;
>  	fmt->field = V4L2_FIELD_NONE;
>  
> -	ccs_propagate(sd, sd_state, is_active, V4L2_SEL_TGT_CROP);
> +	ccs_propagate(sd, sd_state, V4L2_SEL_TGT_CROP);
>  
>  	return 0;
>  }

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 30/38] media: Documentation: ccs: Document routing

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Wed, Mar 13, 2024 at 09:25:08AM +0200, Sakari Ailus wrote:
> Document which routes are available for the CCS driver (source) sub-device
> and what configuration are possible.
> 
> Also update copyright.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../userspace-api/media/drivers/ccs.rst       | 34 ++++++++++++++++++-
>  .../media/v4l/subdev-formats.rst              |  2 ++
>  2 files changed, 35 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/userspace-api/media/drivers/ccs.rst b/Documentation/userspace-api/media/drivers/ccs.rst
> index 03015b33d5ab..bc2804ec663b 100644
> --- a/Documentation/userspace-api/media/drivers/ccs.rst
> +++ b/Documentation/userspace-api/media/drivers/ccs.rst
> @@ -111,4 +111,36 @@ than in the centre.
>  Shading correction needs to be enabled for luminance correction level to have an
>  effect.
>  
> -**Copyright** |copy| 2020 Intel Corporation
> +.. _media-ccs-routes:
> +
> +Routes
> +------
> +
> +The CCS driver implements one or two :ref:`routes <subdev-routing>` in
> +its source sub-device (scaler sub-device if exists for the device, otherwise
> +binner) depending on whether the sensor supports embedded data. (All CCS
> +compliant sensors do but the CCS driver supports preceding standards that did
> +not require embedded data support, too.)

s/too.)/too).

> +
> +The first route of the CCS source sub-device is for pixel data (internal pad
> +1/stream 0 -> pad 0/stream 0) and the second one is for embedded data (internal

I thought the source subdev had one external sink pad for image data,
one internal sink pad for embedded data, and one source pad.

> +pad 2/stream 0 -> pad 0/stream 1).
> +
> +Embedded data
> +~~~~~~~~~~~~~
> +
> +MIPI CCS supports generation of camera sensor embedded data. The media bus code
> +used for this format is :ref:`MEDIA_BUS_FMT_CCS_EMBEDDDED
> +<MEDIA-BUS-FMT-CCS-EMBEDDED>`.

Maybe state that this is the format on the internal ED sink pad.

> +
> +The bit depth of the CCS pixel data affects how the sensor will output the
> +embedded data, adding padding to align with CSI-2 bus :ref:`Data units
> +<media-glossary-data-unit>` for that particular bit depth. This is indicated by
> +the generic metadata format on the sensor's source sub-device's source pad.
> +
> +Embedded data for bit depths greater than or equal to 16 may support more dense
> +packing or legacy single metadata byte per data unit, or both of these,
> +depending on the device. The supported embedded data formats can be enumerated
> +and configured on stream 1 of the source pad (1) of the CCS source sub-device.
> +
> +**Copyright** |copy| 2020, 2023 Intel Corporation

2024 ?

> diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> index c8f982411e70..ca4da6a400ff 100644
> --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
> +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> @@ -8592,3 +8592,5 @@ levels above.
>  This mbus code are only used for "2-byte simplified tagged data format" (code
>  0xa) but their use may be extended further in the future, to cover other CCS
>  embedded data format codes.
> +
> +Also see :ref:`CCS driver documentation <media-ccs-routes>`.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 28/38] media: ccs: Compute scaling configuration from sub-device state

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Wed, Mar 13, 2024 at 09:25:06AM +0200, Sakari Ailus wrote:
> Compute scaling configuration from sub-device state instead of storing it
> to the driver's device context struct.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  drivers/media/i2c/ccs/ccs-core.c | 60 ++++++++++++++++++++++----------
>  drivers/media/i2c/ccs/ccs.h      |  3 --
>  2 files changed, 41 insertions(+), 22 deletions(-)
> 
> diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
> index 3b80c54453cc..a147dbb9f362 100644
> --- a/drivers/media/i2c/ccs/ccs-core.c
> +++ b/drivers/media/i2c/ccs/ccs-core.c
> @@ -547,19 +547,52 @@ ccs_get_binning(struct ccs_sensor *sensor, u8 *binning_mode, u8 *binh, u8 *binv)
>  		*binv = binner_sink_crop->height / binner_sink_comp->height;
>  }
>  
> +static void ccs_get_scaling(struct ccs_sensor *sensor,
> +			    u8 *scaling_mode, u8 *scale_m)
> +{
> +	struct v4l2_subdev_state *scaler_state =
> +		v4l2_subdev_get_locked_active_state(&sensor->scaler->sd);

Have you double-checked that the scaler state is locked in all code
paths that can lead to this ? The function is called at probe time, with
a manual lock of sensor->mutex. That makes me a bit uncomfortable, I
wonder if it wouldn't be better to pass the locked scaler state to the
function explicitly, to let the callers guarantee the locking
requirements.

> +	struct v4l2_rect *scaler_sink_crop =

const

I would also drop the scaler_ prefix here and for the next variable.

> +		v4l2_subdev_state_get_crop(scaler_state, CCS_PAD_SINK,
> +					   CCS_STREAM_PIXEL);
> +	struct v4l2_rect *scaler_sink_comp =
> +		v4l2_subdev_state_get_compose(scaler_state, CCS_PAD_SINK,
> +					      CCS_STREAM_PIXEL);
> +
> +	if (scale_m)
> +		*scale_m = scaler_sink_crop->width *
> +			CCS_LIM(sensor, SCALER_N_MIN) /
> +			scaler_sink_comp->width;
> +
> +	if (scaling_mode) {
> +		if (scaler_sink_crop->width == scaler_sink_comp->width)
> +			*scaling_mode = CCS_SCALING_MODE_NO_SCALING;
> +		else if (scaler_sink_crop->height == scaler_sink_comp->height)
> +			*scaling_mode = CCS_SCALING_MODE_HORIZONTAL;
> +		else
> +			*scaling_mode = SMIAPP_SCALING_MODE_BOTH;
> +	}
> +}
> +
>  static int ccs_pll_update(struct ccs_sensor *sensor)
>  {
>  	struct ccs_pll *pll = &sensor->pll;
>  	u8 binh, binv;
> +	u8 scale_m;
>  	int rval;
>  
>  	ccs_get_binning(sensor, NULL, &binh, &binv);
>  
> +	if (sensor->scaler)
> +		ccs_get_scaling(sensor, NULL, &scale_m);
> +	else
> +		scale_m = CCS_LIM(sensor, SCALER_N_MIN);
> +
>  	pll->binning_horizontal = binh;
>  	pll->binning_vertical = binv;
>  	pll->link_freq =
>  		sensor->link_freq->qmenu_int[sensor->link_freq->val];
> -	pll->scale_m = sensor->scale_m;
> +	pll->scale_m = scale_m;
>  	pll->bits_per_pixel = sensor->csi_format->compressed;
>  
>  	rval = ccs_pll_try(sensor, pll);
> @@ -1202,7 +1235,7 @@ static int ccs_get_mbus_formats(struct ccs_sensor *sensor)
>  	/* Figure out which BPP values can be used with which formats. */
>  	pll->binning_horizontal = 1;
>  	pll->binning_vertical = 1;
> -	pll->scale_m = sensor->scale_m;
> +	pll->scale_m = CCS_LIM(sensor, SCALER_N_MIN);
>  
>  	for (i = 0; i < ARRAY_SIZE(ccs_csi_data_formats); i++) {
>  		sensor->compressed_min_bpp =
> @@ -1950,11 +1983,15 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
>  	/* Scaling */
>  	if (CCS_LIM(sensor, SCALING_CAPABILITY)
>  	    != CCS_SCALING_CAPABILITY_NONE) {
> -		rval = ccs_write(sensor, SCALING_MODE, sensor->scaling_mode);
> +		u8 scaling_mode, scale_m;
> +
> +		ccs_get_scaling(sensor, &scaling_mode, &scale_m);
> +
> +		rval = ccs_write(sensor, SCALING_MODE, scaling_mode);
>  		if (rval < 0)
>  			goto err_pm_put;
>  
> -		rval = ccs_write(sensor, SCALE_M, sensor->scale_m);
> +		rval = ccs_write(sensor, SCALE_M, scale_m);
>  		if (rval < 0)
>  			goto err_pm_put;
>  	}
> @@ -2270,7 +2307,6 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
>  			  struct v4l2_subdev_state *sd_state, int which,
>  			  int target)
>  {
> -	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
>  	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
>  	struct v4l2_rect *comp, *crop;
>  	struct v4l2_mbus_framefmt *fmt;
> @@ -2283,13 +2319,6 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
>  						  CCS_STREAM_PIXEL);
>  		comp->width = crop->width;
>  		comp->height = crop->height;
> -		if (which == V4L2_SUBDEV_FORMAT_ACTIVE) {
> -			if (ssd == sensor->scaler) {
> -				sensor->scale_m = CCS_LIM(sensor, SCALER_N_MIN);
> -				sensor->scaling_mode =
> -					CCS_SCALING_MODE_NO_SCALING;
> -			}
> -		}
>  		fallthrough;
>  	case V4L2_SEL_TGT_COMPOSE:
>  		crop = v4l2_subdev_state_get_crop(sd_state, CCS_PAD_SRC,
> @@ -2674,11 +2703,6 @@ static void ccs_set_compose_scaler(struct v4l2_subdev *subdev,
>  			& ~1;
>  	else
>  		sel->r.height = sink_crop->height;
> -
> -	if (sel->which == V4L2_SUBDEV_FORMAT_ACTIVE) {
> -		sensor->scale_m = scale_m;
> -		sensor->scaling_mode = mode;
> -	}
>  }
>  /* We're only called on source pads. This function sets scaling. */
>  static int ccs_set_compose(struct v4l2_subdev *subdev,
> @@ -3785,8 +3809,6 @@ static int ccs_probe(struct i2c_client *client)
>  	sensor->pixel_array = &sensor->ssds[sensor->ssds_used];
>  	sensor->ssds_used++;
>  
> -	sensor->scale_m = CCS_LIM(sensor, SCALER_N_MIN);
> -
>  	/* prepare PLL configuration input values */
>  	sensor->pll.bus_type = CCS_PLL_BUS_TYPE_CSI2_DPHY;
>  	sensor->pll.csi2.lanes = sensor->hwcfg.lanes;
> diff --git a/drivers/media/i2c/ccs/ccs.h b/drivers/media/i2c/ccs/ccs.h
> index e6fc00a9fa11..d33014f2710b 100644
> --- a/drivers/media/i2c/ccs/ccs.h
> +++ b/drivers/media/i2c/ccs/ccs.h
> @@ -237,9 +237,6 @@ struct ccs_sensor {
>  	u32 embedded_mbus_code;
>  	u8 emb_data_ctrl;
>  
> -	u8 scale_m;
> -	u8 scaling_mode;
> -
>  	u8 frame_skip;
>  	u16 embedded_start; /* embedded data start line */
>  	u16 embedded_end;

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 27/38] media: ccs: Compute binning configuration from sub-device state

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Wed, Mar 13, 2024 at 09:25:05AM +0200, Sakari Ailus wrote:
> Calculate binning configuration from sub-device state so the state related
> configuration can be removed from the driver's device context struct.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  drivers/media/i2c/ccs/ccs-core.c | 72 ++++++++++++++++++++++----------
>  drivers/media/i2c/ccs/ccs.h      |  3 --
>  2 files changed, 49 insertions(+), 26 deletions(-)
> 
> diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
> index a8c48abd2e30..3b80c54453cc 100644
> --- a/drivers/media/i2c/ccs/ccs-core.c
> +++ b/drivers/media/i2c/ccs/ccs-core.c
> @@ -511,13 +511,52 @@ static int ccs_pll_try(struct ccs_sensor *sensor, struct ccs_pll *pll)
>  	return ccs_pll_calculate(&client->dev, &lim, pll);
>  }
>  
> +static void
> +ccs_get_binning(struct ccs_sensor *sensor, u8 *binning_mode, u8 *binh, u8 *binv)
> +{
> +	struct v4l2_subdev_state *binner_state =
> +		v4l2_subdev_get_locked_active_state(&sensor->binner->sd);
> +	struct v4l2_rect *binner_sink_crop =
> +		v4l2_subdev_state_get_crop(binner_state, CCS_PAD_SINK,
> +					   CCS_STREAM_PIXEL);
> +	struct v4l2_rect *binner_sink_comp =
> +		v4l2_subdev_state_get_compose(binner_state, CCS_PAD_SINK,
> +					      CCS_STREAM_PIXEL);

I like this patch very much :-) The comments I made to 28/28 apply there
too.

> +
> +	if (binner_sink_crop->width == binner_sink_comp->width &&
> +	    binner_sink_crop->height == binner_sink_comp->height) {
> +		if (binning_mode)
> +			*binning_mode = 0;
> +
> +		if (binh)
> +			*binh = 1;
> +
> +		if (binv)
> +			*binv = 1;
> +
> +		return;
> +	}
> +
> +	if (binning_mode)
> +		*binning_mode = 1;
> +
> +	if (binh)
> +		*binh = binner_sink_crop->width / binner_sink_comp->width;
> +
> +	if (binv)
> +		*binv = binner_sink_crop->height / binner_sink_comp->height;

binh and binv are never NULL. How about the following ?

	if (binning_mode)
		*binning_mode = sink_crop->width != sink_comp->width ||
				sink_crop->height != sink_comp->height;

	*binh = sink_crop->width / sink_comp->width;
	*binv = sink_crop->height / sink_comp->height;

> +}
> +
>  static int ccs_pll_update(struct ccs_sensor *sensor)
>  {
>  	struct ccs_pll *pll = &sensor->pll;
> +	u8 binh, binv;
>  	int rval;
>  
> -	pll->binning_horizontal = sensor->binning_horizontal;
> -	pll->binning_vertical = sensor->binning_vertical;
> +	ccs_get_binning(sensor, NULL, &binh, &binv);
> +
> +	pll->binning_horizontal = binh;
> +	pll->binning_vertical = binv;
>  	pll->link_freq =
>  		sensor->link_freq->qmenu_int[sensor->link_freq->val];
>  	pll->scale_m = sensor->scale_m;
> @@ -1241,8 +1280,11 @@ static void ccs_update_blanking(struct ccs_sensor *sensor,
>  	struct v4l2_ctrl *hblank = sensor->hblank;
>  	u16 min_fll, max_fll, min_llp, max_llp, min_lbp;
>  	int min, max;
> +	u8 binh, binv;
> +
> +	ccs_get_binning(sensor, NULL, &binh, &binv);
>  
> -	if (sensor->binning_vertical > 1 || sensor->binning_horizontal > 1) {
> +	if (binv > 1 || binh > 1) {
>  		min_fll = CCS_LIM(sensor, MIN_FRAME_LENGTH_LINES_BIN);
>  		max_fll = CCS_LIM(sensor, MAX_FRAME_LENGTH_LINES_BIN);
>  		min_llp = CCS_LIM(sensor, MIN_LINE_LENGTH_PCK_BIN);
> @@ -1813,7 +1855,7 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
>  		v4l2_subdev_state_get_crop(src_state, CCS_PAD_SRC,
>  					   CCS_STREAM_PIXEL);
>  	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
> -	unsigned int binning_mode;
> +	u8 binning_mode, binh, binv;
>  	int rval;
>  
>  	if (pad != CCS_PAD_SRC)
> @@ -1835,19 +1877,12 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
>  		goto err_pm_put;
>  
>  	/* Binning configuration */
> -	if (sensor->binning_horizontal == 1 &&
> -	    sensor->binning_vertical == 1) {
> -		binning_mode = 0;
> -	} else {
> -		u8 binning_type =
> -			(sensor->binning_horizontal << 4)
> -			| sensor->binning_vertical;
> +	ccs_get_binning(sensor,	&binning_mode, &binh, &binv);
>  
> -		rval = ccs_write(sensor, BINNING_TYPE, binning_type);
> +	if (binning_mode) {
> +		rval = ccs_write(sensor, BINNING_TYPE, (binh << 4) | binv);
>  		if (rval < 0)
>  			goto err_pm_put;
> -
> -		binning_mode = 1;
>  	}
>  	rval = ccs_write(sensor, BINNING_MODE, binning_mode);
>  	if (rval < 0)
> @@ -2253,9 +2288,6 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
>  				sensor->scale_m = CCS_LIM(sensor, SCALER_N_MIN);
>  				sensor->scaling_mode =
>  					CCS_SCALING_MODE_NO_SCALING;
> -			} else if (ssd == sensor->binner) {
> -				sensor->binning_horizontal = 1;
> -				sensor->binning_vertical = 1;
>  			}
>  		}
>  		fallthrough;
> @@ -2529,10 +2561,6 @@ static void ccs_set_compose_binner(struct v4l2_subdev *subdev,
>  			best = this;
>  		}
>  	}
> -	if (sel->which == V4L2_SUBDEV_FORMAT_ACTIVE) {
> -		sensor->binning_vertical = binv;
> -		sensor->binning_horizontal = binh;
> -	}
>  
>  	sel->r.width = (sink_crop->width / binh) & ~1;
>  	sel->r.height = (sink_crop->height / binv) & ~1;
> @@ -3721,8 +3749,6 @@ static int ccs_probe(struct i2c_client *client)
>  				sensor->binning_subtypes[i].vertical);
>  		}
>  	}
> -	sensor->binning_horizontal = 1;
> -	sensor->binning_vertical = 1;
>  
>  	if (device_create_file(&client->dev, &dev_attr_ident) != 0) {
>  		dev_err(&client->dev, "sysfs ident entry creation failed\n");
> diff --git a/drivers/media/i2c/ccs/ccs.h b/drivers/media/i2c/ccs/ccs.h
> index a2a461cda338..e6fc00a9fa11 100644
> --- a/drivers/media/i2c/ccs/ccs.h
> +++ b/drivers/media/i2c/ccs/ccs.h
> @@ -237,9 +237,6 @@ struct ccs_sensor {
>  	u32 embedded_mbus_code;
>  	u8 emb_data_ctrl;
>  
> -	u8 binning_horizontal;
> -	u8 binning_vertical;
> -
>  	u8 scale_m;
>  	u8 scaling_mode;
>  

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 25/38] media: ccs: Remove ccs_get_crop_compose helper

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Wed, Mar 13, 2024 at 09:25:03AM +0200, Sakari Ailus wrote:
> As it's now easier to obtain the necessary information on crop and compose
> rectangles after moving to sub-device state, remove the
> ccs_get_crop_compose helper.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  drivers/media/i2c/ccs/ccs-core.c | 130 ++++++++++++++-----------------
>  1 file changed, 58 insertions(+), 72 deletions(-)
> 
> diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
> index 1f91af25eac6..6c7eb4f04070 100644
> --- a/drivers/media/i2c/ccs/ccs-core.c
> +++ b/drivers/media/i2c/ccs/ccs-core.c
> @@ -2222,24 +2222,6 @@ static int ccs_get_format(struct v4l2_subdev *subdev,
>  	return rval;
>  }
>  
> -static void ccs_get_crop_compose(struct v4l2_subdev *subdev,
> -				 struct v4l2_subdev_state *sd_state,
> -				 struct v4l2_rect **crops,
> -				 struct v4l2_rect **comps)
> -{
> -	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
> -	unsigned int i;
> -
> -	if (crops)
> -		for (i = 0; i < subdev->entity.num_pads; i++)
> -			crops[i] =
> -				v4l2_subdev_state_get_crop(sd_state, i,
> -							   CCS_STREAM_PIXEL);
> -	if (comps)
> -		*comps = v4l2_subdev_state_get_compose(sd_state, ssd->sink_pad,
> -						       CCS_STREAM_PIXEL);
> -}
> -
>  /* Changes require propagation only on sink pad. */
>  static void ccs_propagate(struct v4l2_subdev *subdev,
>  			  struct v4l2_subdev_state *sd_state, int which,
> @@ -2247,15 +2229,17 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
>  {
>  	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
>  	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
> -	struct v4l2_rect *comp, *crops[CCS_PADS];
> +	struct v4l2_rect *comp, *crop;
>  	struct v4l2_mbus_framefmt *fmt;
>  
> -	ccs_get_crop_compose(subdev, sd_state, crops, &comp);
> -
> +	comp = v4l2_subdev_state_get_compose(sd_state, ssd->sink_pad,
> +					     CCS_STREAM_PIXEL);
>  	switch (target) {
>  	case V4L2_SEL_TGT_CROP:
> -		comp->width = crops[CCS_PAD_SINK]->width;
> -		comp->height = crops[CCS_PAD_SINK]->height;
> +		crop = v4l2_subdev_state_get_crop(sd_state, CCS_PAD_SINK,
> +						  CCS_STREAM_PIXEL);
> +		comp->width = crop->width;
> +		comp->height = crop->height;
>  		if (which == V4L2_SUBDEV_FORMAT_ACTIVE) {
>  			if (ssd == sensor->scaler) {
>  				sensor->scale_m = CCS_LIM(sensor, SCALER_N_MIN);
> @@ -2269,13 +2253,15 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
>  		}
>  		fallthrough;
>  	case V4L2_SEL_TGT_COMPOSE:
> -		*crops[CCS_PAD_SRC] = *comp;
> +		crop = v4l2_subdev_state_get_crop(sd_state, CCS_PAD_SRC,
> +						  CCS_STREAM_PIXEL);
> +		*crop = *comp;
>  		fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC,
>  						   CCS_STREAM_PIXEL);
>  		fmt->width = comp->width;
>  		fmt->height = comp->height;
>  		if (which == V4L2_SUBDEV_FORMAT_ACTIVE && ssd == sensor->src)
> -			sensor->src_src = *crops[CCS_PAD_SRC];
> +			sensor->src_src = *crop;
>  		break;
>  	default:
>  		WARN_ON_ONCE(1);
> @@ -2416,7 +2402,7 @@ static int ccs_set_format(struct v4l2_subdev *subdev,
>  {
>  	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
>  	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
> -	struct v4l2_rect *crops[CCS_PADS];
> +	struct v4l2_rect *crop;
>  
>  	if (subdev == &sensor->src->sd && fmt->pad == CCS_PAD_META)
>  		return ccs_get_format(subdev, sd_state, fmt);
> @@ -2458,12 +2444,13 @@ static int ccs_set_format(struct v4l2_subdev *subdev,
>  		      CCS_LIM(sensor, MIN_Y_OUTPUT_SIZE),
>  		      CCS_LIM(sensor, MAX_Y_OUTPUT_SIZE));
>  
> -	ccs_get_crop_compose(subdev, sd_state, crops, NULL);
> +	crop = v4l2_subdev_state_get_crop(sd_state, ssd->sink_pad,
> +					  CCS_STREAM_PIXEL);
>  
> -	crops[ssd->sink_pad]->left = 0;
> -	crops[ssd->sink_pad]->top = 0;
> -	crops[ssd->sink_pad]->width = fmt->format.width;
> -	crops[ssd->sink_pad]->height = fmt->format.height;
> +	crop->left = 0;
> +	crop->top = 0;
> +	crop->width = fmt->format.width;
> +	crop->height = fmt->format.height;
>  	ccs_propagate(subdev, sd_state, fmt->which, V4L2_SEL_TGT_CROP);
>  
>  	mutex_unlock(&sensor->mutex);
> @@ -2518,24 +2505,23 @@ static int scaling_goodness(struct v4l2_subdev *subdev, int w, int ask_w,
>  static void ccs_set_compose_binner(struct v4l2_subdev *subdev,
>  				   struct v4l2_subdev_state *sd_state,
>  				   struct v4l2_subdev_selection *sel,
> -				   struct v4l2_rect **crops,
> +				   struct v4l2_rect *sink_crop,

const

>  				   struct v4l2_rect *comp)
>  {
>  	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
>  	unsigned int i;
>  	unsigned int binh = 1, binv = 1;
>  	int best = scaling_goodness(
> -		subdev,
> -		crops[CCS_PAD_SINK]->width, sel->r.width,
> -		crops[CCS_PAD_SINK]->height, sel->r.height, sel->flags);
> +		subdev,	sink_crop->width, sel->r.width,

s/\tsink_crop/ sink_crop/

> +		sink_crop->height, sel->r.height, sel->flags);

	int best = scaling_goodness(subdev, sink_crop->width, sel->r.width,
				    sink_crop->height, sel->r.height,
				    sel->flags);

>  
>  	for (i = 0; i < sensor->nbinning_subtypes; i++) {
>  		int this = scaling_goodness(
>  			subdev,
> -			crops[CCS_PAD_SINK]->width
> +			sink_crop->width
>  			/ sensor->binning_subtypes[i].horizontal,
>  			sel->r.width,
> -			crops[CCS_PAD_SINK]->height
> +			sink_crop->height
>  			/ sensor->binning_subtypes[i].vertical,
>  			sel->r.height, sel->flags);

You know this is horrible to read, right ? :-)

>  
> @@ -2550,8 +2536,8 @@ static void ccs_set_compose_binner(struct v4l2_subdev *subdev,
>  		sensor->binning_horizontal = binh;
>  	}
>  
> -	sel->r.width = (crops[CCS_PAD_SINK]->width / binh) & ~1;
> -	sel->r.height = (crops[CCS_PAD_SINK]->height / binv) & ~1;
> +	sel->r.width = (sink_crop->width / binh) & ~1;
> +	sel->r.height = (sink_crop->height / binv) & ~1;
>  }
>  
>  /*
> @@ -2566,7 +2552,7 @@ static void ccs_set_compose_binner(struct v4l2_subdev *subdev,
>  static void ccs_set_compose_scaler(struct v4l2_subdev *subdev,
>  				   struct v4l2_subdev_state *sd_state,
>  				   struct v4l2_subdev_selection *sel,
> -				   struct v4l2_rect **crops,
> +				   struct v4l2_rect *sink_crop,

const

>  				   struct v4l2_rect *comp)
>  {
>  	struct i2c_client *client = v4l2_get_subdevdata(subdev);
> @@ -2579,17 +2565,12 @@ static void ccs_set_compose_scaler(struct v4l2_subdev *subdev,
>  	unsigned int i;
>  	int best = INT_MIN;
>  
> -	sel->r.width = min_t(unsigned int, sel->r.width,
> -			     crops[CCS_PAD_SINK]->width);
> -	sel->r.height = min_t(unsigned int, sel->r.height,
> -			      crops[CCS_PAD_SINK]->height);
> -
> -	a = crops[CCS_PAD_SINK]->width
> -		* CCS_LIM(sensor, SCALER_N_MIN) / sel->r.width;
> -	b = crops[CCS_PAD_SINK]->height
> -		* CCS_LIM(sensor, SCALER_N_MIN) / sel->r.height;
> -	max_m = crops[CCS_PAD_SINK]->width
> -		* CCS_LIM(sensor, SCALER_N_MIN)
> +	sel->r.width = min_t(unsigned int, sel->r.width, sink_crop->width);
> +	sel->r.height = min_t(unsigned int, sel->r.height, sink_crop->height);
> +
> +	a = sink_crop->width * CCS_LIM(sensor, SCALER_N_MIN) / sel->r.width;
> +	b = sink_crop->height * CCS_LIM(sensor, SCALER_N_MIN) / sel->r.height;
> +	max_m = sink_crop->width * CCS_LIM(sensor, SCALER_N_MIN)
>  		/ CCS_LIM(sensor, MIN_X_OUTPUT_SIZE);
>  
>  	a = clamp(a, CCS_LIM(sensor, SCALER_M_MIN),
> @@ -2622,10 +2603,10 @@ static void ccs_set_compose_scaler(struct v4l2_subdev *subdev,
>  	for (i = 0; i < ntry; i++) {
>  		int this = scaling_goodness(
>  			subdev,
> -			crops[CCS_PAD_SINK]->width
> +			sink_crop->width
>  			/ try[i] * CCS_LIM(sensor, SCALER_N_MIN),
>  			sel->r.width,
> -			crops[CCS_PAD_SINK]->height,
> +			sink_crop->height,
>  			sel->r.height,
>  			sel->flags);
>  
> @@ -2642,12 +2623,10 @@ static void ccs_set_compose_scaler(struct v4l2_subdev *subdev,
>  			continue;
>  
>  		this = scaling_goodness(
> -			subdev, crops[CCS_PAD_SINK]->width
> -			/ try[i]
> +			subdev, sink_crop->width / try[i]
>  			* CCS_LIM(sensor, SCALER_N_MIN),
>  			sel->r.width,
> -			crops[CCS_PAD_SINK]->height
> -			/ try[i]
> +			sink_crop->height / try[i]
>  			* CCS_LIM(sensor, SCALER_N_MIN),
>  			sel->r.height,
>  			sel->flags);
> @@ -2660,17 +2639,15 @@ static void ccs_set_compose_scaler(struct v4l2_subdev *subdev,
>  	}
>  
>  	sel->r.width =
> -		(crops[CCS_PAD_SINK]->width
> -		 / scale_m
> +		(sink_crop->width / scale_m
>  		 * CCS_LIM(sensor, SCALER_N_MIN)) & ~1;

	sel->r.width = (sink_crop->width / scale_m
		* CCS_LIM(sensor, SCALER_N_MIN)) & ~1;

>  	if (mode == SMIAPP_SCALING_MODE_BOTH)
>  		sel->r.height =
> -			(crops[CCS_PAD_SINK]->height
> -			 / scale_m
> +			(sink_crop->height / scale_m
>  			 * CCS_LIM(sensor, SCALER_N_MIN))
>  			& ~1;
>  	else
> -		sel->r.height = crops[CCS_PAD_SINK]->height;
> +		sel->r.height = sink_crop->height;
>  
>  	if (sel->which == V4L2_SUBDEV_FORMAT_ACTIVE) {
>  		sensor->scale_m = scale_m;
> @@ -2684,17 +2661,20 @@ static int ccs_set_compose(struct v4l2_subdev *subdev,
>  {
>  	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
>  	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
> -	struct v4l2_rect *comp, *crops[CCS_PADS];
> +	struct v4l2_rect *comp, *sink_crop;

sink_crop can be const.

>  
> -	ccs_get_crop_compose(subdev, sd_state, crops, &comp);
> +	sink_crop = v4l2_subdev_state_get_crop(sd_state, CCS_PAD_SINK,
> +					       CCS_STREAM_PIXEL);
> +	comp = v4l2_subdev_state_get_compose(sd_state, ssd->sink_pad,
> +					     CCS_STREAM_PIXEL);
>  
>  	sel->r.top = 0;
>  	sel->r.left = 0;
>  
>  	if (ssd == sensor->binner)
> -		ccs_set_compose_binner(subdev, sd_state, sel, crops, comp);
> +		ccs_set_compose_binner(subdev, sd_state, sel, sink_crop, comp);
>  	else
> -		ccs_set_compose_scaler(subdev, sd_state, sel, crops, comp);
> +		ccs_set_compose_scaler(subdev, sd_state, sel, sink_crop, comp);
>  
>  	*comp = sel->r;
>  	ccs_propagate(subdev, sd_state, sel->which, V4L2_SEL_TGT_COMPOSE);
> @@ -2755,9 +2735,12 @@ static int ccs_set_crop(struct v4l2_subdev *subdev,
>  {
>  	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
>  	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
> -	struct v4l2_rect src_size = { 0 }, *crops[CCS_PADS], *comp;
> +	struct v4l2_rect src_size = { 0 }, *crop, *comp;

comp can be const.

>  
> -	ccs_get_crop_compose(subdev, sd_state, crops, &comp);
> +	crop = v4l2_subdev_state_get_crop(sd_state, sel->pad,
> +					  CCS_STREAM_PIXEL);
> +	comp = v4l2_subdev_state_get_compose(sd_state, ssd->sink_pad,
> +					     CCS_STREAM_PIXEL);
>  
>  	if (sel->pad == ssd->sink_pad) {
>  		struct v4l2_mbus_framefmt *mfmt =
> @@ -2781,7 +2764,7 @@ static int ccs_set_crop(struct v4l2_subdev *subdev,
>  	sel->r.left = min_t(int, sel->r.left, src_size.width - sel->r.width);
>  	sel->r.top = min_t(int, sel->r.top, src_size.height - sel->r.height);
>  
> -	*crops[sel->pad] = sel->r;
> +	*crop = sel->r;
>  
>  	if (ssd != sensor->pixel_array && sel->pad == CCS_PAD_SINK)
>  		ccs_propagate(subdev, sd_state, sel->which, V4L2_SEL_TGT_CROP);
> @@ -2806,14 +2789,17 @@ static int ccs_get_selection(struct v4l2_subdev *subdev,
>  {
>  	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
>  	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
> -	struct v4l2_rect *comp, *crops[CCS_PADS];
> +	struct v4l2_rect *crop, *comp;

const

>  	int ret;
>  
>  	ret = ccs_sel_supported(subdev, sel);
>  	if (ret)
>  		return ret;
>  
> -	ccs_get_crop_compose(subdev, sd_state, crops, &comp);
> +	crop = v4l2_subdev_state_get_crop(sd_state, sel->pad,
> +					  CCS_STREAM_PIXEL);
> +	comp = v4l2_subdev_state_get_compose(sd_state, ssd->sink_pad,
> +					     CCS_STREAM_PIXEL);
>  
>  	switch (sel->target) {
>  	case V4L2_SEL_TGT_CROP_BOUNDS:
> @@ -2835,7 +2821,7 @@ static int ccs_get_selection(struct v4l2_subdev *subdev,
>  		break;
>  	case V4L2_SEL_TGT_CROP:
>  	case V4L2_SEL_TGT_COMPOSE_BOUNDS:
> -		sel->r = *crops[sel->pad];
> +		sel->r = *crop;
>  		break;
>  	case V4L2_SEL_TGT_COMPOSE:
>  		sel->r = *comp;

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 03/38] media: uapi: Add generic serial metadata mbus formats

[Tomi Valkeinen]

On 20/03/2024 00:33, Laurent Pinchart wrote:
> On Tue, Mar 19, 2024 at 04:20:35PM +0200, Tomi Valkeinen wrote:
>> On 19/03/2024 15:27, Sakari Ailus wrote:
>>> On Thu, Mar 14, 2024 at 09:30:50AM +0200, Tomi Valkeinen wrote:
>>>> On 13/03/2024 09:24, Sakari Ailus wrote:
>>>>> Add generic serial metadata mbus formats. These formats describe data
>>>>> width and packing but not the content itself. The reason for specifying
>>>>> such formats is that the formats as such are fairly device specific but
>>>>> they are still handled by CSI-2 receiver drivers that should not be aware
>>>>> of device specific formats. What makes generic metadata formats possible
>>>>> is that these formats are parsed by software only, after capturing the
>>>>> data to system memory.
>>>>>
>>>>> Also add a definition for "Data unit" to cover what is essentially a pixel
>>>>> but is not image data.
>>>>
>>>> The CCS spec talks about legacy packing and optimized packing for 16+ bit
>>>> formats. You cover only the "legacy" ones here. Did you look at those?
>>>
>>> The reason is that the bus data layout of the new packing at higher bit
>>> depth matches with packing at lower bit depths (half to be precise). That's
>>> why there's no need to define formats for the new packing methods at higher
>>> bit depths (the driver simply uses the packing at half of the bit depth).
>>
>> Hmm. If we're capturing 10-bit raw format, say, with the width of 640
>> pixels, we'll configure the video stream format according to those. For
>> the embedded stream, we'll set it to V4L2_META_FMT_GENERIC_CSI2_10 and
>> 640 width, right?
>>
>> If we're capturing 20-bit raw, we'll configure the video stream format
>> again accordingly, width to 640, and 20 bit fourcc/mbus code. If the
>> embedded stream uses the "legacy" packing, we'll set the format to
>> V4L2_META_FMT_GENERIC_CSI2_20 with width of 640, right?
>>
>> But if it's using packed format for the embedded stream, we set the
>> format to V4L2_META_FMT_GENERIC_CSI2_10 and width to 1280?
>>
>> Considering that the video and (line-based) embedded data come from the
>> same source, I'd expect the widths to be the same.
> 
> I don't have a strong objection against multiplying the width, but we
> need to figure out the impact on other kernel space components, as well
> as on userspace. I suppose the media bus code for the embedded data
> stream on the sensor source pad when using optimized packing and
> capturing RAW20 images would be MEDIA_BUS_FMT_META_10 ? In that case I
> think the sensor driver should be able to handle the width calculations
> on its own, and the value would just be propagated by userspace.

Yes, I think it works. I just find it more logical if the widths of both 
the video and embedded streams are the same (which is the case for all 
other embedded formats).

However, even the CCS spec says "for RAW16, RAW20, and/or RAW24 Visible 
pixels, top-embedded data may instead be more optimally packed using 
RAW8, RAW10, and/or RAW12 pixels", so that's in line with what Sakari 
suggests. Although the spec specifically says "top-embedded", so does it 
mean that the optimized data is not allowed for bottom data?

  Tomi

Re: [PATCH v8 03/38] media: uapi: Add generic serial metadata mbus formats

[Sakari Ailus]

Moi,

On Fri, Mar 22, 2024 at 08:50:00AM +0200, Tomi Valkeinen wrote:
> On 20/03/2024 00:33, Laurent Pinchart wrote:
> > On Tue, Mar 19, 2024 at 04:20:35PM +0200, Tomi Valkeinen wrote:
> > > On 19/03/2024 15:27, Sakari Ailus wrote:
> > > > On Thu, Mar 14, 2024 at 09:30:50AM +0200, Tomi Valkeinen wrote:
> > > > > On 13/03/2024 09:24, Sakari Ailus wrote:
> > > > > > Add generic serial metadata mbus formats. These formats describe data
> > > > > > width and packing but not the content itself. The reason for specifying
> > > > > > such formats is that the formats as such are fairly device specific but
> > > > > > they are still handled by CSI-2 receiver drivers that should not be aware
> > > > > > of device specific formats. What makes generic metadata formats possible
> > > > > > is that these formats are parsed by software only, after capturing the
> > > > > > data to system memory.
> > > > > > 
> > > > > > Also add a definition for "Data unit" to cover what is essentially a pixel
> > > > > > but is not image data.
> > > > > 
> > > > > The CCS spec talks about legacy packing and optimized packing for 16+ bit
> > > > > formats. You cover only the "legacy" ones here. Did you look at those?
> > > > 
> > > > The reason is that the bus data layout of the new packing at higher bit
> > > > depth matches with packing at lower bit depths (half to be precise). That's
> > > > why there's no need to define formats for the new packing methods at higher
> > > > bit depths (the driver simply uses the packing at half of the bit depth).
> > > 
> > > Hmm. If we're capturing 10-bit raw format, say, with the width of 640
> > > pixels, we'll configure the video stream format according to those. For
> > > the embedded stream, we'll set it to V4L2_META_FMT_GENERIC_CSI2_10 and
> > > 640 width, right?
> > > 
> > > If we're capturing 20-bit raw, we'll configure the video stream format
> > > again accordingly, width to 640, and 20 bit fourcc/mbus code. If the
> > > embedded stream uses the "legacy" packing, we'll set the format to
> > > V4L2_META_FMT_GENERIC_CSI2_20 with width of 640, right?
> > > 
> > > But if it's using packed format for the embedded stream, we set the
> > > format to V4L2_META_FMT_GENERIC_CSI2_10 and width to 1280?
> > > 
> > > Considering that the video and (line-based) embedded data come from the
> > > same source, I'd expect the widths to be the same.
> > 
> > I don't have a strong objection against multiplying the width, but we
> > need to figure out the impact on other kernel space components, as well
> > as on userspace. I suppose the media bus code for the embedded data
> > stream on the sensor source pad when using optimized packing and
> > capturing RAW20 images would be MEDIA_BUS_FMT_META_10 ? In that case I
> > think the sensor driver should be able to handle the width calculations
> > on its own, and the value would just be propagated by userspace.
> 
> Yes, I think it works. I just find it more logical if the widths of both the
> video and embedded streams are the same (which is the case for all other
> embedded formats).
> 
> However, even the CCS spec says "for RAW16, RAW20, and/or RAW24 Visible
> pixels, top-embedded data may instead be more optimally packed using RAW8,
> RAW10, and/or RAW12 pixels", so that's in line with what Sakari suggests.
> Although the spec specifically says "top-embedded", so does it mean that the
> optimized data is not allowed for bottom data?

I haven't read the spec regarding this but I don't think there should be
anything that prevents it: it's the embedded data format. The CCS driver
patches only adds support for the top embedded data anyway.

-- 
Regards,

Sakari Ailus

Re: [PATCH v8 01/38] media: mc: Add INTERNAL pad flag

[Sakari Ailus]

Hi Laurent,

On Thu, Mar 21, 2024 at 07:20:32PM +0200, Laurent Pinchart wrote:
> On Wed, Mar 20, 2024 at 07:49:27AM +0000, Sakari Ailus wrote:
> > On Wed, Mar 20, 2024 at 12:17:07AM +0200, Laurent Pinchart wrote:
> > > On Thu, Mar 14, 2024 at 09:17:08AM +0200, Tomi Valkeinen wrote:
> > > > On 13/03/2024 09:24, Sakari Ailus wrote:
> > > > > Internal source pads will be used as routing endpoints in V4L2
> > > > > [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
> > > > > Internal source pads are pads that have both SINK and INTERNAL flags set.
> > > > > 
> > > > > Also prevent creating links to pads that have been flagged as internal and
> > > > > initialising SOURCE pads with INTERNAL flag set.
> > > > > 
> > > > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > > > ---
> > > > >   .../userspace-api/media/mediactl/media-types.rst       |  8 ++++++++
> > > > >   drivers/media/mc/mc-entity.c                           | 10 ++++++++--
> > > > >   include/uapi/linux/media.h                             |  1 +
> > > > >   3 files changed, 17 insertions(+), 2 deletions(-)
> > > > > 
> > > > > diff --git a/Documentation/userspace-api/media/mediactl/media-types.rst b/Documentation/userspace-api/media/mediactl/media-types.rst
> > > > > index 6332e8395263..f55ef055bcf8 100644
> > > > > --- a/Documentation/userspace-api/media/mediactl/media-types.rst
> > > > > +++ b/Documentation/userspace-api/media/mediactl/media-types.rst
> > > > > @@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
> > > > >   .. _MEDIA-PAD-FL-SINK:
> > > > >   .. _MEDIA-PAD-FL-SOURCE:
> > > > >   .. _MEDIA-PAD-FL-MUST-CONNECT:
> > > > > +.. _MEDIA-PAD-FL-INTERNAL:
> > > > >   
> > > > >   .. flat-table:: Media pad flags
> > > > >       :header-rows:  0
> > > > > @@ -381,6 +382,13 @@ Types and flags used to represent the media graph elements
> > > > >   	  enabled links even when this flag isn't set; the absence of the flag
> > > > >   	  doesn't imply there is none.
> > > > >   
> > > > > +    *  -  ``MEDIA_PAD_FL_INTERNAL``
> > > > > +       -  The internal flag indicates an internal pad that has no external
> > > > > +	  connections. Such a pad shall not be connected with a link.
> > > 
> > > I would expand this slightly, as it's the only source of documentation
> > > regarding internal pads.
> > 
> > Patch 9 adds more documentation, this patch is for MC only.
> 
> That's my point, it's the only source of documentation for internal
> pads from an MC point of view, so expanding the documentation would be
> good :-)

This is MC documentation, camera sensors aren't relevant in this context.

> 
> > > 
> > >        -  The internal flag indicates an internal pad that has no external
> > > 	  connections. This can be used to model, for instance, the pixel array
> > > 	  internal to an image sensor. As they are internal to entities,
> > > 	  internal pads shall not be connected with links.
> > 
> > I'd drop the sentence related to sensors.
> 
> I'm fine with another example, or a more generic explanation, but with
> that sentence dropped, I think this will leave the reader wondering what
> an internal pad is and what it's used for.

What we could and probably have here is that the internal sink pad
indicates a source of data. That's what it really is, whether that data is
image data or something else.

So I'd change this to:

The internal flag indicates an internal pad that has no external
connections. Such a pad shall not be connected with a link. The internal
pad flag is allowed only in conjunction with the sink pad flag. Together
the two flags indicate the pad is a source of data inside the entity.


> 
> > > > > +
> > > > > +	  The internal flag may currently be present only in a source pad where
> > > > 
> > > > s/source/sink/
> > > > 
> > > > Reviewed-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > > > > +	  it indicates that the :ref:``stream <media-glossary-stream>``
> > > > > +	  originates from within the entity.
> > > > >   
> > > > >   One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
> > > > >   must be set for every pad.
> > > > > diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> > > > > index 0e28b9a7936e..1973e9e1013e 100644
> > > > > --- a/drivers/media/mc/mc-entity.c
> > > > > +++ b/drivers/media/mc/mc-entity.c
> > > > > @@ -213,7 +213,9 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
> > > > >   		iter->index = i++;
> > > > >   
> > > > >   		if (hweight32(iter->flags & (MEDIA_PAD_FL_SINK |
> > > > > -					     MEDIA_PAD_FL_SOURCE)) != 1) {
> > > > > +					     MEDIA_PAD_FL_SOURCE)) != 1 ||
> > > > > +		    (iter->flags & MEDIA_PAD_FL_INTERNAL &&
> > > > > +		     !(iter->flags & MEDIA_PAD_FL_SINK))) {
> > > > >   			ret = -EINVAL;
> > > > >   			break;
> > > > >   		}
> > > 
> > > This may become more readable written as a switch statement:
> > > 
> > > 		const u32 pad_flags_mask = MEDIA_PAD_FL_SINK |
> > > 					   MEDIA_PAD_FL_SOURCE |
> > > 					   MEDIA_PAD_FL_INTERNAL;
> > > 
> > > 		switch (iter->flags & pad_flags_mask) {
> > > 		case MEDIA_PAD_FL_SINK:
> > > 		case MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL:
> > > 		case MEDIA_PAD_FL_SOURCE:
> > > 			break;
> > > 
> > > 		default:
> > > 			ret = -EINVAL;
> > > 			break;
> > > 		}
> > > 
> > > 		if (ret)
> > > 			break;
> > > 
> > > And now that I've written this, I'm not too sure anymore :-) Another
> > > option would be
> > > 
> > > 
> > > 	const u32 pad_flags = iter->flags & (MEDIA_PAD_FL_SINK |
> > > 					     MEDIA_PAD_FL_SOURCE |
> > > 					     MEDIA_PAD_FL_INTERNAL);
> > > 
> > > 	if (pad_flags != MEDIA_PAD_FL_SINK &&
> > > 	    pad_flags != MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL
> > > 	    pad_flags != MEDIA_PAD_FL_SOURCE) {
> > > 		ret = -EINVAL;
> > > 		break;
> > > 	}
> > > 
> > > Up to you.
> > 
> > I'd prefer to keep it as-is since the original check is testing two
> > independent things instead of merging them: that only either SINK or SOURCE
> > is set, and then separately that if INTERNAL is set, then SINK is set, too.
> > 
> > Of the two options you suggested, I prefer the latter.
> 
> I prefer the latter too, and I think it's more readable than the current
> code. If we later end up having to test for more rules, we can separate
> the checks.

I can switch to the latter.

> 
> > > 
> > > Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > 
> > Thank you!
> > 
> > > > > @@ -1112,7 +1114,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
> > > > >   
> > > > >   	for (i = 0; i < entity->num_pads; i++) {
> > > > >   		if ((entity->pads[i].flags &
> > > > > -		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
> > > > > +		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
> > > > > +		      MEDIA_PAD_FL_INTERNAL)) != pad_type)
> > > > >   			continue;
> > > > >   
> > > > >   		if (entity->pads[i].sig_type == sig_type)
> > > > > @@ -1142,6 +1145,9 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
> > > > >   		return -EINVAL;
> > > > >   	if (WARN_ON(!(sink->pads[sink_pad].flags & MEDIA_PAD_FL_SINK)))
> > > > >   		return -EINVAL;
> > > > > +	if (WARN_ON(source->pads[source_pad].flags & MEDIA_PAD_FL_INTERNAL) ||
> > > > > +	    WARN_ON(sink->pads[sink_pad].flags & MEDIA_PAD_FL_INTERNAL))
> > > > > +		return -EINVAL;
> > > > >   
> > > > >   	link = media_add_link(&source->links);
> > > > >   	if (link == NULL)
> > > > > diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
> > > > > index 1c80b1d6bbaf..80cfd12a43fc 100644
> > > > > --- a/include/uapi/linux/media.h
> > > > > +++ b/include/uapi/linux/media.h
> > > > > @@ -208,6 +208,7 @@ struct media_entity_desc {
> > > > >   #define MEDIA_PAD_FL_SINK			(1U << 0)
> > > > >   #define MEDIA_PAD_FL_SOURCE			(1U << 1)
> > > > >   #define MEDIA_PAD_FL_MUST_CONNECT		(1U << 2)
> > > > > +#define MEDIA_PAD_FL_INTERNAL			(1U << 3)
> > > > >   
> > > > >   struct media_pad_desc {
> > > > >   	__u32 entity;		/* entity ID */

-- 
Regards,

Sakari Ailus

Re: [PATCH v8 01/38] media: mc: Add INTERNAL pad flag

[Sakari Ailus]

Hi Laurent,

On Thu, Mar 28, 2024 at 09:47:26AM +0000, Sakari Ailus wrote:
> > > >        -  The internal flag indicates an internal pad that has no external
> > > > 	  connections. This can be used to model, for instance, the pixel array
> > > > 	  internal to an image sensor. As they are internal to entities,
> > > > 	  internal pads shall not be connected with links.
> > > 
> > > I'd drop the sentence related to sensors.
> > 
> > I'm fine with another example, or a more generic explanation, but with
> > that sentence dropped, I think this will leave the reader wondering what
> > an internal pad is and what it's used for.
> 
> What we could and probably have here is that the internal sink pad
> indicates a source of data. That's what it really is, whether that data is
> image data or something else.
> 
> So I'd change this to:
> 
> The internal flag indicates an internal pad that has no external
> connections. Such a pad shall not be connected with a link. The internal
> pad flag is allowed only in conjunction with the sink pad flag. Together
> the two flags indicate the pad is a source of data inside the entity.

A similar text already exists in the following paragraph already so I don't
think additions should be needed.

> 
> 
> > 
> > > > > > +
> > > > > > +	  The internal flag may currently be present only in a source pad where
> > > > > 
> > > > > s/source/sink/
> > > > > 
> > > > > Reviewed-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > > > > > +	  it indicates that the :ref:``stream <media-glossary-stream>``
> > > > > > +	  originates from within the entity.
> > > > > >   
> > > > > >   One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
> > > > > >   must be set for every pad.

-- 
Regards,

Sakari Ailus

Re: [PATCH v8 01/38] media: mc: Add INTERNAL pad flag

[Laurent Pinchart]

Hi Sakari,

On Thu, Mar 28, 2024 at 09:47:26AM +0000, Sakari Ailus wrote:
> On Thu, Mar 21, 2024 at 07:20:32PM +0200, Laurent Pinchart wrote:
> > On Wed, Mar 20, 2024 at 07:49:27AM +0000, Sakari Ailus wrote:
> > > On Wed, Mar 20, 2024 at 12:17:07AM +0200, Laurent Pinchart wrote:
> > > > On Thu, Mar 14, 2024 at 09:17:08AM +0200, Tomi Valkeinen wrote:
> > > > > On 13/03/2024 09:24, Sakari Ailus wrote:
> > > > > > Internal source pads will be used as routing endpoints in V4L2
> > > > > > [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
> > > > > > Internal source pads are pads that have both SINK and INTERNAL flags set.
> > > > > > 
> > > > > > Also prevent creating links to pads that have been flagged as internal and
> > > > > > initialising SOURCE pads with INTERNAL flag set.
> > > > > > 
> > > > > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > > > > ---
> > > > > >   .../userspace-api/media/mediactl/media-types.rst       |  8 ++++++++
> > > > > >   drivers/media/mc/mc-entity.c                           | 10 ++++++++--
> > > > > >   include/uapi/linux/media.h                             |  1 +
> > > > > >   3 files changed, 17 insertions(+), 2 deletions(-)
> > > > > > 
> > > > > > diff --git a/Documentation/userspace-api/media/mediactl/media-types.rst b/Documentation/userspace-api/media/mediactl/media-types.rst
> > > > > > index 6332e8395263..f55ef055bcf8 100644
> > > > > > --- a/Documentation/userspace-api/media/mediactl/media-types.rst
> > > > > > +++ b/Documentation/userspace-api/media/mediactl/media-types.rst
> > > > > > @@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
> > > > > >   .. _MEDIA-PAD-FL-SINK:
> > > > > >   .. _MEDIA-PAD-FL-SOURCE:
> > > > > >   .. _MEDIA-PAD-FL-MUST-CONNECT:
> > > > > > +.. _MEDIA-PAD-FL-INTERNAL:
> > > > > >   
> > > > > >   .. flat-table:: Media pad flags
> > > > > >       :header-rows:  0
> > > > > > @@ -381,6 +382,13 @@ Types and flags used to represent the media graph elements
> > > > > >   	  enabled links even when this flag isn't set; the absence of the flag
> > > > > >   	  doesn't imply there is none.
> > > > > >   
> > > > > > +    *  -  ``MEDIA_PAD_FL_INTERNAL``
> > > > > > +       -  The internal flag indicates an internal pad that has no external
> > > > > > +	  connections. Such a pad shall not be connected with a link.
> > > > 
> > > > I would expand this slightly, as it's the only source of documentation
> > > > regarding internal pads.
> > > 
> > > Patch 9 adds more documentation, this patch is for MC only.
> > 
> > That's my point, it's the only source of documentation for internal
> > pads from an MC point of view, so expanding the documentation would be
> > good :-)
> 
> This is MC documentation, camera sensors aren't relevant in this context.

They are only relevant in the same that they are one of the many
possible users of the MC API.

> > > > 
> > > >        -  The internal flag indicates an internal pad that has no external
> > > > 	  connections. This can be used to model, for instance, the pixel array
> > > > 	  internal to an image sensor. As they are internal to entities,
> > > > 	  internal pads shall not be connected with links.
> > > 
> > > I'd drop the sentence related to sensors.
> > 
> > I'm fine with another example, or a more generic explanation, but with
> > that sentence dropped, I think this will leave the reader wondering what
> > an internal pad is and what it's used for.
> 
> What we could and probably have here is that the internal sink pad
> indicates a source of data. That's what it really is, whether that data is
> image data or something else.
> 
> So I'd change this to:
> 
> The internal flag indicates an internal pad that has no external
> connections. Such a pad shall not be connected with a link. The internal
> pad flag is allowed only in conjunction with the sink pad flag. Together
> the two flags indicate the pad is a source of data inside the entity.

I think that's still difficult to understand. What's the issue with
mentioning a particular example to make the concept easier to understand
?

> > > > > > +
> > > > > > +	  The internal flag may currently be present only in a source pad where
> > > > > 
> > > > > s/source/sink/
> > > > > 
> > > > > Reviewed-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > > > > > +	  it indicates that the :ref:``stream <media-glossary-stream>``
> > > > > > +	  originates from within the entity.
> > > > > >   
> > > > > >   One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
> > > > > >   must be set for every pad.
> > > > > > diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> > > > > > index 0e28b9a7936e..1973e9e1013e 100644
> > > > > > --- a/drivers/media/mc/mc-entity.c
> > > > > > +++ b/drivers/media/mc/mc-entity.c
> > > > > > @@ -213,7 +213,9 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
> > > > > >   		iter->index = i++;
> > > > > >   
> > > > > >   		if (hweight32(iter->flags & (MEDIA_PAD_FL_SINK |
> > > > > > -					     MEDIA_PAD_FL_SOURCE)) != 1) {
> > > > > > +					     MEDIA_PAD_FL_SOURCE)) != 1 ||
> > > > > > +		    (iter->flags & MEDIA_PAD_FL_INTERNAL &&
> > > > > > +		     !(iter->flags & MEDIA_PAD_FL_SINK))) {
> > > > > >   			ret = -EINVAL;
> > > > > >   			break;
> > > > > >   		}
> > > > 
> > > > This may become more readable written as a switch statement:
> > > > 
> > > > 		const u32 pad_flags_mask = MEDIA_PAD_FL_SINK |
> > > > 					   MEDIA_PAD_FL_SOURCE |
> > > > 					   MEDIA_PAD_FL_INTERNAL;
> > > > 
> > > > 		switch (iter->flags & pad_flags_mask) {
> > > > 		case MEDIA_PAD_FL_SINK:
> > > > 		case MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL:
> > > > 		case MEDIA_PAD_FL_SOURCE:
> > > > 			break;
> > > > 
> > > > 		default:
> > > > 			ret = -EINVAL;
> > > > 			break;
> > > > 		}
> > > > 
> > > > 		if (ret)
> > > > 			break;
> > > > 
> > > > And now that I've written this, I'm not too sure anymore :-) Another
> > > > option would be
> > > > 
> > > > 
> > > > 	const u32 pad_flags = iter->flags & (MEDIA_PAD_FL_SINK |
> > > > 					     MEDIA_PAD_FL_SOURCE |
> > > > 					     MEDIA_PAD_FL_INTERNAL);
> > > > 
> > > > 	if (pad_flags != MEDIA_PAD_FL_SINK &&
> > > > 	    pad_flags != MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL
> > > > 	    pad_flags != MEDIA_PAD_FL_SOURCE) {
> > > > 		ret = -EINVAL;
> > > > 		break;
> > > > 	}
> > > > 
> > > > Up to you.
> > > 
> > > I'd prefer to keep it as-is since the original check is testing two
> > > independent things instead of merging them: that only either SINK or SOURCE
> > > is set, and then separately that if INTERNAL is set, then SINK is set, too.
> > > 
> > > Of the two options you suggested, I prefer the latter.
> > 
> > I prefer the latter too, and I think it's more readable than the current
> > code. If we later end up having to test for more rules, we can separate
> > the checks.
> 
> I can switch to the latter.
> 
> > > > 
> > > > Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > > 
> > > Thank you!
> > > 
> > > > > > @@ -1112,7 +1114,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
> > > > > >   
> > > > > >   	for (i = 0; i < entity->num_pads; i++) {
> > > > > >   		if ((entity->pads[i].flags &
> > > > > > -		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
> > > > > > +		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
> > > > > > +		      MEDIA_PAD_FL_INTERNAL)) != pad_type)
> > > > > >   			continue;
> > > > > >   
> > > > > >   		if (entity->pads[i].sig_type == sig_type)
> > > > > > @@ -1142,6 +1145,9 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
> > > > > >   		return -EINVAL;
> > > > > >   	if (WARN_ON(!(sink->pads[sink_pad].flags & MEDIA_PAD_FL_SINK)))
> > > > > >   		return -EINVAL;
> > > > > > +	if (WARN_ON(source->pads[source_pad].flags & MEDIA_PAD_FL_INTERNAL) ||
> > > > > > +	    WARN_ON(sink->pads[sink_pad].flags & MEDIA_PAD_FL_INTERNAL))
> > > > > > +		return -EINVAL;
> > > > > >   
> > > > > >   	link = media_add_link(&source->links);
> > > > > >   	if (link == NULL)
> > > > > > diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
> > > > > > index 1c80b1d6bbaf..80cfd12a43fc 100644
> > > > > > --- a/include/uapi/linux/media.h
> > > > > > +++ b/include/uapi/linux/media.h
> > > > > > @@ -208,6 +208,7 @@ struct media_entity_desc {
> > > > > >   #define MEDIA_PAD_FL_SINK			(1U << 0)
> > > > > >   #define MEDIA_PAD_FL_SOURCE			(1U << 1)
> > > > > >   #define MEDIA_PAD_FL_MUST_CONNECT		(1U << 2)
> > > > > > +#define MEDIA_PAD_FL_INTERNAL			(1U << 3)
> > > > > >   
> > > > > >   struct media_pad_desc {
> > > > > >   	__u32 entity;		/* entity ID */

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 17/38] media: v4l: subdev: Add trivial set_routing support

[Laurent Pinchart]

On Wed, Mar 20, 2024 at 03:55:58AM +0200, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.
> 
> On Wed, Mar 13, 2024 at 09:24:55AM +0200, Sakari Ailus wrote:
> > Add trivial S_ROUTING IOCTL support for drivers where routing is static.
> > Essentially this means returning the same information G_ROUTING call would
> > have done.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> 
> Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

Actually...

> > ---
> >  drivers/media/v4l2-core/v4l2-subdev.c | 14 ++++++++++++++
> >  1 file changed, 14 insertions(+)
> > 
> > diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> > index a6107e440ef0..c8c435df92c8 100644
> > --- a/drivers/media/v4l2-core/v4l2-subdev.c
> > +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> > @@ -930,6 +930,20 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
> >  
> >  		memset(routing->reserved, 0, sizeof(routing->reserved));
> >  
> > +		/*
> > +		 * If the driver doesn't support setting routing, just return
> > +		 * the routing table here.
> > +		 */
> > +		if (!v4l2_subdev_has_op(sd, pad, set_routing)) {
> > +			memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
> > +			       state->routing.routes,
> > +			       min(state->routing.num_routes, routing->len_routes) *
> > +			       sizeof(*state->routing.routes));
> > +			routing->num_routes = state->routing.num_routes;
> > +
> > +			return 0;
> > +		}
> > +

I think this should be done after the next code block that validates the
arguments. Otherwise the S_ROUTING ioctl will behave differently with
regard to blatantly invalid arguments, depending on whether or not the
subdev implements the .set_routing() operation. This can confuse
userspace, and does confuse v4l2-compliance.

I have the following change in my tree that fixes the problem, feel free
to squash it with this patch in v9.

commit 1e1f03eb8bc118c53a9deab05063d71a2fe7aa5f
Author: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Date:   Tue Apr 2 02:06:01 2024 +0300

    fixup! media: v4l: subdev: Add trivial set_routing support

    Validate arguments before handling the trivial set_routing support to
    expose a consistent behaviour to userspace. This fixes an issue with
    v4l2-compliance.

    Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index def91ab32c07..129867ed2bad 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -933,19 +933,12 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 		memset(routing->reserved, 0, sizeof(routing->reserved));

 		/*
-		 * If the driver doesn't support setting routing, just return
-		 * the routing table here.
+		 * Perform argument validation first, or subdevs that don't
+		 * support setting routing will not return an error when
+		 * arguments are blatantly wrong. The difference in behaviour
+		 * could be confusing for userspace, and in particular for API
+		 * compliance checkers.
 		 */
-		if (!v4l2_subdev_has_op(sd, pad, set_routing)) {
-			memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
-			       state->routing.routes,
-			       min(state->routing.num_routes, routing->len_routes) *
-			       sizeof(*state->routing.routes));
-			routing->num_routes = state->routing.num_routes;
-
-			return 0;
-		}
-
 		for (i = 0; i < routing->num_routes; ++i) {
 			const struct v4l2_subdev_route *route = &routes[i];
 			const struct media_pad *pads = sd->entity.pads;
@@ -969,6 +962,20 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 				return -EINVAL;
 		}

+		/*
+		 * If the driver doesn't support setting routing, just return
+		 * the routing table here.
+		 */
+		if (!v4l2_subdev_has_op(sd, pad, set_routing)) {
+			memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
+			       state->routing.routes,
+			       min(state->routing.num_routes, routing->len_routes) *
+			       sizeof(*state->routing.routes));
+			routing->num_routes = state->routing.num_routes;
+
+			return 0;
+		}
+
 		krouting.num_routes = routing->num_routes;
 		krouting.len_routes = routing->len_routes;
 		krouting.routes = routes;


> >  		for (i = 0; i < routing->num_routes; ++i) {
> >  			const struct v4l2_subdev_route *route = &routes[i];
> >  			const struct media_pad *pads = sd->entity.pads;

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 08/38] media: Documentation: Document embedded data guidelines for camera sensors

[Sakari Ailus]

Hi Laurent,

On Wed, Mar 20, 2024 at 02:03:00AM +0200, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.
> 
> On Wed, Mar 13, 2024 at 09:24:46AM +0200, Sakari Ailus wrote:
> > Document how embedded data support should be implemented for camera
> > sensors, and when and how CCS embedded data format should be referenced.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  .../media/drivers/camera-sensor.rst           | 29 +++++++++++++++++++
> >  1 file changed, 29 insertions(+)
> > 
> > diff --git a/Documentation/userspace-api/media/drivers/camera-sensor.rst b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> > index 919a50e8b9d9..92545538b855 100644
> > --- a/Documentation/userspace-api/media/drivers/camera-sensor.rst
> > +++ b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> > @@ -102,3 +102,32 @@ register programming sequences shall initialize the :ref:`V4L2_CID_HFLIP
> >  values programmed by the register sequences. The default values of these
> >  controls shall be 0 (disabled). Especially these controls shall not be inverted,
> >  independently of the sensor's mounting rotation.
> > +
> > +Embedded data
> > +-------------
> > +
> > +Many sensors, mostly raw sensors, support embedded data which is used to convey
> > +the sensor configuration for the captured frame back to the host. While CSI-2 is
> > +the most common bus used by such sensors, embedded data can be available on
> > +other bus types as well.
> > +
> > +Embedded data support shall use an internal source pad (a pad that has
> 
> s/source pad/sink pad/
> 
> Or do we call those "internal source pad" ?

As I wrote in the earlier e-mail, I'm quite unhappy with the "internal
source pad" term, mainly because of the conflict with the term itself and the
flags that indicate it. I'd just call them "internal sink pads" and then
explain they're actually a source of data.

> 
> As this is uAPI documentation, it is mainly targetting (in my opinion)
> userspace developers. I would write "Embedded data support uses ..." to
> describe the behaviour seen from userspace, instead of using "shall" to
> describe the requirements towards drivers.

Sounds good.

> 
> > +``MEDIA_PAD_FL_SINK <MEDIA-PAD-FL-SINK>`` and ``MEDIA_PAD_FL_INTERNAL
> > +<MEDIA-PAD-FL-INTERNAL>`` flags set) and route to the external pad. If embedded
> > +data output can be disabled in hardware, it should be possible to disable the
> > +embedded data route via ``VIDIOC_SUBDEV_S_ROUTING`` IOCTL.
> 
> You should mention the image pad here too:
> 
> Such sensors expose two internal sink pads (pads that have both the
> ``MEDIA_PAD_FL_SINK <MEDIA-PAD-FL-SINK>`` and ``MEDIA_PAD_FL_INTERNAL
> <MEDIA-PAD-FL-INTERNAL>`` flags set) to model the source of the image and
> embedded data streams. Each of those pads produces a single stream, and the

s/Each of those/Both of these/

> sub-device routes those streams to the external (source) pad. If embedded data
> output can be disabled in hardware, the sub-device allows disabling the
> embedded data route via the ``VIDIOC_SUBDEV_S_ROUTING`` IOCTL.

In the last sentence, we need to refer to the driver.

But generally I agree. I'll use:

"If the sub-device driver supports disabling embedded data, this can be
done by disabling the embedded data route via the
``VIDIOC_SUBDEV_S_ROUTING`` IOCTL."

> 
> > +
> > +In general, changing the embedded data format from the driver-configured values
> > +is not supported. The height of the metadata is hardware specific and the width
> 
> s/hardware specific/device-specific/

Yes.

> 
> > +is that (or less of that) of the image width, as configured on the pixel data
> > +stream.
> > +
> > +CCS and non-CCS embedded data
> > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > +
> > +Embedded data which is compliant with CCS definitions shall use ``CCS embedded
> > +data format <MEDIA-BUS-FMT-CCS-EMBEDDED>``. Device specific embedded data which
> 
> You should clarify here that you're talking about the internal sink pad.
> 
> s/Device specific/Only device-specific/
> 
> > +is compliant up to MIPI CCS embedded data levels 1 or 2 only shall refer to CCS
> 
> s/up to/with the/
> s/only shall/may/
> 
> > +embedded data formats and document the level of conformance. The rest of the
> > +device specific embedded data format shall be documented in the context of the
> 
> s/device specific/device-specific/
> 
> > +data format itself.
> 
> Not sure what you mean by the context in the last sentence.

This refers to CCS embedded data support levels which appears to be
documented in a later patch in the series ("media: uapi: ccs: Add media bus
code for MIPI CCS embedded data"). I'll add this paragraph after that patch.

The paragraph became finally:

Embedded data which is fully compliant with CCS definitions uses ``CCS embedded
data format <MEDIA-BUS-FMT-CCS-EMBEDDED>`` media bus code (level
3). Device-specific embedded data compliant with either MIPI CCS embedded data
levels 1 or 2 only shall not use CCS embedded data mbus code, but may refer to
CCS embedded data documentation with the level of conformance specified and omit
documenting these aspects of the format. The rest of the device-specific
embedded data format is documented in the context of the data format itself.

-- 
Regards,

Sakari Ailus

Re: [PATCH v8 09/38] media: Documentation: v4l: Document internal source pads

[Sakari Ailus]

Hi Laurent,

On Wed, Mar 20, 2024 at 02:26:25AM +0200, Laurent Pinchart wrote:
> On Tue, Mar 19, 2024 at 01:47:07PM +0000, Sakari Ailus wrote:
> > On Fri, Mar 15, 2024 at 04:32:59PM +0100, Julien Massot wrote:
> > > On 3/13/24 08:24, Sakari Ailus wrote:
> > > > Document internal source pads, pads that have both SINK and INTERNAL flags
> > > > set. Use the IMX219 camera sensor as an example.
> > > > 
> > > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > > ---
> > > >   .../userspace-api/media/v4l/dev-subdev.rst    | 145 ++++++++++++++++++
> > > >   1 file changed, 145 insertions(+)
> > > > 
> > > > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > > index a387e8a15b8d..1808f40f63e3 100644
> > > > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > > @@ -553,6 +553,27 @@ A stream at a specific point in the media pipeline is identified by the
> > > >   sub-device and a (pad, stream) pair. For sub-devices that do not support
> > > >   multiplexed streams the 'stream' field is always 0.
> > > > +.. _v4l2-subdev-internal-source-pads:
> > > > +
> > > > +Internal source pads and routing
> > > > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > > +
> > > > +Cases where a single sub-device source pad is traversed by multiple streams, one
> > > > +or more of which originate from within the sub-device itself, are special as
> > > > +there is no external sink pad for such routes. In those cases, the sources of
> > > > +the internally generated streams are represented by internal source pads, which
> > > > +are sink pads that have the :ref:`MEDIA_PAD_FL_INTERNAL <MEDIA-PAD-FL-INTERNAL>`
> > > > +pad flag set.
> 
> This seems to answer a question in a previous patch, where I wondered if
> you meant "internal sink pad" instead of "internal source pad".
> 
> Given that this is will most likely be a source of confusion, I think
> you should explain this very clearly.
> 
> .. note::
> 
>    Internal source pads model the source of data *within* a sub-device, and are
>    therefore conceptually located on the "sink" side of the sub-device from the
>    point of view of sub-device internal routes. For this reason, and despite
>    their name, they use the :ref:`MEDIA_PAD_FL_SINK <MEDIA-PAD-FL-SINK>` flag.
>    When used in the routing API, they are addressed through the ``sink_pad``
>    and ``sink_stream`` fields of the :c:type:`v4l2_subdev_route` structure.
> 
> I hope this will be enough... We'll need to be extra careful during
> review in the beginning, making sure to always use the right terms.

Let's see if we need something like this with the "internal source pad"
concept removed. But let's revisit the topic in v9.

> 
> > > > +
> > > > +Internal pads have all the properties of an external pad, including formats and
> > > > +selections. The format in this case is the source format of the stream. An
> > > > +internal pad always has a single stream only (0).
> > > > +
> > > > +Routes from an internal source pad to an external source pad are typically not
> > > > +modifiable but they can be activated and deactivated using the
> > > > +:ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE <v4l2-subdev-routing-flags>` flag, depending
> > > > +on driver capabilities.
> > > > +
> > > >   Interaction between routes, streams, formats and selections
> > > >   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > > @@ -668,3 +689,127 @@ To configure this pipeline, the userspace must take the following steps:
> > > >      the configurations along the stream towards the receiver, using
> > > >      :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each
> > > >      stream endpoint in each sub-device.
> > > > +
> > > > +Internal pads setup example
> > > > +^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > > +
> > > > +A simple example of a multiplexed stream setup might be as follows:
> > > > +
> > > > +- An IMX219 camera sensor source sub-device, with one sink pad (0), one source pad
> > > > +  (1), an internal sink pad (2) that represents the source of embedded
> > > 
> > > The pixel pad is an internal pad as well ?
> > 
> > How about:
> > 
> > - An IMX219 camera sensor source sub-device, with one internal sink pad
> >   (0) for image data, one source pad (1), an internal sink pad (2) that
> >   represents the source of embedded
> 
> Sounds better, except you're now talking about internal sink pads, not
> internal source pads :-) We have to pick one option and stick to it.
> Same below.

Let's discuss this in the context of patch 8.

> 
> > > > +  data. There are two routes, one from the sink pad to the source, and another
> 
> "from the internal image source pad to the source" sounds weird, so you
> may need to write "from the internal image source pad to the external
> source pad". If we instead talked about "internal sink pads" to model
> the source of the streams, you could write "from the internal image sink
> pad to the source pad".

I think this text was from the original CCS example. I'll update it for
IMX290.

> 
> > > > +  from the internal sink pad to the source pad. Both streams are always active,
> > > > +  i.e. there is no need to separately enable the embedded data stream. The
> > > > +  sensor uses the CSI-2 bus.
> > > > +
> > > > +- A CSI-2 receiver in the SoC (Receiver). The receiver has a single sink pad
> > > > +  (pad 0), connected to the bridge, and two source pads (pads 1-2), going to the
> 
> I think you meant s/bridge/sensor/

Thanks. In an earlier version there was a bridge, this was a leftover from
that it seems. I'll check if there are other remaining references, too.

> 
> > > > +  DMA engine. The receiver demultiplexes the incoming streams to the source
> 
> And "DMA engines" here ?

Correct.

> 
> > > > +  pads.
> > > > +
> > > > +- DMA Engines in the SoC (DMA Engine), one for each stream. Each DMA engine is
> 
> s/Engines/engine/

Sounds good.

> 
> > > > +  connected to a single source pad in the receiver.
> 
> s/in the/of the/

Works for me.

> 
> > > > +
> > > > +The sensor, the bridge and the receiver are modeled as V4L2 sub-devices,
> 
> What bridge ? Same below.
> 
> > > > +exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are
> > > > +modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
> > > > +
> > > > +To configure this pipeline, the userspace must take the following steps:
> > > > +
> > > > +1) Set up media links between entities: connect the sensors to the bridge,
> > > > +   bridge to the receiver, and the receiver to the DMA engines. This step does
> > > > +   not differ from normal non-multiplexed media controller setup.
> > > > +
> > > > +2) Configure routing
> > > > +
> > > > +.. flat-table:: Camera sensor. There are no configurable routes.
> > > > +    :header-rows: 1
> > > > +
> > > > +    * - Sink Pad/Stream
> > > > +      - Source Pad/Stream
> > > > +      - Routing Flags
> > > > +      - Comments
> > > > +    * - 0/0
> > > > +      - 1/0
> > >
> > > - 1/0
> > > - 0/0
> > >
> > > > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > > > +      - Pixel data stream from the sink pad
> > > > +    * - 2/0
> > > > +      - 1/1
> > >
> > > - 0/1
> > >
> > > > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > > > +      - Metadata stream from the internal sink pad
> > > 
> > > In latest patch "[PATCH v6 05/15] media: i2c: imx219: Add embedded data
> > > support"
> > > we have:
> > > - 0 -> source pad
> > > - 1 -> pixel/image
> > > - 2 -> EDATA
> > > 
> > > This is also what you did for ov2740.
> > 
> > Yes, these were leftovers from the CCS example. I've fixed them.
> > 
> > > With that fixed:
> > > Reviewed-by Julien Massot <julien.massot@collabora.com>
> > 
> > Thank you!
> 

-- 
Kind regards,

Sakari Ailus

Re: [PATCH v8 36/38] media: v4l: Add V4L2_SUBDEV_ROUTE_FL_IMMUTABLE sub-device routing flag

[Sakari Ailus]

Hi Laurent,

On Thu, Mar 21, 2024 at 07:03:08PM +0200, Laurent Pinchart wrote:
> Hello,
> 
> On Wed, Mar 13, 2024 at 07:39:17AM +0000, Sakari Ailus wrote:
> > On Wed, Mar 13, 2024 at 09:34:13AM +0200, Tomi Valkeinen wrote:
> > > On 13/03/2024 09:25, Sakari Ailus wrote:
> > > > Add a flag to denote immutable routes, V4L2_SUBDEV_ROUTE_FL_IMMUTABLE.
> > > > Such routes cannot be changed and they're always active.
> > > > 
> > > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > > ---
> > > >   Documentation/userspace-api/media/v4l/dev-subdev.rst         | 3 ++-
> > > >   .../userspace-api/media/v4l/vidioc-subdev-g-routing.rst      | 5 +++++
> > > >   include/uapi/linux/v4l2-subdev.h                             | 5 +++++
> > > >   3 files changed, 12 insertions(+), 1 deletion(-)
> > > > 
> > > > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > > index 08495cc6f4a6..2f2423f676cf 100644
> > > > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > > @@ -572,7 +572,8 @@ internal pad always has a single stream only (0).
> > > >   Routes from an internal source pad to an external source pad are typically not
> > > >   modifiable but they can be activated and deactivated using the
> > > >   :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE <v4l2-subdev-routing-flags>` flag, depending
> > > > -on driver capabilities.
> > > > +on driver capabilities. This capatibility is indicated by the
> > > > +:ref:`V4L2_SUBDEV_ROUTE_FL_IMMUTABLE <v4l2-subdev-routing-flags>` flag.
> 
> That's not very clear, it sounds like V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
> indicates that the route can be enabled/disabled. I'd rewrite this.

I'll use instead:

The :ref:`V4L2_SUBDEV_ROUTE_FL_IMMUTABLE <v4l2-subdev-routing-flags>` flag
indicates that the ``V4L2_SUBDEV_ROUTE_FLAG_ACTIVE`` of the route may not
be unset.

> 
> > > >   Interaction between routes, streams, formats and selections
> > > >   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > > diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > > > index 08b8d17cef3f..cd7735f9104e 100644
> > > > --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > > > +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > > > @@ -139,6 +139,11 @@ Also ``VIDIOC_SUBDEV_S_ROUTING`` may return more route than the user provided in
> > > >       * - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > > >         - 0x0001
> > > >         - The route is enabled. Set by applications.
> > > > +    * - V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
> > > > +      - 0x0002
> > > > +      - The route is immutable. Set by the driver. The
> > > > +	``V4L2_SUBDEV_ROUTE_FL_ACTIVE`` flag of an immutable route may not be
> > > > +	changed.
> 
> May not be changed and will always be set ?

How about "may not be unset"?

> 
> > > >   Return Value
> > > >   ============
> > > > diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
> > > > index ca543982460c..7e501cb45e4e 100644
> > > > --- a/include/uapi/linux/v4l2-subdev.h
> > > > +++ b/include/uapi/linux/v4l2-subdev.h
> > > > @@ -200,6 +200,11 @@ struct v4l2_subdev_capability {
> > > >    * on a video node.
> > > >    */
> > > >   #define V4L2_SUBDEV_ROUTE_FL_ACTIVE		(1U << 0)
> > > > +/*
> > > > + * Is the route immutable. The ACTIVE flag of an immutable route may not be
> > > > + * changed.
> > > > + */
> > > > +#define V4L2_SUBDEV_ROUTE_FL_IMMUTABLE		(1U << 1)
> > > >   /**
> > > >    * struct v4l2_subdev_route - A route inside a subdev
> > > 
> > > Is the route fully immutable? The sink/source stream ID cannot be changed
> > > (or any new fields we might come up with in the future)?
> > 
> > I think the new fields should be considered separately when they're added.
> > This also applies to the stream IDs, I'll add this to the documentation.
> > 
> > The naming of the flag is aligned with MC link flag with a similar purpose.
> > 
> > > Hmm, or would a route with different stream IDs be a, well, different
> > > route...
> > > 
> > > The docs here only talk about the ACTIVE flag. Would
> > > V4L2_SUBDEV_ROUTE_FL_ALWAYS_ACTIVE be a better name, to be more explicit on
> > > the meaning?
> > 
> > I prefer immutable. I wonder what Laurent and Hans think.
> 
> I'm fine with either. IMMUTABLE is shorter, if that makes a difference.
> 

-- 
Kind regards,

Sakari Ailus

Re: [PATCH v8 36/38] media: v4l: Add V4L2_SUBDEV_ROUTE_FL_IMMUTABLE sub-device routing flag

[Laurent Pinchart]

Hi Sakari,

On Tue, Apr 09, 2024 at 01:21:16PM +0000, Sakari Ailus wrote:
> On Thu, Mar 21, 2024 at 07:03:08PM +0200, Laurent Pinchart wrote:
> > On Wed, Mar 13, 2024 at 07:39:17AM +0000, Sakari Ailus wrote:
> > > On Wed, Mar 13, 2024 at 09:34:13AM +0200, Tomi Valkeinen wrote:
> > > > On 13/03/2024 09:25, Sakari Ailus wrote:
> > > > > Add a flag to denote immutable routes, V4L2_SUBDEV_ROUTE_FL_IMMUTABLE.
> > > > > Such routes cannot be changed and they're always active.
> > > > > 
> > > > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > > > ---
> > > > >   Documentation/userspace-api/media/v4l/dev-subdev.rst         | 3 ++-
> > > > >   .../userspace-api/media/v4l/vidioc-subdev-g-routing.rst      | 5 +++++
> > > > >   include/uapi/linux/v4l2-subdev.h                             | 5 +++++
> > > > >   3 files changed, 12 insertions(+), 1 deletion(-)
> > > > > 
> > > > > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > > > index 08495cc6f4a6..2f2423f676cf 100644
> > > > > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > > > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > > > @@ -572,7 +572,8 @@ internal pad always has a single stream only (0).
> > > > >   Routes from an internal source pad to an external source pad are typically not
> > > > >   modifiable but they can be activated and deactivated using the
> > > > >   :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE <v4l2-subdev-routing-flags>` flag, depending
> > > > > -on driver capabilities.
> > > > > +on driver capabilities. This capatibility is indicated by the
> > > > > +:ref:`V4L2_SUBDEV_ROUTE_FL_IMMUTABLE <v4l2-subdev-routing-flags>` flag.
> > 
> > That's not very clear, it sounds like V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
> > indicates that the route can be enabled/disabled. I'd rewrite this.
> 
> I'll use instead:
> 
> The :ref:`V4L2_SUBDEV_ROUTE_FL_IMMUTABLE <v4l2-subdev-routing-flags>` flag
> indicates that the ``V4L2_SUBDEV_ROUTE_FLAG_ACTIVE`` of the route may not
> be unset.

Sounds good to me.

> > > > >   Interaction between routes, streams, formats and selections
> > > > >   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > > > diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > > > > index 08b8d17cef3f..cd7735f9104e 100644
> > > > > --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > > > > +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > > > > @@ -139,6 +139,11 @@ Also ``VIDIOC_SUBDEV_S_ROUTING`` may return more route than the user provided in
> > > > >       * - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > > > >         - 0x0001
> > > > >         - The route is enabled. Set by applications.
> > > > > +    * - V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
> > > > > +      - 0x0002
> > > > > +      - The route is immutable. Set by the driver. The
> > > > > +	``V4L2_SUBDEV_ROUTE_FL_ACTIVE`` flag of an immutable route may not be
> > > > > +	changed.
> > 
> > May not be changed and will always be set ?
> 
> How about "may not be unset"?

Even better :-)

> > > > >   Return Value
> > > > >   ============
> > > > > diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
> > > > > index ca543982460c..7e501cb45e4e 100644
> > > > > --- a/include/uapi/linux/v4l2-subdev.h
> > > > > +++ b/include/uapi/linux/v4l2-subdev.h
> > > > > @@ -200,6 +200,11 @@ struct v4l2_subdev_capability {
> > > > >    * on a video node.
> > > > >    */
> > > > >   #define V4L2_SUBDEV_ROUTE_FL_ACTIVE		(1U << 0)
> > > > > +/*
> > > > > + * Is the route immutable. The ACTIVE flag of an immutable route may not be
> > > > > + * changed.
> > > > > + */
> > > > > +#define V4L2_SUBDEV_ROUTE_FL_IMMUTABLE		(1U << 1)
> > > > >   /**
> > > > >    * struct v4l2_subdev_route - A route inside a subdev
> > > > 
> > > > Is the route fully immutable? The sink/source stream ID cannot be changed
> > > > (or any new fields we might come up with in the future)?
> > > 
> > > I think the new fields should be considered separately when they're added.
> > > This also applies to the stream IDs, I'll add this to the documentation.
> > > 
> > > The naming of the flag is aligned with MC link flag with a similar purpose.
> > > 
> > > > Hmm, or would a route with different stream IDs be a, well, different
> > > > route...
> > > > 
> > > > The docs here only talk about the ACTIVE flag. Would
> > > > V4L2_SUBDEV_ROUTE_FL_ALWAYS_ACTIVE be a better name, to be more explicit on
> > > > the meaning?
> > > 
> > > I prefer immutable. I wonder what Laurent and Hans think.
> > 
> > I'm fine with either. IMMUTABLE is shorter, if that makes a difference.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 35/38] media: ov2740: Add support for embedded data

[Sakari Ailus]

Hi Laurent,

Thanks for the review.

On Thu, Mar 21, 2024 at 07:16:08PM +0200, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.
> 
> On Wed, Mar 13, 2024 at 09:25:13AM +0200, Sakari Ailus wrote:
> > Add support for embedded data. This introduces two internal pads for pixel
> > and embedded data streams. As the driver supports a single mode only,
> > there's no need for backward compatibility in mode selection.
> > 
> > The embedded data is configured to be placed before the image data whereas
> > after the image data is the default.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  drivers/media/i2c/ov2740.c | 150 +++++++++++++++++++++++++++++++++----
> >  1 file changed, 137 insertions(+), 13 deletions(-)
> > 
> > diff --git a/drivers/media/i2c/ov2740.c b/drivers/media/i2c/ov2740.c
> > index df57f0096e98..7488b2535071 100644
> > --- a/drivers/media/i2c/ov2740.c
> > +++ b/drivers/media/i2c/ov2740.c
> > @@ -11,6 +11,7 @@
> >  #include <linux/pm_runtime.h>
> >  #include <linux/nvmem-provider.h>
> >  #include <linux/regmap.h>
> > +#include <media/mipi-csi2.h>
> >  #include <media/v4l2-ctrls.h>
> >  #include <media/v4l2-device.h>
> >  #include <media/v4l2-fwnode.h>
> > @@ -71,11 +72,31 @@
> >  #define OV2740_REG_ISP_CTRL00		0x5000
> >  /* ISP CTRL01 */
> >  #define OV2740_REG_ISP_CTRL01		0x5001
> > +
> > +/* Embedded data line location control */
> > +#define OV2740_REG_EMBEDDED_FLAG	0x5a08
> > +#define OV2740_EMBEDDED_FLAG_FOOTER	BIT(2) /* otherwise it's in header */
> > +#define OV2740_EMBEDDED_FLAG_MYSTERY	BIT(1)
> >  /* Customer Addresses: 0x7010 - 0x710F */
> >  #define CUSTOMER_USE_OTP_SIZE		0x100
> >  /* OTP registers from sensor */
> >  #define OV2740_REG_OTP_CUSTOMER		0x7010
> >  
> > +enum {
> > +	OV2740_PAD_SOURCE,
> > +	OV2740_PAD_PIXEL,
> > +	OV2740_PAD_META,
> > +	OV2740_NUM_PADS,
> > +};
> > +
> > +enum {
> > +	OV2740_STREAM_PIXEL,
> > +	OV2740_STREAM_META,
> > +};
> > +
> > +#define OV2740_META_WIDTH		100U /* 97 bytes of actual data */
> > +#define OV2740_META_HEIGHT		1U
> > +
> >  struct nvm_data {
> >  	struct nvmem_device *nvmem;
> >  	struct regmap *regmap;
> > @@ -513,7 +534,7 @@ static const struct ov2740_mode supported_modes_180mhz[] = {
> >  
> >  struct ov2740 {
> >  	struct v4l2_subdev sd;
> > -	struct media_pad pad;
> > +	struct media_pad pads[OV2740_NUM_PADS];
> >  	struct v4l2_ctrl_handler ctrl_handler;
> >  
> >  	/* V4L2 Controls */
> > @@ -976,6 +997,11 @@ static int ov2740_enable_streams(struct v4l2_subdev *sd,
> >  	if (ret)
> >  		goto out_pm_put;
> >  
> > +	ret = ov2740_write_reg(ov2740, OV2740_REG_EMBEDDED_FLAG, 1,
> > +			       OV2740_EMBEDDED_FLAG_MYSTERY);
> > +	if (ret)
> > +		return ret;
> > +
> >  	ret = ov2740_write_reg(ov2740, OV2740_REG_MODE_SELECT, 1,
> >  			       OV2740_MODE_STREAMING);
> >  	if (ret) {
> > @@ -1013,23 +1039,49 @@ static int ov2740_disable_streams(struct v4l2_subdev *sd,
> >  	return ret;
> >  }
> >  
> > -static int ov2740_set_format(struct v4l2_subdev *sd,
> > -			     struct v4l2_subdev_state *sd_state,
> > -			     struct v4l2_subdev_format *fmt)
> > +static int __ov2740_set_format(struct v4l2_subdev *sd,
> > +			       struct v4l2_subdev_state *sd_state,
> > +			       struct v4l2_mbus_framefmt *format,
> > +			       enum v4l2_subdev_format_whence which,
> > +			       unsigned int pad, unsigned int stream)
> >  {
> > +	struct v4l2_mbus_framefmt *src_pix_fmt, *src_meta_fmt, *pix_fmt,
> > +		*meta_fmt;
> >  	struct ov2740 *ov2740 = to_ov2740(sd);
> >  	const struct ov2740_mode *mode;
> >  	s32 vblank_def, h_blank;
> >  
> > +	/*
> > +	 * Allow setting format on internal pixel pad as well as the source
> > +	 * pad's pixel stream (for compatibility).
> 
> The internal image pad represents the pixel array, it should thus report
> the full pixel array size, and be unaffected by the selected mode.

Yes, I also realised this later on. I think we'll need more time to define
a proper API for sensors and thus postpone the sensor patches. :-I But I
don't see a reason to delay the rest.

> 
> > +	 */
> > +	if (pad == OV2740_PAD_SOURCE || pad == OV2740_PAD_META ||
> > +	    stream == OV2740_STREAM_META) {
> 
> As Julien pointed out, this isn't right.

I've fixed that for the next version.

> 
> > +		*format = *v4l2_subdev_state_get_format(sd_state, pad, stream);
> > +		return 0;
> > +	}
> > +
> > +	pix_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_PIXEL, 0);
> > +	meta_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_META, 0);
> > +	src_pix_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_SOURCE,
> > +						   OV2740_STREAM_PIXEL);
> > +	src_meta_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_SOURCE,
> > +						    OV2740_STREAM_META);
> > +
> >  	mode = v4l2_find_nearest_size(ov2740->supported_modes,
> >  				      ov2740->supported_modes_count,
> >  				      width, height,
> > -				      fmt->format.width, fmt->format.height);
> > +				      format->width, format->height);
> > +	ov2740_update_pad_format(mode, pix_fmt);
> > +	*format = *src_pix_fmt = *pix_fmt;
> >  
> > -	ov2740_update_pad_format(mode, &fmt->format);
> > -	*v4l2_subdev_state_get_format(sd_state, fmt->pad) = fmt->format;
> > +	meta_fmt->code = MEDIA_BUS_FMT_OV2740_EMBEDDED;
> > +	meta_fmt->width = OV2740_META_WIDTH;
> > +	meta_fmt->height = OV2740_META_HEIGHT;
> > +	*src_meta_fmt = *meta_fmt;
> > +	src_meta_fmt->code = MEDIA_BUS_FMT_META_10;
> >  
> > -	if (fmt->which == V4L2_SUBDEV_FORMAT_TRY)
> > +	if (which == V4L2_SUBDEV_FORMAT_TRY)
> >  		return 0;
> >  
> >  	ov2740->cur_mode = mode;
> > @@ -1049,6 +1101,14 @@ static int ov2740_set_format(struct v4l2_subdev *sd,
> >  	return 0;
> >  }
> >  
> > +static int ov2740_set_format(struct v4l2_subdev *sd,
> > +			     struct v4l2_subdev_state *sd_state,
> > +			     struct v4l2_subdev_format *fmt)
> > +{
> > +	return __ov2740_set_format(sd, sd_state, &fmt->format, fmt->which,
> > +				   fmt->pad, fmt->stream);
> > +}
> > +
> >  static int ov2740_enum_mbus_code(struct v4l2_subdev *sd,
> >  				 struct v4l2_subdev_state *sd_state,
> >  				 struct v4l2_subdev_mbus_code_enum *code)
> 
> You need to update this too, to enumerate the correct format on the
> different pads and streams.
> 
> > @@ -1085,10 +1145,68 @@ static int ov2740_enum_frame_size(struct v4l2_subdev *sd,
> >  static int ov2740_init_state(struct v4l2_subdev *sd,
> >  			     struct v4l2_subdev_state *sd_state)
> >  {
> > +	struct v4l2_subdev_route routes[] = {
> > +		{
> > +			.sink_pad = OV2740_PAD_PIXEL,
> > +			.source_pad = OV2740_PAD_SOURCE,
> > +			.source_stream = OV2740_STREAM_PIXEL,
> > +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> > +		}, {
> > +			.sink_pad = OV2740_PAD_META,
> > +			.source_pad = OV2740_PAD_SOURCE,
> > +			.source_stream = OV2740_STREAM_META,
> > +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> > +		},
> > +	};
> > +	struct v4l2_subdev_krouting routing = {
> > +		.routes = routes,
> > +		.num_routes = ARRAY_SIZE(routes),
> > +	};
> > +	struct v4l2_subdev_state *active_state;
> > +	struct v4l2_mbus_framefmt format = { 0 };
> >  	struct ov2740 *ov2740 = to_ov2740(sd);
> > +	int ret;
> > +
> > +	ret = v4l2_subdev_set_routing(sd, sd_state, &routing);
> > +	if (ret)
> > +		return ret;
> > +
> > +	active_state = v4l2_subdev_get_locked_active_state(sd);
> 
> There's a lockdep assertion that will trip when initializing any try
> state.

Uh-oh.

> 
> > +
> > +	ov2740_update_pad_format(&ov2740->supported_modes[0], &format);
> > +
> > +	return __ov2740_set_format(sd, sd_state, &format,
> > +				   active_state == sd_state ?
> > +				   V4L2_SUBDEV_FORMAT_ACTIVE :
> > +				   V4L2_SUBDEV_FORMAT_TRY, OV2740_PAD_PIXEL, 0);
> 
> Move the code specific to the active state from __ov2740_set_format() to
> ov2740_set_format(), and drop the which parameter to
> __ov2740_set_format(). You'll simplify the code here.

I'll see how this would work for v9.

> 
> > +}
> > +
> > +static int ov2740_get_frame_desc(struct v4l2_subdev *sd, unsigned int pad,
> > +				 struct v4l2_mbus_frame_desc *desc)
> > +{
> > +	struct v4l2_mbus_frame_desc_entry *entry = desc->entry;
> > +	struct v4l2_subdev_state *sd_state;
> > +	struct v4l2_mbus_framefmt *fmt;
> > +
> > +	desc->type = V4L2_MBUS_FRAME_DESC_TYPE_CSI2;
> > +
> > +	sd_state = v4l2_subdev_lock_and_get_active_state(sd);
> > +	fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_SOURCE,
> > +					   OV2740_STREAM_PIXEL);
> > +	entry->pixelcode = fmt->code;
> > +	v4l2_subdev_unlock_state(sd_state);
> > +
> > +	entry->stream = OV2740_STREAM_PIXEL;
> > +	entry->bus.csi2.dt = MIPI_CSI2_DT_RAW10;
> > +	entry++;
> > +	desc->num_entries++;
> 
> I think addressing entries explicitly would be clearer.
> 
> 	entry[0].stream = ...;
> 	entry[0].bus.csi2.dt = ...;
> 
> 	...
> 
> 	desc->num_entries = 2;

I think we can disagree on this. I asked you to do the other way around in
the other patch. ;-)

> 
> > +
> > +	entry->pixelcode = MEDIA_BUS_FMT_META_8;
> > +	entry->stream = OV2740_STREAM_META;
> > +	entry->bus.csi2.dt = MIPI_CSI2_DT_GENERIC_LONG(1);
> 
> As Bingbu mentioned, this is not correct.

With default settings it would have seemed to be, but I think we can switch
to the embedded data type.

> 
> > +	entry++;
> > +	desc->num_entries++;
> >  
> > -	ov2740_update_pad_format(&ov2740->supported_modes[0],
> > -				 v4l2_subdev_state_get_format(sd_state, 0));
> >  	return 0;
> >  }
> >  
> > @@ -1103,6 +1221,7 @@ static const struct v4l2_subdev_pad_ops ov2740_pad_ops = {
> >  	.enum_frame_size = ov2740_enum_frame_size,
> >  	.enable_streams = ov2740_enable_streams,
> >  	.disable_streams = ov2740_disable_streams,
> > +	.get_frame_desc = ov2740_get_frame_desc,
> >  };
> >  
> >  static const struct v4l2_subdev_ops ov2740_subdev_ops = {
> > @@ -1369,11 +1488,16 @@ static int ov2740_probe(struct i2c_client *client)
> >  	}
> >  
> >  	ov2740->sd.state_lock = ov2740->ctrl_handler.lock;
> > -	ov2740->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
> > +	ov2740->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_STREAMS;
> >  	ov2740->sd.entity.ops = &ov2740_subdev_entity_ops;
> >  	ov2740->sd.entity.function = MEDIA_ENT_F_CAM_SENSOR;
> > -	ov2740->pad.flags = MEDIA_PAD_FL_SOURCE;
> > -	ret = media_entity_pads_init(&ov2740->sd.entity, 1, &ov2740->pad);
> > +	ov2740->pads[OV2740_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
> > +	ov2740->pads[OV2740_PAD_PIXEL].flags =
> > +		MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL;
> > +	ov2740->pads[OV2740_PAD_META].flags =
> > +		MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL;
> > +	ret = media_entity_pads_init(&ov2740->sd.entity,
> > +				     ARRAY_SIZE(ov2740->pads), ov2740->pads);
> >  	if (ret) {
> >  		dev_err_probe(dev, ret, "failed to init entity pads\n");
> >  		goto probe_error_v4l2_ctrl_handler_free;
> 

-- 
Kind regards,

Sakari Ailus

Re: [PATCH v8 01/38] media: mc: Add INTERNAL pad flag

[Sakari Ailus]

Hi Laurent,

On Thu, Mar 28, 2024 at 05:25:26PM +0200, Laurent Pinchart wrote:
> Hi Sakari,
> 
> On Thu, Mar 28, 2024 at 09:47:26AM +0000, Sakari Ailus wrote:
> > On Thu, Mar 21, 2024 at 07:20:32PM +0200, Laurent Pinchart wrote:
> > > On Wed, Mar 20, 2024 at 07:49:27AM +0000, Sakari Ailus wrote:
> > > > On Wed, Mar 20, 2024 at 12:17:07AM +0200, Laurent Pinchart wrote:
> > > > > On Thu, Mar 14, 2024 at 09:17:08AM +0200, Tomi Valkeinen wrote:
> > > > > > On 13/03/2024 09:24, Sakari Ailus wrote:
> > > > > > > Internal source pads will be used as routing endpoints in V4L2
> > > > > > > [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
> > > > > > > Internal source pads are pads that have both SINK and INTERNAL flags set.
> > > > > > > 
> > > > > > > Also prevent creating links to pads that have been flagged as internal and
> > > > > > > initialising SOURCE pads with INTERNAL flag set.
> > > > > > > 
> > > > > > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > > > > > ---
> > > > > > >   .../userspace-api/media/mediactl/media-types.rst       |  8 ++++++++
> > > > > > >   drivers/media/mc/mc-entity.c                           | 10 ++++++++--
> > > > > > >   include/uapi/linux/media.h                             |  1 +
> > > > > > >   3 files changed, 17 insertions(+), 2 deletions(-)
> > > > > > > 
> > > > > > > diff --git a/Documentation/userspace-api/media/mediactl/media-types.rst b/Documentation/userspace-api/media/mediactl/media-types.rst
> > > > > > > index 6332e8395263..f55ef055bcf8 100644
> > > > > > > --- a/Documentation/userspace-api/media/mediactl/media-types.rst
> > > > > > > +++ b/Documentation/userspace-api/media/mediactl/media-types.rst
> > > > > > > @@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
> > > > > > >   .. _MEDIA-PAD-FL-SINK:
> > > > > > >   .. _MEDIA-PAD-FL-SOURCE:
> > > > > > >   .. _MEDIA-PAD-FL-MUST-CONNECT:
> > > > > > > +.. _MEDIA-PAD-FL-INTERNAL:
> > > > > > >   
> > > > > > >   .. flat-table:: Media pad flags
> > > > > > >       :header-rows:  0
> > > > > > > @@ -381,6 +382,13 @@ Types and flags used to represent the media graph elements
> > > > > > >   	  enabled links even when this flag isn't set; the absence of the flag
> > > > > > >   	  doesn't imply there is none.
> > > > > > >   
> > > > > > > +    *  -  ``MEDIA_PAD_FL_INTERNAL``
> > > > > > > +       -  The internal flag indicates an internal pad that has no external
> > > > > > > +	  connections. Such a pad shall not be connected with a link.
> > > > > 
> > > > > I would expand this slightly, as it's the only source of documentation
> > > > > regarding internal pads.
> > > > 
> > > > Patch 9 adds more documentation, this patch is for MC only.
> > > 
> > > That's my point, it's the only source of documentation for internal
> > > pads from an MC point of view, so expanding the documentation would be
> > > good :-)
> > 
> > This is MC documentation, camera sensors aren't relevant in this context.
> 
> They are only relevant in the same that they are one of the many
> possible users of the MC API.

Cameras are mentioned in MC documentation only in generic sense, outside
the context of V4L2. The patch is currently in line with existing practices
with it comes to MC documentation.

If some V4L2 documentation would be added under MC documentation, it should
be discussed (and implemented if so decided) separately from this patch.

There is more documentation later in the series, also relating to internal
pads.

> 
> > > > > 
> > > > >        -  The internal flag indicates an internal pad that has no external
> > > > > 	  connections. This can be used to model, for instance, the pixel array
> > > > > 	  internal to an image sensor. As they are internal to entities,
> > > > > 	  internal pads shall not be connected with links.
> > > > 
> > > > I'd drop the sentence related to sensors.
> > > 
> > > I'm fine with another example, or a more generic explanation, but with
> > > that sentence dropped, I think this will leave the reader wondering what
> > > an internal pad is and what it's used for.
> > 
> > What we could and probably have here is that the internal sink pad
> > indicates a source of data. That's what it really is, whether that data is
> > image data or something else.
> > 
> > So I'd change this to:
> > 
> > The internal flag indicates an internal pad that has no external
> > connections. Such a pad shall not be connected with a link. The internal
> > pad flag is allowed only in conjunction with the sink pad flag. Together
> > the two flags indicate the pad is a source of data inside the entity.
> 
> I think that's still difficult to understand. What's the issue with
> mentioning a particular example to make the concept easier to understand
> ?
> 
> > > > > > > +
> > > > > > > +	  The internal flag may currently be present only in a source pad where
> > > > > > 
> > > > > > s/source/sink/
> > > > > > 
> > > > > > Reviewed-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > > > > > > +	  it indicates that the :ref:``stream <media-glossary-stream>``
> > > > > > > +	  originates from within the entity.
> > > > > > >   
> > > > > > >   One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
> > > > > > >   must be set for every pad.
> > > > > > > diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> > > > > > > index 0e28b9a7936e..1973e9e1013e 100644
> > > > > > > --- a/drivers/media/mc/mc-entity.c
> > > > > > > +++ b/drivers/media/mc/mc-entity.c
> > > > > > > @@ -213,7 +213,9 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
> > > > > > >   		iter->index = i++;
> > > > > > >   
> > > > > > >   		if (hweight32(iter->flags & (MEDIA_PAD_FL_SINK |
> > > > > > > -					     MEDIA_PAD_FL_SOURCE)) != 1) {
> > > > > > > +					     MEDIA_PAD_FL_SOURCE)) != 1 ||
> > > > > > > +		    (iter->flags & MEDIA_PAD_FL_INTERNAL &&
> > > > > > > +		     !(iter->flags & MEDIA_PAD_FL_SINK))) {
> > > > > > >   			ret = -EINVAL;
> > > > > > >   			break;
> > > > > > >   		}
> > > > > 
> > > > > This may become more readable written as a switch statement:
> > > > > 
> > > > > 		const u32 pad_flags_mask = MEDIA_PAD_FL_SINK |
> > > > > 					   MEDIA_PAD_FL_SOURCE |
> > > > > 					   MEDIA_PAD_FL_INTERNAL;
> > > > > 
> > > > > 		switch (iter->flags & pad_flags_mask) {
> > > > > 		case MEDIA_PAD_FL_SINK:
> > > > > 		case MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL:
> > > > > 		case MEDIA_PAD_FL_SOURCE:
> > > > > 			break;
> > > > > 
> > > > > 		default:
> > > > > 			ret = -EINVAL;
> > > > > 			break;
> > > > > 		}
> > > > > 
> > > > > 		if (ret)
> > > > > 			break;
> > > > > 
> > > > > And now that I've written this, I'm not too sure anymore :-) Another
> > > > > option would be
> > > > > 
> > > > > 
> > > > > 	const u32 pad_flags = iter->flags & (MEDIA_PAD_FL_SINK |
> > > > > 					     MEDIA_PAD_FL_SOURCE |
> > > > > 					     MEDIA_PAD_FL_INTERNAL);
> > > > > 
> > > > > 	if (pad_flags != MEDIA_PAD_FL_SINK &&
> > > > > 	    pad_flags != MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL
> > > > > 	    pad_flags != MEDIA_PAD_FL_SOURCE) {
> > > > > 		ret = -EINVAL;
> > > > > 		break;
> > > > > 	}
> > > > > 
> > > > > Up to you.
> > > > 
> > > > I'd prefer to keep it as-is since the original check is testing two
> > > > independent things instead of merging them: that only either SINK or SOURCE
> > > > is set, and then separately that if INTERNAL is set, then SINK is set, too.
> > > > 
> > > > Of the two options you suggested, I prefer the latter.
> > > 
> > > I prefer the latter too, and I think it's more readable than the current
> > > code. If we later end up having to test for more rules, we can separate
> > > the checks.
> > 
> > I can switch to the latter.
> > 
> > > > > 
> > > > > Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > > > 
> > > > Thank you!
> > > > 
> > > > > > > @@ -1112,7 +1114,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
> > > > > > >   
> > > > > > >   	for (i = 0; i < entity->num_pads; i++) {
> > > > > > >   		if ((entity->pads[i].flags &
> > > > > > > -		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
> > > > > > > +		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
> > > > > > > +		      MEDIA_PAD_FL_INTERNAL)) != pad_type)
> > > > > > >   			continue;
> > > > > > >   
> > > > > > >   		if (entity->pads[i].sig_type == sig_type)
> > > > > > > @@ -1142,6 +1145,9 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
> > > > > > >   		return -EINVAL;
> > > > > > >   	if (WARN_ON(!(sink->pads[sink_pad].flags & MEDIA_PAD_FL_SINK)))
> > > > > > >   		return -EINVAL;
> > > > > > > +	if (WARN_ON(source->pads[source_pad].flags & MEDIA_PAD_FL_INTERNAL) ||
> > > > > > > +	    WARN_ON(sink->pads[sink_pad].flags & MEDIA_PAD_FL_INTERNAL))
> > > > > > > +		return -EINVAL;
> > > > > > >   
> > > > > > >   	link = media_add_link(&source->links);
> > > > > > >   	if (link == NULL)
> > > > > > > diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
> > > > > > > index 1c80b1d6bbaf..80cfd12a43fc 100644
> > > > > > > --- a/include/uapi/linux/media.h
> > > > > > > +++ b/include/uapi/linux/media.h
> > > > > > > @@ -208,6 +208,7 @@ struct media_entity_desc {
> > > > > > >   #define MEDIA_PAD_FL_SINK			(1U << 0)
> > > > > > >   #define MEDIA_PAD_FL_SOURCE			(1U << 1)
> > > > > > >   #define MEDIA_PAD_FL_MUST_CONNECT		(1U << 2)
> > > > > > > +#define MEDIA_PAD_FL_INTERNAL			(1U << 3)
> > > > > > >   
> > > > > > >   struct media_pad_desc {
> > > > > > >   	__u32 entity;		/* entity ID */
> 

-- 
Regards,

Sakari Ailus

Re: [PATCH v8 10/38] media: Documentation: Document S_ROUTING behaviour

[Sakari Ailus]

Hi Laurent,

On Wed, Mar 20, 2024 at 02:33:16AM +0200, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.
> 
> On Wed, Mar 13, 2024 at 09:24:48AM +0200, Sakari Ailus wrote:
> > Document S_ROUTING behaviour for different devices.
> > 
> > Generally in devices that produce streams the streams are static and some
> > can be enabled and disabled, whereas in devices that just transport them
> > or write them to memory, more configurability is allowed. Document this.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  .../userspace-api/media/v4l/dev-subdev.rst    | 24 +++++++++++++++++++
> >  1 file changed, 24 insertions(+)
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > index 1808f40f63e3..08495cc6f4a6 100644
> > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > @@ -593,6 +593,30 @@ Any configurations of a stream within a pad, such as format or selections,
> >  are independent of similar configurations on other streams. This is
> >  subject to change in the future.
> >  
> > +Device types and routing setup
> > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > +
> > +Different kinds of sub-devices have differing behaviour for route activation,
> > +depending on the hardware. In all cases, however, only routes that have the
> > +``V4L2_SUBDEV_STREAM_FL_ACTIVE`` flag set are active.
> > +
> > +Devices generating the streams may allow enabling and disabling some of the
> > +routes or the configuration is fixed. If the routes can be disabled, not
> > +declaring the routes (or declaring them without
> > +``VIDIOC_SUBDEV_STREAM_FL_ACTIVE`` flag set) in ``VIDIOC_SUBDEV_S_ROUTING`` will
> > +disable the routes while the sub-device driver retains the streams and their
> 
> No idea what you mean by "retains the streams and their configuration".

I'll add that this means format and selection configuration.

> 
> > +configuration. The ``VIDIOC_SUBDEV_S_ROUTING`` will still return such routes
> > +back to the user in the routes array, with the ``V4L2_SUBDEV_STREAM_FL_ACTIVE``
> > +flag unset.
> 
> I find this quite confusing. There's no explanation as to why the API
> behaves this way. I think you should rewrite this documentation, trying
> to evaluate it from the point of view of someone who doesn't know about
> the routing API.

I wonder if this would be easier for the user if such routes had a flag
that would tell they're always there. It could be called e.g. STATIC. Then
the user would be able to expect certain behaviour before disabling a link.

The purpose is really to avoid removing streams that the user would not
know how to re-create in order to make the device function again.

> 
> > +
> > +Devices transporting the streams almost always have more configurability with
> > +respect to routing. Typically any route between the sub-device's sink and source
> > +pads is possible, and multiple routes (usually up to certain limited number) may
> > +be active simultaneously. For such devices, no routes are created by the driver
> > +and user-created routes are fully replaced when ``VIDIOC_SUBDEV_S_ROUTING`` is
> > +called on the sub-device. Such newly created routes have the device's default
> > +configuration for format and selection rectangles.
> > +
> >  Configuring streams
> >  ^^^^^^^^^^^^^^^^^^^
> >  
> 

-- 
Regards,

Sakari Ailus

Re: [PATCH v8 13/38] media: v4l: subdev: Copy argument back to user also for S_ROUTING

[Sakari Ailus]

Hi Laurent,

Thanks for the review.

On Wed, Mar 20, 2024 at 02:39:07AM +0200, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.
> 
> On Wed, Mar 13, 2024 at 09:24:51AM +0200, Sakari Ailus wrote:
> > As the user needs to know what went wrong for S_ROUTING, copy array
> > arguments back to the user.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  drivers/media/v4l2-core/v4l2-ioctl.c | 9 ++++++---
> >  1 file changed, 6 insertions(+), 3 deletions(-)
> > 
> > diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> > index d125d23e4e61..95bd56145d38 100644
> > --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> > +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> > @@ -3415,11 +3415,14 @@ video_usercopy(struct file *file, unsigned int orig_cmd, unsigned long arg,
> >  	 * FIXME: subdev IOCTLS are partially handled here and partially in
> >  	 * v4l2-subdev.c and the 'always_copy' flag can only be set for IOCTLS
> >  	 * defined here as part of the 'v4l2_ioctls' array. As
> > -	 * VIDIOC_SUBDEV_G_ROUTING needs to return results to applications even
> > -	 * in case of failure, but it is not defined here as part of the
> > +	 * VIDIOC_SUBDEV_[GS]_ROUTING needs to return results to applications
> 
> s/needs/need/

I'd say "needs" is correct.

> 
> Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> 
> > +	 * even in case of failure, but it is not defined here as part of the
> >  	 * 'v4l2_ioctls' array, insert an ad-hoc check to address that.
> >  	 */
> > -	if (err < 0 && !always_copy && cmd != VIDIOC_SUBDEV_G_ROUTING)
> > +	if (cmd == VIDIOC_SUBDEV_G_ROUTING || cmd == VIDIOC_SUBDEV_S_ROUTING)
> > +		always_copy = true;
> > +
> > +	if (err < 0 && !always_copy)
> >  		goto out;
> >  
> >  	if (has_array_args) {
> 

-- 
Regards,

Sakari Ailus

Re: [PATCH v8 17/38] media: v4l: subdev: Add trivial set_routing support

[Sakari Ailus]

Hi Laurent,

On Tue, Apr 02, 2024 at 02:41:05AM +0300, Laurent Pinchart wrote:
> On Wed, Mar 20, 2024 at 03:55:58AM +0200, Laurent Pinchart wrote:
> > Hi Sakari,
> > 
> > Thank you for the patch.
> > 
> > On Wed, Mar 13, 2024 at 09:24:55AM +0200, Sakari Ailus wrote:
> > > Add trivial S_ROUTING IOCTL support for drivers where routing is static.
> > > Essentially this means returning the same information G_ROUTING call would
> > > have done.
> > > 
> > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > 
> > Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> 
> Actually...
> 
> > > ---
> > >  drivers/media/v4l2-core/v4l2-subdev.c | 14 ++++++++++++++
> > >  1 file changed, 14 insertions(+)
> > > 
> > > diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> > > index a6107e440ef0..c8c435df92c8 100644
> > > --- a/drivers/media/v4l2-core/v4l2-subdev.c
> > > +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> > > @@ -930,6 +930,20 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
> > >  
> > >  		memset(routing->reserved, 0, sizeof(routing->reserved));
> > >  
> > > +		/*
> > > +		 * If the driver doesn't support setting routing, just return
> > > +		 * the routing table here.
> > > +		 */
> > > +		if (!v4l2_subdev_has_op(sd, pad, set_routing)) {
> > > +			memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
> > > +			       state->routing.routes,
> > > +			       min(state->routing.num_routes, routing->len_routes) *
> > > +			       sizeof(*state->routing.routes));
> > > +			routing->num_routes = state->routing.num_routes;
> > > +
> > > +			return 0;
> > > +		}
> > > +
> 
> I think this should be done after the next code block that validates the
> arguments. Otherwise the S_ROUTING ioctl will behave differently with
> regard to blatantly invalid arguments, depending on whether or not the
> subdev implements the .set_routing() operation. This can confuse
> userspace, and does confuse v4l2-compliance.
> 
> I have the following change in my tree that fixes the problem, feel free
> to squash it with this patch in v9.
> 
> commit 1e1f03eb8bc118c53a9deab05063d71a2fe7aa5f
> Author: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Date:   Tue Apr 2 02:06:01 2024 +0300
> 
>     fixup! media: v4l: subdev: Add trivial set_routing support
> 
>     Validate arguments before handling the trivial set_routing support to
>     expose a consistent behaviour to userspace. This fixes an issue with
>     v4l2-compliance.
> 
>     Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Thanks, I'll squash this into the patch.

> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index def91ab32c07..129867ed2bad 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -933,19 +933,12 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>  		memset(routing->reserved, 0, sizeof(routing->reserved));
> 
>  		/*
> -		 * If the driver doesn't support setting routing, just return
> -		 * the routing table here.
> +		 * Perform argument validation first, or subdevs that don't
> +		 * support setting routing will not return an error when
> +		 * arguments are blatantly wrong. The difference in behaviour
> +		 * could be confusing for userspace, and in particular for API
> +		 * compliance checkers.

This is more fit for development time discussion, not something that should
be left in the code IMO.

>  		 */
> -		if (!v4l2_subdev_has_op(sd, pad, set_routing)) {
> -			memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
> -			       state->routing.routes,
> -			       min(state->routing.num_routes, routing->len_routes) *
> -			       sizeof(*state->routing.routes));
> -			routing->num_routes = state->routing.num_routes;
> -
> -			return 0;
> -		}
> -
>  		for (i = 0; i < routing->num_routes; ++i) {
>  			const struct v4l2_subdev_route *route = &routes[i];
>  			const struct media_pad *pads = sd->entity.pads;
> @@ -969,6 +962,20 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>  				return -EINVAL;
>  		}
> 
> +		/*
> +		 * If the driver doesn't support setting routing, just return
> +		 * the routing table here.
> +		 */
> +		if (!v4l2_subdev_has_op(sd, pad, set_routing)) {
> +			memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
> +			       state->routing.routes,
> +			       min(state->routing.num_routes, routing->len_routes) *
> +			       sizeof(*state->routing.routes));
> +			routing->num_routes = state->routing.num_routes;
> +
> +			return 0;
> +		}
> +
>  		krouting.num_routes = routing->num_routes;
>  		krouting.len_routes = routing->len_routes;
>  		krouting.routes = routes;
> 
> 
> > >  		for (i = 0; i < routing->num_routes; ++i) {
> > >  			const struct v4l2_subdev_route *route = &routes[i];
> > >  			const struct media_pad *pads = sd->entity.pads;
> 

-- 
Regards,

Sakari Ailus

Re: [PATCH v8 22/38] media: ccs: Support frame descriptors

[Sakari Ailus]

Hi Laurent,

On Thu, Mar 21, 2024 at 06:44:48PM +0200, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.

Thanks for the review!

> 
> On Wed, Mar 13, 2024 at 09:25:00AM +0200, Sakari Ailus wrote:
> > Provide information on the frame layout using frame descriptors.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  drivers/media/i2c/ccs/ccs-core.c  | 60 +++++++++++++++++++++++++++++++
> >  drivers/media/i2c/ccs/ccs-quirk.h |  7 ++++
> >  drivers/media/i2c/ccs/ccs.h       |  4 +++
> >  3 files changed, 71 insertions(+)
> > 
> > diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
> > index 0efbc63534bc..9cc2080b73ec 100644
> > --- a/drivers/media/i2c/ccs/ccs-core.c
> > +++ b/drivers/media/i2c/ccs/ccs-core.c
> > @@ -25,6 +25,7 @@
> >  #include <linux/slab.h>
> >  #include <linux/smiapp.h>
> >  #include <linux/v4l2-mediabus.h>
> > +#include <media/mipi-csi2.h>
> >  #include <media/v4l2-cci.h>
> >  #include <media/v4l2-device.h>
> >  #include <media/v4l2-fwnode.h>
> > @@ -245,6 +246,33 @@ static int ccs_read_all_limits(struct ccs_sensor *sensor)
> >  	return ret;
> >  }
> >  
> > +static u8 ccs_mipi_csi2_data_type(unsigned int bpp)
> > +{
> > +	switch (bpp) {
> > +	case 6:
> > +		return MIPI_CSI2_DT_RAW6;
> > +	case 7:
> > +		return MIPI_CSI2_DT_RAW7;
> > +	case 8:
> > +		return MIPI_CSI2_DT_RAW8;
> > +	case 10:
> > +		return MIPI_CSI2_DT_RAW10;
> > +	case 12:
> > +		return MIPI_CSI2_DT_RAW12;
> > +	case 14:
> > +		return MIPI_CSI2_DT_RAW14;
> > +	case 16:
> > +		return MIPI_CSI2_DT_RAW16;
> > +	case 20:
> > +		return MIPI_CSI2_DT_RAW20;
> > +	case 24:
> > +		return MIPI_CSI2_DT_RAW24;
> > +	default:
> > +		WARN_ON(1);
> > +		return 0;
> > +	}
> > +}
> > +
> >  static int ccs_read_frame_fmt(struct ccs_sensor *sensor)
> >  {
> >  	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
> > @@ -2632,6 +2660,37 @@ static int ccs_set_selection(struct v4l2_subdev *subdev,
> >  	return ret;
> >  }
> >  
> > +static int ccs_get_frame_desc(struct v4l2_subdev *subdev, unsigned int pad,
> > +				 struct v4l2_mbus_frame_desc *desc)
> > +{
> > +	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
> > +	struct v4l2_mbus_frame_desc_entry *entry = desc->entry;
> > +
> > +	if (ccs_has_quirk(sensor, frame_desc))
> > +		return ccs_call_quirk(sensor, frame_desc, desc);
> 
> I would introduce the quirk later, along with the patch that will use
> it.

There is no user for this quirk in the series. It's been added for quirk
infrastructure completeness, I guess I can move it in a separate patch and
postpone that.

> 
> > +
> > +	switch (sensor->hwcfg.csi_signalling_mode) {
> > +	case CCS_CSI_SIGNALING_MODE_CSI_2_DPHY:
> > +	case CCS_CSI_SIGNALING_MODE_CSI_2_CPHY:
> > +		desc->type = V4L2_MBUS_FRAME_DESC_TYPE_CSI2;
> > +		break;
> > +	default:
> > +		/* FIXME: CCP2 support */
> > +		return -EINVAL;
> > +	}
> > +
> > +	entry->pixelcode = sensor->csi_format->code;
> > +	entry->stream = CCS_STREAM_PIXEL;
> > +	entry->bus.csi2.dt =
> > +		sensor->csi_format->width == sensor->csi_format->compressed ?
> > +		ccs_mipi_csi2_data_type(sensor->csi_format->compressed) :
> 
> Functionally equivalent,
> 
> 		ccs_mipi_csi2_data_type(sensor->csi_format->width) :
> 
> would be clearer I think. The way it's written today made me wonder why
> you want the DT for the compressed format, which is not what you're
> doing.

Sounds good.

> 
> > +		CCS_DEFAULT_COMPRESSED_DT;
> > +	entry++;
> > +	desc->num_entries++;
> > +
> > +	return 0;
> > +}
> > +
> >  static int ccs_get_skip_frames(struct v4l2_subdev *subdev, u32 *frames)
> >  {
> >  	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
> > @@ -3054,6 +3113,7 @@ static const struct v4l2_subdev_pad_ops ccs_pad_ops = {
> >  	.set_selection = ccs_set_selection,
> >  	.enable_streams = ccs_enable_streams,
> >  	.disable_streams = ccs_disable_streams,
> > +	.get_frame_desc = ccs_get_frame_desc,
> >  };
> >  
> >  static const struct v4l2_subdev_sensor_ops ccs_sensor_ops = {
> > diff --git a/drivers/media/i2c/ccs/ccs-quirk.h b/drivers/media/i2c/ccs/ccs-quirk.h
> > index 392c97109617..3e1d9eaa33fa 100644
> > --- a/drivers/media/i2c/ccs/ccs-quirk.h
> > +++ b/drivers/media/i2c/ccs/ccs-quirk.h
> > @@ -36,6 +36,7 @@ struct ccs_sensor;
> >   *			 access may be done by the caller (default read
> >   *			 value is zero), else negative error code on error
> >   * @flags: Quirk flags
> > + * @frame_desc: Obtain the frame descriptor
> >   */
> >  struct ccs_quirk {
> >  	int (*limits)(struct ccs_sensor *sensor);
> > @@ -46,6 +47,8 @@ struct ccs_quirk {
> >  	int (*init)(struct ccs_sensor *sensor);
> >  	int (*reg_access)(struct ccs_sensor *sensor, bool write, u32 *reg,
> >  			  u32 *val);
> > +	int (*frame_desc)(struct ccs_sensor *sensor,
> > +			  struct v4l2_mbus_frame_desc *desc);
> >  	unsigned long flags;
> >  };
> >  
> > @@ -62,6 +65,10 @@ struct ccs_reg_8 {
> >  		.val = _val,		\
> >  	}
> >  
> > +#define ccs_has_quirk(sensor, _quirk)					\
> > +	((sensor)->minfo.quirk &&					\
> > +	 (sensor)->minfo.quirk->_quirk)
> > +
> >  #define ccs_call_quirk(sensor, _quirk, ...)				\
> >  	((sensor)->minfo.quirk &&					\
> >  	 (sensor)->minfo.quirk->_quirk ?				\
> > diff --git a/drivers/media/i2c/ccs/ccs.h b/drivers/media/i2c/ccs/ccs.h
> > index 4725e6eca8d0..adb152366ea2 100644
> > --- a/drivers/media/i2c/ccs/ccs.h
> > +++ b/drivers/media/i2c/ccs/ccs.h
> > @@ -46,6 +46,8 @@
> >  
> >  #define CCS_COLOUR_COMPONENTS		4
> >  
> > +#define CCS_DEFAULT_COMPRESSED_DT	0x30
> 
> I'd write
> 
> #define CCS_DEFAULT_COMPRESSED_DT	MIPI_CSI2_DT_USER_DEFINED(0)

Yes.

> 
> > +
> >  #define SMIAPP_NAME			"smiapp"
> >  #define CCS_NAME			"ccs"
> >  
> > @@ -175,6 +177,8 @@ struct ccs_csi_data_format {
> >  #define CCS_PAD_SRC			1
> >  #define CCS_PADS			2
> >  
> > +#define CCS_STREAM_PIXEL		0
> > +
> >  struct ccs_binning_subtype {
> >  	u8 horizontal:4;
> >  	u8 vertical:4;
> 

-- 
Sakari Ailus

Re: [PATCH v8 23/38] media: uapi: ccs: Add media bus code for MIPI CCS embedded data

[Sakari Ailus]

Hi Laurent,

On Thu, Mar 21, 2024 at 06:49:14PM +0200, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.
> 
> On Wed, Mar 13, 2024 at 09:25:01AM +0200, Sakari Ailus wrote:
> > Add new MIPI CCS embedded data media bus code
> > (MEDIA_BUS_FMT_CCS_EMBEDDED).
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  .../media/v4l/subdev-formats.rst              | 28 +++++++++++++++++++
> >  include/uapi/linux/media-bus-format.h         |  3 ++
> >  2 files changed, 31 insertions(+)
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > index cbd475f7cae9..c8f982411e70 100644
> > --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > @@ -8564,3 +8564,31 @@ and finally the bit number in subscript. "X" indicates a padding bit.
> >        - X
> >        - X
> >        - X
> > +
> > +.. _MEDIA-BUS-FMT-CCS-EMBEDDED:
> > +
> > +MIPI CCS Embedded Data Formats
> > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > +
> > +`MIPI CCS <https://www.mipi.org/specifications/camera-command-set>`_ defines a
> > +metadata format for sensor embedded data, which is used to store the register
> > +configuration used for capturing a given frame. The format is defined in the CCS
> > +specification. The media bus code for this format is
> > +``MEDIA_BUS_FMT_CCS_EMBEDDED``.
> > +
> > +The CCS embedded data format definition includes three levels:
> > +
> > +1. Padding within CSI-2 bus :ref:`Data unit <media-glossary-data-unit>` as
> 
> s/Data unit/Data Unit/

Yes.

> 
> > +   documented in the MIPI CCS specification.
> > +
> > +2. The tagged data format as documented in the MIPI CCS specification.
> > +
> > +3. Register addresses and register documentation as documented in the MIPI CCS
> > +   specification.
> > +
> > +The format definition shall be used only by devices that fulfill all three
> > +levels above.
> 
> Hmmmm... Do we need to mandate level 3 ? There are lots of sensors that
> comply with the first two levels but have their own register set. Would
> you like a sensor-specific embedded data format for each of them ? If
> so, how would we document it without essentially copying the datasheet ?

You could refer to the datasheet if it's publicly available.

The CCS static data format is meaningful only for accessing (level 1) and
parsing (level 2) the data without knowing its meaning, which is the most
important aspect. The user space will need to know the meaning of the data
and can additionally select the same data unpacker than for CCS and
possibly also the CCS data parser if applicable.

The user space should especially not have to have a list of CCS-compliant
sensors which would become necessary, should we skip on the 3rd level
requirement here.

> 
> > +
> > +This mbus code are only used for "2-byte simplified tagged data format" (code
> > +0xa) but their use may be extended further in the future, to cover other CCS
> > +embedded data format codes.
> > diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
> > index d4c1d991014b..03f7e9ab517b 100644
> > --- a/include/uapi/linux/media-bus-format.h
> > +++ b/include/uapi/linux/media-bus-format.h
> > @@ -183,4 +183,7 @@
> >  #define MEDIA_BUS_FMT_META_20			0x8006
> >  #define MEDIA_BUS_FMT_META_24			0x8007
> >  
> > +/* Specific metadata formats. Next is 0x9002. */
> > +#define MEDIA_BUS_FMT_CCS_EMBEDDED		0x9001
> > +
> >  #endif /* __LINUX_MEDIA_BUS_FORMAT_H */
> 

-- 
Regards,

Sakari Ailus

Re: [PATCH v8 37/38] media: ccs: Add IMMUTABLE route flag

[Sakari Ailus]

Hi Laurent,

On Thu, Mar 21, 2024 at 06:59:35PM +0200, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.
> 
> On Wed, Mar 13, 2024 at 09:25:15AM +0200, Sakari Ailus wrote:
> > Add immutable route flag to the routing table.
> 
> I would write here that the metadata stream route is immutable as CCS
> sensors don't offer a way to disable embedded data. Or if they do, that
> it's currently unimplemented :-) I would then also possibly add a todo
> comment to the code.

They don't. Probably because it's mandatory and there are no important
reasons to disable it.

> 
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> 
> Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

Thanks!

> 
> > ---
> >  drivers/media/i2c/ccs/ccs-core.c | 6 ++++--
> >  1 file changed, 4 insertions(+), 2 deletions(-)
> > 
> > diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
> > index 838daab212f2..5205d1ecf8f2 100644
> > --- a/drivers/media/i2c/ccs/ccs-core.c
> > +++ b/drivers/media/i2c/ccs/ccs-core.c
> > @@ -3368,12 +3368,14 @@ static int ccs_src_init_state(struct v4l2_subdev *sd,
> >  			.sink_pad = CCS_PAD_SINK,
> >  			.source_pad = CCS_PAD_SRC,
> >  			.source_stream = CCS_STREAM_PIXEL,
> > -			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> > +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE |
> > +				 V4L2_SUBDEV_ROUTE_FL_IMMUTABLE,
> >  		}, {
> >  			.sink_pad = CCS_PAD_META,
> >  			.source_pad = CCS_PAD_SRC,
> >  			.source_stream = CCS_STREAM_META,
> > -			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> > +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE |
> > +				 V4L2_SUBDEV_ROUTE_FL_IMMUTABLE,
> >  		}
> >  	};
> >  	struct v4l2_subdev_krouting routing = {
> 

-- 
Regards,

Sakari Ailus

Re: [PATCH v8 23/38] media: uapi: ccs: Add media bus code for MIPI CCS embedded data

[Laurent Pinchart]

Hi Sakari,

On Thu, Apr 11, 2024 at 09:04:47AM +0000, Sakari Ailus wrote:
> On Thu, Mar 21, 2024 at 06:49:14PM +0200, Laurent Pinchart wrote:
> > On Wed, Mar 13, 2024 at 09:25:01AM +0200, Sakari Ailus wrote:
> > > Add new MIPI CCS embedded data media bus code
> > > (MEDIA_BUS_FMT_CCS_EMBEDDED).
> > > 
> > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > ---
> > >  .../media/v4l/subdev-formats.rst              | 28 +++++++++++++++++++
> > >  include/uapi/linux/media-bus-format.h         |  3 ++
> > >  2 files changed, 31 insertions(+)
> > > 
> > > diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > > index cbd475f7cae9..c8f982411e70 100644
> > > --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > > +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > > @@ -8564,3 +8564,31 @@ and finally the bit number in subscript. "X" indicates a padding bit.
> > >        - X
> > >        - X
> > >        - X
> > > +
> > > +.. _MEDIA-BUS-FMT-CCS-EMBEDDED:
> > > +
> > > +MIPI CCS Embedded Data Formats
> > > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > +
> > > +`MIPI CCS <https://www.mipi.org/specifications/camera-command-set>`_ defines a
> > > +metadata format for sensor embedded data, which is used to store the register
> > > +configuration used for capturing a given frame. The format is defined in the CCS
> > > +specification. The media bus code for this format is
> > > +``MEDIA_BUS_FMT_CCS_EMBEDDED``.
> > > +
> > > +The CCS embedded data format definition includes three levels:
> > > +
> > > +1. Padding within CSI-2 bus :ref:`Data unit <media-glossary-data-unit>` as
> > 
> > s/Data unit/Data Unit/
> 
> Yes.
> 
> > > +   documented in the MIPI CCS specification.
> > > +
> > > +2. The tagged data format as documented in the MIPI CCS specification.
> > > +
> > > +3. Register addresses and register documentation as documented in the MIPI CCS
> > > +   specification.
> > > +
> > > +The format definition shall be used only by devices that fulfill all three
> > > +levels above.
> > 
> > Hmmmm... Do we need to mandate level 3 ? There are lots of sensors that
> > comply with the first two levels but have their own register set. Would
> > you like a sensor-specific embedded data format for each of them ? If
> > so, how would we document it without essentially copying the datasheet ?
> 
> You could refer to the datasheet if it's publicly available.

What if it's not ? How would you expect documentation to be organized ?
Will we then end up adding one media bus code per sensor ?

> The CCS static data format is meaningful only for accessing (level 1) and
> parsing (level 2) the data without knowing its meaning, which is the most
> important aspect. The user space will need to know the meaning of the data
> and can additionally select the same data unpacker than for CCS and
> possibly also the CCS data parser if applicable.
> 
> The user space should especially not have to have a list of CCS-compliant
> sensors which would become necessary, should we skip on the 3rd level
> requirement here.
> 
> > > +
> > > +This mbus code are only used for "2-byte simplified tagged data format" (code
> > > +0xa) but their use may be extended further in the future, to cover other CCS
> > > +embedded data format codes.
> > > diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
> > > index d4c1d991014b..03f7e9ab517b 100644
> > > --- a/include/uapi/linux/media-bus-format.h
> > > +++ b/include/uapi/linux/media-bus-format.h
> > > @@ -183,4 +183,7 @@
> > >  #define MEDIA_BUS_FMT_META_20			0x8006
> > >  #define MEDIA_BUS_FMT_META_24			0x8007
> > >  
> > > +/* Specific metadata formats. Next is 0x9002. */
> > > +#define MEDIA_BUS_FMT_CCS_EMBEDDED		0x9001
> > > +
> > >  #endif /* __LINUX_MEDIA_BUS_FORMAT_H */

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 17/38] media: v4l: subdev: Add trivial set_routing support

[Laurent Pinchart]

On Thu, Apr 11, 2024 at 08:13:31AM +0000, Sakari Ailus wrote:
> Hi Laurent,
> 
> On Tue, Apr 02, 2024 at 02:41:05AM +0300, Laurent Pinchart wrote:
> > On Wed, Mar 20, 2024 at 03:55:58AM +0200, Laurent Pinchart wrote:
> > > Hi Sakari,
> > > 
> > > Thank you for the patch.
> > > 
> > > On Wed, Mar 13, 2024 at 09:24:55AM +0200, Sakari Ailus wrote:
> > > > Add trivial S_ROUTING IOCTL support for drivers where routing is static.
> > > > Essentially this means returning the same information G_ROUTING call would
> > > > have done.
> > > > 
> > > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > 
> > > Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> > 
> > Actually...
> > 
> > > > ---
> > > >  drivers/media/v4l2-core/v4l2-subdev.c | 14 ++++++++++++++
> > > >  1 file changed, 14 insertions(+)
> > > > 
> > > > diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> > > > index a6107e440ef0..c8c435df92c8 100644
> > > > --- a/drivers/media/v4l2-core/v4l2-subdev.c
> > > > +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> > > > @@ -930,6 +930,20 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
> > > >  
> > > >  		memset(routing->reserved, 0, sizeof(routing->reserved));
> > > >  
> > > > +		/*
> > > > +		 * If the driver doesn't support setting routing, just return
> > > > +		 * the routing table here.
> > > > +		 */
> > > > +		if (!v4l2_subdev_has_op(sd, pad, set_routing)) {
> > > > +			memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
> > > > +			       state->routing.routes,
> > > > +			       min(state->routing.num_routes, routing->len_routes) *
> > > > +			       sizeof(*state->routing.routes));
> > > > +			routing->num_routes = state->routing.num_routes;
> > > > +
> > > > +			return 0;
> > > > +		}
> > > > +
> > 
> > I think this should be done after the next code block that validates the
> > arguments. Otherwise the S_ROUTING ioctl will behave differently with
> > regard to blatantly invalid arguments, depending on whether or not the
> > subdev implements the .set_routing() operation. This can confuse
> > userspace, and does confuse v4l2-compliance.
> > 
> > I have the following change in my tree that fixes the problem, feel free
> > to squash it with this patch in v9.
> > 
> > commit 1e1f03eb8bc118c53a9deab05063d71a2fe7aa5f
> > Author: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > Date:   Tue Apr 2 02:06:01 2024 +0300
> > 
> >     fixup! media: v4l: subdev: Add trivial set_routing support
> > 
> >     Validate arguments before handling the trivial set_routing support to
> >     expose a consistent behaviour to userspace. This fixes an issue with
> >     v4l2-compliance.
> > 
> >     Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> 
> Thanks, I'll squash this into the patch.
> 
> > 
> > diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> > index def91ab32c07..129867ed2bad 100644
> > --- a/drivers/media/v4l2-core/v4l2-subdev.c
> > +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> > @@ -933,19 +933,12 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
> >  		memset(routing->reserved, 0, sizeof(routing->reserved));
> > 
> >  		/*
> > -		 * If the driver doesn't support setting routing, just return
> > -		 * the routing table here.
> > +		 * Perform argument validation first, or subdevs that don't
> > +		 * support setting routing will not return an error when
> > +		 * arguments are blatantly wrong. The difference in behaviour
> > +		 * could be confusing for userspace, and in particular for API
> > +		 * compliance checkers.
> 
> This is more fit for development time discussion, not something that should
> be left in the code IMO.

I added that comment to make sure that the next developer who refactors
the code will not end up changing the order and introducing subtle
breakages without realizing it. There's a high chance we wouldn't catch
the problem during review if this happens after our brain caches get
flushed.

> > -		if (!v4l2_subdev_has_op(sd, pad, set_routing)) {
> > -			memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
> > -			       state->routing.routes,
> > -			       min(state->routing.num_routes, routing->len_routes) *
> > -			       sizeof(*state->routing.routes));
> > -			routing->num_routes = state->routing.num_routes;
> > -
> > -			return 0;
> > -		}
> > -
> >  		for (i = 0; i < routing->num_routes; ++i) {
> >  			const struct v4l2_subdev_route *route = &routes[i];
> >  			const struct media_pad *pads = sd->entity.pads;
> > @@ -969,6 +962,20 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
> >  				return -EINVAL;
> >  		}
> > 
> > +		/*
> > +		 * If the driver doesn't support setting routing, just return
> > +		 * the routing table here.
> > +		 */
> > +		if (!v4l2_subdev_has_op(sd, pad, set_routing)) {
> > +			memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
> > +			       state->routing.routes,
> > +			       min(state->routing.num_routes, routing->len_routes) *
> > +			       sizeof(*state->routing.routes));
> > +			routing->num_routes = state->routing.num_routes;
> > +
> > +			return 0;
> > +		}
> > +
> >  		krouting.num_routes = routing->num_routes;
> >  		krouting.len_routes = routing->len_routes;
> >  		krouting.routes = routes;
> > 
> > 
> > > >  		for (i = 0; i < routing->num_routes; ++i) {
> > > >  			const struct v4l2_subdev_route *route = &routes[i];
> > > >  			const struct media_pad *pads = sd->entity.pads;

-- 
Regards,

Laurent Pinchart

Re: [PATCH v8 23/38] media: uapi: ccs: Add media bus code for MIPI CCS embedded data

[Sakari Ailus]

Hi Laurent,

On Fri, Apr 12, 2024 at 10:07:46PM +0300, Laurent Pinchart wrote:
> Hi Sakari,
> 
> On Thu, Apr 11, 2024 at 09:04:47AM +0000, Sakari Ailus wrote:
> > On Thu, Mar 21, 2024 at 06:49:14PM +0200, Laurent Pinchart wrote:
> > > On Wed, Mar 13, 2024 at 09:25:01AM +0200, Sakari Ailus wrote:
> > > > Add new MIPI CCS embedded data media bus code
> > > > (MEDIA_BUS_FMT_CCS_EMBEDDED).
> > > > 
> > > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > > ---
> > > >  .../media/v4l/subdev-formats.rst              | 28 +++++++++++++++++++
> > > >  include/uapi/linux/media-bus-format.h         |  3 ++
> > > >  2 files changed, 31 insertions(+)
> > > > 
> > > > diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > > > index cbd475f7cae9..c8f982411e70 100644
> > > > --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > > > +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > > > @@ -8564,3 +8564,31 @@ and finally the bit number in subscript. "X" indicates a padding bit.
> > > >        - X
> > > >        - X
> > > >        - X
> > > > +
> > > > +.. _MEDIA-BUS-FMT-CCS-EMBEDDED:
> > > > +
> > > > +MIPI CCS Embedded Data Formats
> > > > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > > +
> > > > +`MIPI CCS <https://www.mipi.org/specifications/camera-command-set>`_ defines a
> > > > +metadata format for sensor embedded data, which is used to store the register
> > > > +configuration used for capturing a given frame. The format is defined in the CCS
> > > > +specification. The media bus code for this format is
> > > > +``MEDIA_BUS_FMT_CCS_EMBEDDED``.
> > > > +
> > > > +The CCS embedded data format definition includes three levels:
> > > > +
> > > > +1. Padding within CSI-2 bus :ref:`Data unit <media-glossary-data-unit>` as
> > > 
> > > s/Data unit/Data Unit/
> > 
> > Yes.
> > 
> > > > +   documented in the MIPI CCS specification.
> > > > +
> > > > +2. The tagged data format as documented in the MIPI CCS specification.
> > > > +
> > > > +3. Register addresses and register documentation as documented in the MIPI CCS
> > > > +   specification.
> > > > +
> > > > +The format definition shall be used only by devices that fulfill all three
> > > > +levels above.
> > > 
> > > Hmmmm... Do we need to mandate level 3 ? There are lots of sensors that
> > > comply with the first two levels but have their own register set. Would
> > > you like a sensor-specific embedded data format for each of them ? If
> > > so, how would we document it without essentially copying the datasheet ?
> > 
> > You could refer to the datasheet if it's publicly available.
> 
> What if it's not ? How would you expect documentation to be organized ?
> Will we then end up adding one media bus code per sensor ?

Probably. The user space needs a way to find out what to expect. If a
sensor produces its own custom metadata format, it needs to be documented.

There's also documentation for OV2740 embedded data in the set.

-- 
Sakari Ailus

Re: [PATCH v8 17/38] media: v4l: subdev: Add trivial set_routing support

[Sakari Ailus]

Hi Laurent,

On Fri, Apr 12, 2024 at 10:14:26PM +0300, Laurent Pinchart wrote:
> > > I think this should be done after the next code block that validates the
> > > arguments. Otherwise the S_ROUTING ioctl will behave differently with
> > > regard to blatantly invalid arguments, depending on whether or not the
> > > subdev implements the .set_routing() operation. This can confuse
> > > userspace, and does confuse v4l2-compliance.
> > > 
> > > I have the following change in my tree that fixes the problem, feel free
> > > to squash it with this patch in v9.
> > > 
> > > commit 1e1f03eb8bc118c53a9deab05063d71a2fe7aa5f
> > > Author: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > > Date:   Tue Apr 2 02:06:01 2024 +0300
> > > 
> > >     fixup! media: v4l: subdev: Add trivial set_routing support
> > > 
> > >     Validate arguments before handling the trivial set_routing support to
> > >     expose a consistent behaviour to userspace. This fixes an issue with
> > >     v4l2-compliance.
> > > 
> > >     Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > 
> > Thanks, I'll squash this into the patch.
> > 
> > > 
> > > diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> > > index def91ab32c07..129867ed2bad 100644
> > > --- a/drivers/media/v4l2-core/v4l2-subdev.c
> > > +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> > > @@ -933,19 +933,12 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
> > >  		memset(routing->reserved, 0, sizeof(routing->reserved));
> > > 
> > >  		/*
> > > -		 * If the driver doesn't support setting routing, just return
> > > -		 * the routing table here.
> > > +		 * Perform argument validation first, or subdevs that don't
> > > +		 * support setting routing will not return an error when
> > > +		 * arguments are blatantly wrong. The difference in behaviour
> > > +		 * could be confusing for userspace, and in particular for API
> > > +		 * compliance checkers.
> > 
> > This is more fit for development time discussion, not something that should
> > be left in the code IMO.
> 
> I added that comment to make sure that the next developer who refactors
> the code will not end up changing the order and introducing subtle
> breakages without realizing it. There's a high chance we wouldn't catch
> the problem during review if this happens after our brain caches get
> flushed.

We don't have other comments like that in the code either. Input validation
is generally needed, in this case it wasn't needed to carry out the task
but to align the API behaviour with the drivers that do and don't support
setting routing. It's not worth a comment here: it's the same for other
IOCTLs as well. So if a comment were to be added, I'd add it before the
switch.

-- 
Regards,

Sakari Ailus

Re: [PATCH v8 05/38] media: uapi: Add generic 8-bit metadata format definitions

[Sakari Ailus]

On Wed, Mar 20, 2024 at 01:37:01AM +0200, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.
> 
> Mentioning V4L2 on the subject line would help differentiating this from
> the media bus codes.

Yes, I'll add that.

> On Wed, Mar 13, 2024 at 09:24:43AM +0200, Sakari Ailus wrote:
> > Generic 8-bit metadata formats define the in-memory data layout but not
> > the format of the data itself. The reasoning for having such formats is to
> > allow CSI-2 receiver drivers to receive and DMA drivers to write the data
> > to memory without knowing a large number of device specific formats.
> 
> s/device specific/device-specific/
> 
> > These formats may be used only in conjunction of a Media controller
> 
> s/of a/with a/

Yes...

> 
> > pipeline where the internal pad of the source sub-device defines the
> > specific format of the data (using an mbus code).
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  .../userspace-api/media/v4l/dev-subdev.rst    |   2 +
> >  .../userspace-api/media/v4l/meta-formats.rst  |   1 +
> >  .../media/v4l/metafmt-generic.rst             | 304 ++++++++++++++++++
> >  drivers/media/v4l2-core/v4l2-ioctl.c          |   7 +
> >  include/uapi/linux/videodev2.h                |   8 +
> >  5 files changed, 322 insertions(+)
> >  create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > index 43988516acdd..f375b820ab68 100644
> > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > @@ -506,6 +506,8 @@ source pads.
> >  
> >      subdev-formats
> >  
> > +.. _subdev-routing:
> > +
> >  Streams, multiplexed media pads and internal routing
> >  ----------------------------------------------------
> >  
> > diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
> > index 0bb61fc5bc00..919f595576b9 100644
> > --- a/Documentation/userspace-api/media/v4l/meta-formats.rst
> > +++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
> > @@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only.
> >      metafmt-vsp1-hgo
> >      metafmt-vsp1-hgt
> >      metafmt-vivid
> > +    metafmt-generic
> 
> I think the intention was to keep those alphabetically sorted, even if
> vivid is out of order already.

I'll sort these while at it.

> 
> > diff --git a/Documentation/userspace-api/media/v4l/metafmt-generic.rst b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > new file mode 100644
> > index 000000000000..2ebab1f895e0
> > --- /dev/null
> > +++ b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > @@ -0,0 +1,304 @@
> > +.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
> > +
> > +**************************************************************************************************************************************************************************************************************************************************************************************************************************
> > +V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O')
> > +**************************************************************************************************************************************************************************************************************************************************************************************************************************
> 
> Aren't you supposed to match the line lengths ?

Fixed.

> 
> > +
> > +
> > +Generic line-based metadata formats
> > +
> > +
> > +Description
> > +===========
> > +
> > +These generic line-based metadata formats define the memory layout of the data
> > +without defining the format or meaning of the metadata itself. These formats may
> > +only be used with a Media controller pipeline where the more specific format is
> 
> "Media Controller" or "media controller", but not a hybrid mix.

I'll use Media Controller.

> 
> > +defined in an :ref:`internal source pad <MEDIA-PAD-FL-INTERNAL>` of the source
> 
> s/defined in/reported by/

Yes...

> 
> > +sub-device. See also :ref:`source routes <subdev-routing>`.
> > +
> > +.. _v4l2-meta-fmt-generic-8:
> > +
> > +V4L2_META_FMT_GENERIC_8
> > +-----------------------
> > +
> > +The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format.
> > +
> > +This format is also used on CSI-2 for both 8 bits per ``Data unit
> 
> s/Data unit/Data Unit/ (through the whole patch series)

Yes.

> 
> > +<media-glossary-data-unit>`` as well as for 16 bits per Data unit when two bytes
> > +of metadata are packed into one 16-bit Data unit.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_8.**
> > +Each cell is one byte. "M" denotes a byte of metadata.
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|
> > +
> > +.. flat-table::
> 
> Let's add a title here
> 
> .. flat-table:: Sample 4x2 Metadata Frame
> 
> (feel free to adjust). Same below.

Sounds good.

> 
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - M\ :sub:`10`
> > +      - M\ :sub:`20`
> > +      - M\ :sub:`30`
> > +    * - start + 4:
> > +      - M\ :sub:`01`
> > +      - M\ :sub:`11`
> > +      - M\ :sub:`21`
> > +      - M\ :sub:`31`
> > +
> > +.. _v4l2-meta-fmt-generic-csi2-10:
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_10
> > +-----------------------------
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
> > +for each 8 bits of data. Every four bytes of metadata is followed by a single
> > +byte of padding. The way the data is packed follows the MIPI CSI-2 specification
> > +and the padding is defined in the MIPI CCS specification.
> 
> Doesn't really sound packed, when you have one byte of padding after
> four bytes of data :-) I understand where this "packing" is from, but
> the documentation seems confusing for people not familiar with the
> topic. How about simplifying it as follows ?
> 
> V4L2_META_FMT_GENERIC_CSI2_10 contains 8-bit generic metadata, stored with one
> padding byte after every four bytes of metadata. This is usually produced by
> CSI-2 receivers with a source that transmits MEDIA_BUS_FMT_META_10, when the
> CSI-2 receiver packs the received data as defined in the MIPI CSI-2
> specification.

The receiver doesn't pack the data, it already arrives packed on the bus.

"Packing" is also generally used term to describe such arrangements (see
e.g. the CCS spec). This also includes packed V4L2 pixel formats.

How about:

V4L2_META_FMT_GENERIC_CSI2_10 contains 8-bit generic metadata packed in 10-bit
data units, with one padding byte after every four bytes of metadata, compliant
with the MIPI CSI-2 specification. This format is typically used by CSI-2
receivers with a source that transmits MEDIA_BUS_FMT_META_10 and the CSI-2
receiver writes the received data to memory as-is.

> 
> > +
> > +This format is also used in conjunction with 20 bits per ``Data unit
> > +<media-glossary-data-unit>`` formats that pack two bytes of metadata into one
> > +Data unit.
> > +
> > +This format is little endian.
> 
> How is it little endian when it's byte-oriented ?

Some bits are still split across bytes.

> 
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
> > +Each cell is one byte. "M" denotes a byte of metadata and "X" a byte of padding.
> 
> 'M' and 'x'.
> 
> These comments apply to the formats below too.

Sounds good.

> 
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
> > +
> > +.. flat-table::
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - M\ :sub:`10`
> > +      - M\ :sub:`20`
> > +      - M\ :sub:`30`
> > +      - X
> > +    * - start + 5:
> > +      - M\ :sub:`01`
> > +      - M\ :sub:`11`
> > +      - M\ :sub:`21`
> > +      - M\ :sub:`31`
> > +      - X
> > +
> > +.. _v4l2-meta-fmt-generic-csi2-12:
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_12
> > +-----------------------------
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_12 contains packed 8-bit generic metadata, 12 bits
> > +for each 8 bits of data. Every four bytes of metadata is followed by a single
> > +byte of padding. The way the data is packed follows the MIPI CSI-2 specification
> > +and the padding is defined in the MIPI CCS specification.
> > +
> > +This format is also used in conjunction with 24 bits per ``Data unit
> > +<media-glossary-data-unit>`` formats that pack two bytes of metadata into one
> > +Data unit.
> > +
> > +This format is little endian.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_12.**
> > +Each cell is one byte. "M" denotes a byte of metadata and "X" a byte of padding.
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> > +
> > +.. flat-table::
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - M\ :sub:`10`
> > +      - X
> > +      - M\ :sub:`20`
> > +      - M\ :sub:`30`
> > +      - X
> > +    * - start + 6:
> > +      - M\ :sub:`01`
> > +      - M\ :sub:`11`
> > +      - X
> > +      - M\ :sub:`21`
> > +      - M\ :sub:`31`
> > +      - X
> > +
> > +.. _v4l2-meta-fmt-generic-csi2-14:
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_14
> > +-----------------------------
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_14 contains packed 8-bit generic metadata, 14 bits
> > +for each 8 bits of data. Every four bytes of metadata is followed by three bytes
> > +of padding. The way the data is packed follows the MIPI CSI-2 specification and
> > +the padding is defined in the MIPI CCS specification.
> > +
> > +This format is little endian.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_14.**
> > +Each cell is one byte. "M" denotes a byte of metadata and "X" a byte of padding.
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{.8cm}|
> > +
> > +.. flat-table::
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - M\ :sub:`10`
> > +      - M\ :sub:`20`
> > +      - M\ :sub:`30`
> > +      - X
> > +      - X
> > +      - X
> > +    * - start + 7:
> > +      - M\ :sub:`01`
> > +      - M\ :sub:`11`
> > +      - M\ :sub:`21`
> > +      - M\ :sub:`31`
> > +      - X
> > +      - X
> > +      - X
> > +
> > +.. _v4l2-meta-fmt-generic-csi2-16:
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_16
> > +-----------------------------
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_16 contains packed 8-bit generic metadata, 16 bits
> > +for each 8 bits of data. Every byte of metadata is followed by one byte of
> > +padding. The way the data is packed follows the MIPI CSI-2 specification and the
> > +padding is defined in the MIPI CCS specification.
> > +
> > +This format is little endian.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_16.**
> > +Each cell is one byte. "M" denotes a byte of metadata and "X" a byte of padding.
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|
> > +
> > +.. flat-table::
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - X
> > +      - M\ :sub:`10`
> > +      - X
> > +      - M\ :sub:`20`
> > +      - X
> > +      - M\ :sub:`30`
> > +      - X
> > +    * - start + 8:
> > +      - M\ :sub:`01`
> > +      - X
> > +      - M\ :sub:`11`
> > +      - X
> > +      - M\ :sub:`21`
> > +      - X
> > +      - M\ :sub:`31`
> > +      - X
> > +
> > +.. _v4l2-meta-fmt-generic-csi2-20:
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_20
> > +-----------------------------
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_20 contains packed 8-bit generic metadata, 20 bits
> > +for each 8 bits of data. Every byte of metadata is followed by alternating one
> > +and two bytes of padding. The way the data is packed follows the MIPI CSI-2
> > +specification and the padding is defined in the MIPI CCS specification.
> > +
> > +This format is little endian.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_20.**
> > +Each cell is one byte. "M" denotes a byte of metadata and "X" a byte of padding.
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> > +
> > +.. flat-table::
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8 8 8 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - X
> > +      - M\ :sub:`10`
> > +      - X
> > +      - X
> > +      - M\ :sub:`20`
> > +      - X
> > +      - M\ :sub:`30`
> > +      - X
> > +      - X
> > +    * - start + 10:
> > +      - M\ :sub:`01`
> > +      - X
> > +      - M\ :sub:`11`
> > +      - X
> > +      - X
> > +      - M\ :sub:`21`
> > +      - X
> > +      - M\ :sub:`31`
> > +      - X
> > +      - X
> > +
> > +.. _v4l2-meta-fmt-generic-csi2-24:
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_24
> > +-----------------------------
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_24 contains packed 8-bit generic metadata, 24 bits
> > +for each 8 bits of data. Every byte of metadata is followed by two bytes of
> > +padding. The way the data is packed follows the MIPI CSI-2 specification and the
> > +padding is defined in the MIPI CCS specification.
> > +
> > +This format is little endian.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_24.**
> > +Each cell is one byte. "M" denotes a byte of metadata and "X" a byte of padding.
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> > +
> > +.. flat-table::
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8 8 8 8 8 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - X
> > +      - X
> > +      - M\ :sub:`10`
> > +      - X
> > +      - X
> > +      - M\ :sub:`20`
> > +      - X
> > +      - X
> > +      - M\ :sub:`30`
> > +      - X
> > +      - X
> > +    * - start + 12:
> > +      - M\ :sub:`01`
> > +      - X
> > +      - X
> > +      - M\ :sub:`11`
> > +      - X
> > +      - X
> > +      - M\ :sub:`21`
> > +      - X
> > +      - X
> > +      - M\ :sub:`31`
> > +      - X
> > +      - X
> > diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> > index 33076af4dfdb..7cb6063f7056 100644
> > --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> > +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> > @@ -1452,6 +1452,13 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
> >  	case V4L2_PIX_FMT_Y210:		descr = "10-bit YUYV Packed"; break;
> >  	case V4L2_PIX_FMT_Y212:		descr = "12-bit YUYV Packed"; break;
> >  	case V4L2_PIX_FMT_Y216:		descr = "16-bit YUYV Packed"; break;
> > +	case V4L2_META_FMT_GENERIC_8:	descr = "8-bit Generic Metadata"; break;
> > +	case V4L2_META_FMT_GENERIC_CSI2_10:	descr = "8b Generic Meta, 10b CSI-2"; break;
> 
> You could write "8-bit Generic Meta, 10b CSI-2" and it would still fit.
> Up to you.

I'll change it. The rest are using "x-bit" already.

> 
> > +	case V4L2_META_FMT_GENERIC_CSI2_12:	descr = "8b Generic Meta, 12b CSI-2"; break;
> > +	case V4L2_META_FMT_GENERIC_CSI2_14:	descr = "8b Generic Meta, 14b CSI-2"; break;
> > +	case V4L2_META_FMT_GENERIC_CSI2_16:	descr = "8b Generic Meta, 16b CSI-2"; break;
> > +	case V4L2_META_FMT_GENERIC_CSI2_20:	descr = "8b Generic Meta, 20b CSI-2"; break;
> > +	case V4L2_META_FMT_GENERIC_CSI2_24:	descr = "8b Generic Meta, 24b CSI-2"; break;
> >  
> >  	default:
> >  		/* Compressed formats */
> > diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> > index a8015e5e7fa4..6a4f8ae30186 100644
> > --- a/include/uapi/linux/videodev2.h
> > +++ b/include/uapi/linux/videodev2.h
> > @@ -839,6 +839,14 @@ struct v4l2_pix_format {
> >  #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
> >  #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
> >  
> > +#define V4L2_META_FMT_GENERIC_8		v4l2_fourcc('M', 'E', 'T', '8') /* Generic 8-bit metadata */
> > +#define V4L2_META_FMT_GENERIC_CSI2_10	v4l2_fourcc('M', 'C', '1', 'A') /* 10-bit CSI-2 packed 8-bit metadata */
> > +#define V4L2_META_FMT_GENERIC_CSI2_12	v4l2_fourcc('M', 'C', '1', 'C') /* 12-bit CSI-2 packed 8-bit metadata */
> > +#define V4L2_META_FMT_GENERIC_CSI2_14	v4l2_fourcc('M', 'C', '1', 'E') /* 14-bit CSI-2 packed 8-bit metadata */
> > +#define V4L2_META_FMT_GENERIC_CSI2_16	v4l2_fourcc('M', 'C', '1', 'G') /* 16-bit CSI-2 packed 8-bit metadata */
> > +#define V4L2_META_FMT_GENERIC_CSI2_20	v4l2_fourcc('M', 'C', '1', 'K') /* 20-bit CSI-2 packed 8-bit metadata */
> > +#define V4L2_META_FMT_GENERIC_CSI2_24	v4l2_fourcc('M', 'C', '1', 'O') /* 24-bit CSI-2 packed 8-bit metadata */
> > +
> >  /* priv field value to indicates that subsequent fields are valid. */
> >  #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
> >  
> 

-- 
Regards,

Sakari Ailus

Re: [PATCH v8 07/38] media: Documentation: Additional streams generally don't harm capture

[Sakari Ailus]

Hi Laurent,

On Wed, Mar 20, 2024 at 01:48:31AM +0200, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.
> 
> On Wed, Mar 13, 2024 at 09:24:45AM +0200, Sakari Ailus wrote:
> > Having extra streams on the source end of the link that cannot be captured
> > by the sink sub-device generally are not an issue, at least not on CSI-2
> > bus. Still document that there may be hardware specific limitations. For
> 
> s/hardware specific/hardware-specific/

Yes.

> 
> > example on parallel bus this might not work on all cases.
> 
> s/bus/buses/

Works for me. We should probably consider renaming "the parallel bus" as
DVP (a spec exists) or something alike, outside this context naturally.

> 
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > Reviewed-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > ---
> >  Documentation/userspace-api/media/v4l/dev-subdev.rst | 6 +++---
> >  1 file changed, 3 insertions(+), 3 deletions(-)
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > index f375b820ab68..a387e8a15b8d 100644
> > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > @@ -529,9 +529,9 @@ the its sink pad and allows to route them individually to one of its source
> >  pads.
> >  
> >  Subdevice drivers that support multiplexed streams are compatible with
> > -non-multiplexed subdev drivers, but, of course, require a routing configuration
> > -where the link between those two types of drivers contains only a single
> > -stream.
> > +non-multiplexed subdev drivers. However, if the driver at the sink end of a link
> > +does not support streams, then only the stream 0 on source end may be
> 
> s/the stream 0 on source end/stream 0 of the source/
> 
> > +captured. There may be additional hardware specific limitations.
> 
> s/hardware specific/hardware-specific/
> 
> Or maybe
> 
> There may be additional limitations specific to the sink device.

Sounds good.

> 
> Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

Thanks!

> 
> >  
> >  Understanding streams
> >  ^^^^^^^^^^^^^^^^^^^^^
> 

-- 
Kind regards,

Sakari Ailus

Re: [PATCH v8 14/38] media: v4l: subdev: Add len_routes field to struct v4l2_subdev_routing

[Sakari Ailus]

Hi Laurent,

On Wed, Mar 20, 2024 at 03:36:35AM +0200, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.
> 
> On Wed, Mar 13, 2024 at 09:24:52AM +0200, Sakari Ailus wrote:
> > The len_routes field is used to tell the size of the routes array in
> > struct v4l2_subdev_routing. This way the number of routes returned from
> > S_ROUTING IOCTL may be larger than the number of routes provided, in case
> > there are more routes returned by the driver.
> > 
> > Note that this uAPI is still disabled in the code, so this change can
> > safely be done. Anyone who manually patched the code to enable this uAPI
> > must update their code.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  .../media/v4l/vidioc-subdev-g-routing.rst     | 33 ++++++++++++-------
> >  drivers/media/v4l2-core/v4l2-ioctl.c          |  4 +--
> >  drivers/media/v4l2-core/v4l2-subdev.c         |  6 +++-
> >  include/media/v4l2-subdev.h                   |  2 ++
> >  include/uapi/linux/v4l2-subdev.h              |  9 +++--
> >  5 files changed, 37 insertions(+), 17 deletions(-)
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > index 26b5004bfe6d..8f9aa83275c5 100644
> > --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > @@ -46,20 +46,28 @@ with the ``VIDIOC_SUBDEV_S_ROUTING`` ioctl, by adding or removing routes and
> >  setting or clearing flags of the  ``flags`` field of a
> >  struct :c:type:`v4l2_subdev_route`.
> >  
> > -All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called. This
> > -means that the userspace must reconfigure all streams after calling the ioctl
> > -with e.g. ``VIDIOC_SUBDEV_S_FMT``.
> > +All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is
> > +called. This means that the userspace must reconfigure all streams after calling
> > +the ioctl with e.g. ``VIDIOC_SUBDEV_S_FMT``.
> 
> Unrelated. If you really want to reflow, the first line is too short,
> 'called.' should be moved there.

Seems to be, indeed. This would look like a bug in an Emacs major mode. :-I

> 
> >  
> >  Only subdevices which have both sink and source pads can support routing.
> >  
> > -When inspecting routes through ``VIDIOC_SUBDEV_G_ROUTING`` and the application
> > -provided ``num_routes`` is not big enough to contain all the available routes
> > -the subdevice exposes, drivers return the ENOSPC error code and adjust the
> > -value of the ``num_routes`` field. Application should then reserve enough memory
> > -for all the route entries and call ``VIDIOC_SUBDEV_G_ROUTING`` again.
> > +The ``num_routes`` field is used to denote the number of routes set (set by user
> > +space on ``VIDIOC_SUBDEV_S_ROUTING`` argument) on the routing table as well as
> > +the number of routes returned back from both IOCTLs. ``len_routes`` signifies
> > +the number of routes that can fit into the ``routes`` array. The userspace shall
> > +set ``len_routes`` for both IOCTLs and ``num_routes`` for
> > +``VIDIOC_SUBDEV_S_ROUTING``.
> >  
> > -On a successful ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the
> > -``num_routes`` field to reflect the actual number of routes returned.
> > +On a ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the ``num_routes``
> > +field to reflect the actual number of routes known by the driver.
> > +``num_routes`` larger than ``len_routes`` may be returned by both IOCTLs to
> > +indicate there are more routes than fits to the ``routes`` array. In this
> > +case first ``len_routes`` were returned back to the userspace in the
> > +``routes`` array. This is not considered as an error.
> > +
> > +Also ``VIDIOC_SUBDEV_S_ROUTING`` may return more route than the user provided in
> > +``num_routes`` field due to e.g. hardware properties.
> >  
> 
> An attempt at making this clearer (in my opinion, feel free to pick the
> pieces you like, or reject it outright).
> 
> 
> The ``len_routes`` field indicates the number of routes that can fit in the
> ``routes`` array allocated by userspace. It is set by applications for both
> ioctls to indicate how many routes the kernel can return, and is never modified
> by the kernel.
> 
> The ``num_routes`` field indicates the number of routes in the subdevice routing
> table. For ``VIDIOC_SUBDEV_S_ROUTING``, it is set by userspace to the number of

For S_ROUTING this is the number of routes in the IOCTL argument. The
routing table may contain more (static routes).

> routes that the application stored in the ``routes`` array. For both ioctls, it
> is returned by the kernel and indicates how many routes are stored in the
> subdevice routing table. This may be smaller or larger than the value of
> ``num_routes`` set by the application for ``VIDIOC_SUBDEV_S_ROUTING``, as
> drivers may adjust the requested routing table.

How about this (first sentence modified):

The ``num_routes`` field, when returned from the kernel on both IOCTLs,
indicates the number of routes in the subdevice routing table and when calling
``VIDIOC_SUBDEV_S_ROUTING``, it is set by userspace to the number of routes that
the application stored in the ``routes`` array. The value returned by the kernel
may be smaller or larger than the value of ``num_routes`` set by the application
for ``VIDIOC_SUBDEV_S_ROUTING``, as drivers may adjust the requested routing
table.

> 
> The kernel can return a ``num_routes`` value larger than ``len_routes`` from
> both ioctls. This indicates thare are more routes in the routing table than fits
> the ``routes`` array. In this case, the ``routes`` array is filled by the kernel
> with the first ``len_routes`` entries of the subdevice routing table. This is
> not considered an array, and the ioctl call succeeds. If the applications wants

Too many arrays? :-)

s/not considered \Kan array/to be an error/

> to retrieve the missing routes, it can issue a new ``VIDIOC_SUBDEV_G_ROUTING``
> call with a large enough ``routes`` array.
> 
> >  .. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
> >  
> > @@ -74,6 +82,9 @@ On a successful ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the
> >        - ``which``
> >        - Routing table to be accessed, from enum
> >          :ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
> > +    * - __u32
> > +      - ``len_routes``
> > +      - The length of the array (as in memory reserved for the array)
> >      * - struct :c:type:`v4l2_subdev_route`
> >        - ``routes[]``
> >        - Array of struct :c:type:`v4l2_subdev_route` entries
> > @@ -81,7 +92,7 @@ On a successful ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the
> >        - ``num_routes``
> >        - Number of entries of the routes array
> >      * - __u32
> > -      - ``reserved``\ [5]
> > +      - ``reserved``\ [11]
> >        - Reserved for future extensions. Applications and drivers must set
> >  	the array to zero.
> >  
> > diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> > index 95bd56145d38..018e7efc21ca 100644
> > --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> > +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> > @@ -3155,13 +3155,13 @@ static int check_array_args(unsigned int cmd, void *parg, size_t *array_size,
> >  	case VIDIOC_SUBDEV_S_ROUTING: {
> >  		struct v4l2_subdev_routing *routing = parg;
> >  
> > -		if (routing->num_routes > 256)
> > +		if (routing->len_routes > 256)
> >  			return -E2BIG;
> >  
> >  		*user_ptr = u64_to_user_ptr(routing->routes);
> >  		*kernel_ptr = (void **)&routing->routes;
> >  		*array_size = sizeof(struct v4l2_subdev_route)
> > -			    * routing->num_routes;
> > +			    * routing->len_routes;
> >  		ret = 1;
> >  		break;
> >  	}
> > diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> > index ec8de3f872b9..486a5c3016de 100644
> > --- a/drivers/media/v4l2-core/v4l2-subdev.c
> > +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> > @@ -925,6 +925,9 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
> >  		if (routing->which != V4L2_SUBDEV_FORMAT_TRY && ro_subdev)
> >  			return -EPERM;
> >  
> > +		if (routing->num_routes > routing->len_routes)
> > +			return -EINVAL;
> > +
> >  		memset(routing->reserved, 0, sizeof(routing->reserved));
> >  
> >  		for (i = 0; i < routing->num_routes; ++i) {
> > @@ -951,6 +954,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
> >  		}
> >  
> >  		krouting.num_routes = routing->num_routes;
> > +		krouting.len_routes = routing->len_routes;
> >  		krouting.routes = routes;
> >  
> >  		return v4l2_subdev_call(sd, pad, set_routing, state,
> > @@ -971,7 +975,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
> >  
> >  		krouting = &state->routing;
> >  
> > -		if (routing->num_routes < krouting->num_routes) {
> > +		if (routing->len_routes < krouting->num_routes) {
> >  			routing->num_routes = krouting->num_routes;
> >  			return -ENOSPC;
> 
> Do we still want to return -ENOSPC here ?

Indeed, this part should have been changed, too.

The code in the case would then become:

		struct v4l2_subdev_routing *routing = arg;
		struct v4l2_subdev_krouting *krouting;

		if (!v4l2_subdev_enable_streams_api)
			return -ENOIOCTLCMD;

		if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS))
			return -ENOIOCTLCMD;

		memset(routing->reserved, 0, sizeof(routing->reserved));

		krouting = &state->routing;

		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
		       krouting->routes,
		       min(krouting->num_routes, routing->len_routes) *
		       sizeof(*krouting->routes));
		routing->num_routes = krouting->num_routes;

		return 0;

> 
> >  		}
> > diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> > index 9cce48365975..1df6b963a1c9 100644
> > --- a/include/media/v4l2-subdev.h
> > +++ b/include/media/v4l2-subdev.h
> > @@ -728,12 +728,14 @@ struct v4l2_subdev_stream_configs {
> >  /**
> >   * struct v4l2_subdev_krouting - subdev routing table
> >   *
> > + * @len_routes: length of routes array, in routes
> >   * @num_routes: number of routes
> >   * @routes: &struct v4l2_subdev_route
> >   *
> >   * This structure contains the routing table for a subdev.
> >   */
> >  struct v4l2_subdev_krouting {
> > +	unsigned int len_routes;
> >  	unsigned int num_routes;
> >  	struct v4l2_subdev_route *routes;
> >  };
> > diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
> > index 7048c51581c6..ca543982460c 100644
> > --- a/include/uapi/linux/v4l2-subdev.h
> > +++ b/include/uapi/linux/v4l2-subdev.h
> > @@ -224,15 +224,18 @@ struct v4l2_subdev_route {
> >   * struct v4l2_subdev_routing - Subdev routing information
> >   *
> >   * @which: configuration type (from enum v4l2_subdev_format_whence)
> > - * @num_routes: the total number of routes in the routes array
> > + * @len_routes: the length of the routes array, in routes
> >   * @routes: pointer to the routes array
> > + * @num_routes: the total number of routes, possibly more than fits in the
> > + *		routes array
> >   * @reserved: drivers and applications must zero this array
> >   */
> >  struct v4l2_subdev_routing {
> >  	__u32 which;
> > -	__u32 num_routes;
> > +	__u32 len_routes;
> >  	__u64 routes;
> > -	__u32 reserved[6];
> > +	__u32 num_routes;
> > +	__u32 reserved[11];
> >  };
> >  
> >  /*
> 

-- 
Kind regards,

Sakari Ailus

Re: [PATCH v8 15/38] media: v4l: subdev: Return routes set using S_ROUTING

[Sakari Ailus]

On Wed, Mar 20, 2024 at 03:45:14AM +0200, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.
> 
> On Wed, Mar 13, 2024 at 09:24:53AM +0200, Sakari Ailus wrote:
> > Return the routes set using S_ROUTING back to the user. Also reflect this
> > in documentation.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  .../media/v4l/vidioc-subdev-g-routing.rst            |  5 +++--
> >  drivers/media/v4l2-core/v4l2-subdev.c                | 12 +++++++++++-
> >  2 files changed, 14 insertions(+), 3 deletions(-)
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > index 8f9aa83275c5..6eb6a59570dc 100644
> > --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > @@ -43,8 +43,9 @@ The routing configuration determines the flows of data inside an entity.
> >  Drivers report their current routing tables using the
> >  ``VIDIOC_SUBDEV_G_ROUTING`` ioctl and application may enable or disable routes
> >  with the ``VIDIOC_SUBDEV_S_ROUTING`` ioctl, by adding or removing routes and
> > -setting or clearing flags of the  ``flags`` field of a
> > -struct :c:type:`v4l2_subdev_route`.
> > +setting or clearing flags of the ``flags`` field of a struct
> > +:c:type:`v4l2_subdev_route`. Similarly to ``VIDIOC_SUBDEV_G_ROUTING``, also
> > +``VIDIOC_SUBDEV_S_ROUTING`` returns the routes back to the user.
> >  
> >  All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is
> >  called. This means that the userspace must reconfigure all streams after calling
> > diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> > index 486a5c3016de..a357ce318192 100644
> > --- a/drivers/media/v4l2-core/v4l2-subdev.c
> > +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> > @@ -957,8 +957,18 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
> >  		krouting.len_routes = routing->len_routes;
> >  		krouting.routes = routes;
> >  
> > -		return v4l2_subdev_call(sd, pad, set_routing, state,
> > +		rval = v4l2_subdev_call(sd, pad, set_routing, state,
> >  					routing->which, &krouting);
> > +		if (rval < 0)
> > +			return rval;
> > +
> > +		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
> > +		       state->routing.routes,
> > +		       state->routing.num_routes *
> 
> I think this should be capped at len_routes.

Good catch!

> 
> > +		       sizeof(*state->routing.routes));
> > +		routing->num_routes = state->routing.num_routes;
> > +
> > +		return 0;
> >  	}
> >  
> >  	case VIDIOC_SUBDEV_G_ROUTING: {
> > -- 
> > 2.39.2
> > 
> 
> -- 
> Regards,
> 
> Laurent Pinchart

-- 
Sakari Ailus

Re: [PATCH v8 25/38] media: ccs: Remove ccs_get_crop_compose helper

[Sakari Ailus]

Hi Laurent,

On Thu, Mar 21, 2024 at 08:05:15PM +0200, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.
> 
> On Wed, Mar 13, 2024 at 09:25:03AM +0200, Sakari Ailus wrote:
> > As it's now easier to obtain the necessary information on crop and compose
> > rectangles after moving to sub-device state, remove the
> > ccs_get_crop_compose helper.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  drivers/media/i2c/ccs/ccs-core.c | 130 ++++++++++++++-----------------
> >  1 file changed, 58 insertions(+), 72 deletions(-)
> > 
> > diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
> > index 1f91af25eac6..6c7eb4f04070 100644
> > --- a/drivers/media/i2c/ccs/ccs-core.c
> > +++ b/drivers/media/i2c/ccs/ccs-core.c
> > @@ -2222,24 +2222,6 @@ static int ccs_get_format(struct v4l2_subdev *subdev,
> >  	return rval;
> >  }
> >  
> > -static void ccs_get_crop_compose(struct v4l2_subdev *subdev,
> > -				 struct v4l2_subdev_state *sd_state,
> > -				 struct v4l2_rect **crops,
> > -				 struct v4l2_rect **comps)
> > -{
> > -	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
> > -	unsigned int i;
> > -
> > -	if (crops)
> > -		for (i = 0; i < subdev->entity.num_pads; i++)
> > -			crops[i] =
> > -				v4l2_subdev_state_get_crop(sd_state, i,
> > -							   CCS_STREAM_PIXEL);
> > -	if (comps)
> > -		*comps = v4l2_subdev_state_get_compose(sd_state, ssd->sink_pad,
> > -						       CCS_STREAM_PIXEL);
> > -}
> > -
> >  /* Changes require propagation only on sink pad. */
> >  static void ccs_propagate(struct v4l2_subdev *subdev,
> >  			  struct v4l2_subdev_state *sd_state, int which,
> > @@ -2247,15 +2229,17 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
> >  {
> >  	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
> >  	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
> > -	struct v4l2_rect *comp, *crops[CCS_PADS];
> > +	struct v4l2_rect *comp, *crop;
> >  	struct v4l2_mbus_framefmt *fmt;
> >  
> > -	ccs_get_crop_compose(subdev, sd_state, crops, &comp);
> > -
> > +	comp = v4l2_subdev_state_get_compose(sd_state, ssd->sink_pad,
> > +					     CCS_STREAM_PIXEL);
> >  	switch (target) {
> >  	case V4L2_SEL_TGT_CROP:
> > -		comp->width = crops[CCS_PAD_SINK]->width;
> > -		comp->height = crops[CCS_PAD_SINK]->height;
> > +		crop = v4l2_subdev_state_get_crop(sd_state, CCS_PAD_SINK,
> > +						  CCS_STREAM_PIXEL);
> > +		comp->width = crop->width;
> > +		comp->height = crop->height;
> >  		if (which == V4L2_SUBDEV_FORMAT_ACTIVE) {
> >  			if (ssd == sensor->scaler) {
> >  				sensor->scale_m = CCS_LIM(sensor, SCALER_N_MIN);
> > @@ -2269,13 +2253,15 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
> >  		}
> >  		fallthrough;
> >  	case V4L2_SEL_TGT_COMPOSE:
> > -		*crops[CCS_PAD_SRC] = *comp;
> > +		crop = v4l2_subdev_state_get_crop(sd_state, CCS_PAD_SRC,
> > +						  CCS_STREAM_PIXEL);
> > +		*crop = *comp;
> >  		fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC,
> >  						   CCS_STREAM_PIXEL);
> >  		fmt->width = comp->width;
> >  		fmt->height = comp->height;
> >  		if (which == V4L2_SUBDEV_FORMAT_ACTIVE && ssd == sensor->src)
> > -			sensor->src_src = *crops[CCS_PAD_SRC];
> > +			sensor->src_src = *crop;
> >  		break;
> >  	default:
> >  		WARN_ON_ONCE(1);
> > @@ -2416,7 +2402,7 @@ static int ccs_set_format(struct v4l2_subdev *subdev,
> >  {
> >  	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
> >  	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
> > -	struct v4l2_rect *crops[CCS_PADS];
> > +	struct v4l2_rect *crop;
> >  
> >  	if (subdev == &sensor->src->sd && fmt->pad == CCS_PAD_META)
> >  		return ccs_get_format(subdev, sd_state, fmt);
> > @@ -2458,12 +2444,13 @@ static int ccs_set_format(struct v4l2_subdev *subdev,
> >  		      CCS_LIM(sensor, MIN_Y_OUTPUT_SIZE),
> >  		      CCS_LIM(sensor, MAX_Y_OUTPUT_SIZE));
> >  
> > -	ccs_get_crop_compose(subdev, sd_state, crops, NULL);
> > +	crop = v4l2_subdev_state_get_crop(sd_state, ssd->sink_pad,
> > +					  CCS_STREAM_PIXEL);
> >  
> > -	crops[ssd->sink_pad]->left = 0;
> > -	crops[ssd->sink_pad]->top = 0;
> > -	crops[ssd->sink_pad]->width = fmt->format.width;
> > -	crops[ssd->sink_pad]->height = fmt->format.height;
> > +	crop->left = 0;
> > +	crop->top = 0;
> > +	crop->width = fmt->format.width;
> > +	crop->height = fmt->format.height;
> >  	ccs_propagate(subdev, sd_state, fmt->which, V4L2_SEL_TGT_CROP);
> >  
> >  	mutex_unlock(&sensor->mutex);
> > @@ -2518,24 +2505,23 @@ static int scaling_goodness(struct v4l2_subdev *subdev, int w, int ask_w,
> >  static void ccs_set_compose_binner(struct v4l2_subdev *subdev,
> >  				   struct v4l2_subdev_state *sd_state,
> >  				   struct v4l2_subdev_selection *sel,
> > -				   struct v4l2_rect **crops,
> > +				   struct v4l2_rect *sink_crop,
> 
> const

Yes. Also it seems the comp argument can be dropped.

> 
> >  				   struct v4l2_rect *comp)
> >  {
> >  	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
> >  	unsigned int i;
> >  	unsigned int binh = 1, binv = 1;
> >  	int best = scaling_goodness(
> > -		subdev,
> > -		crops[CCS_PAD_SINK]->width, sel->r.width,
> > -		crops[CCS_PAD_SINK]->height, sel->r.height, sel->flags);
> > +		subdev,	sink_crop->width, sel->r.width,
> 
> s/\tsink_crop/ sink_crop/

Yes.

> 
> > +		sink_crop->height, sel->r.height, sel->flags);
> 
> 	int best = scaling_goodness(subdev, sink_crop->width, sel->r.width,
> 				    sink_crop->height, sel->r.height,
> 				    sel->flags);
> 
> >  
> >  	for (i = 0; i < sensor->nbinning_subtypes; i++) {
> >  		int this = scaling_goodness(
> >  			subdev,
> > -			crops[CCS_PAD_SINK]->width
> > +			sink_crop->width
> >  			/ sensor->binning_subtypes[i].horizontal,
> >  			sel->r.width,
> > -			crops[CCS_PAD_SINK]->height
> > +			sink_crop->height
> >  			/ sensor->binning_subtypes[i].vertical,
> >  			sel->r.height, sel->flags);
> 
> You know this is horrible to read, right ? :-)

I'll rewrap the calls to start the lines on the opening parenthesis.

> 
> >  
> > @@ -2550,8 +2536,8 @@ static void ccs_set_compose_binner(struct v4l2_subdev *subdev,
> >  		sensor->binning_horizontal = binh;
> >  	}
> >  
> > -	sel->r.width = (crops[CCS_PAD_SINK]->width / binh) & ~1;
> > -	sel->r.height = (crops[CCS_PAD_SINK]->height / binv) & ~1;
> > +	sel->r.width = (sink_crop->width / binh) & ~1;
> > +	sel->r.height = (sink_crop->height / binv) & ~1;
> >  }
> >  
> >  /*
> > @@ -2566,7 +2552,7 @@ static void ccs_set_compose_binner(struct v4l2_subdev *subdev,
> >  static void ccs_set_compose_scaler(struct v4l2_subdev *subdev,
> >  				   struct v4l2_subdev_state *sd_state,
> >  				   struct v4l2_subdev_selection *sel,
> > -				   struct v4l2_rect **crops,
> > +				   struct v4l2_rect *sink_crop,
> 
> const

Yes.

> 
> >  				   struct v4l2_rect *comp)
> >  {
> >  	struct i2c_client *client = v4l2_get_subdevdata(subdev);
> > @@ -2579,17 +2565,12 @@ static void ccs_set_compose_scaler(struct v4l2_subdev *subdev,
> >  	unsigned int i;
> >  	int best = INT_MIN;
> >  
> > -	sel->r.width = min_t(unsigned int, sel->r.width,
> > -			     crops[CCS_PAD_SINK]->width);
> > -	sel->r.height = min_t(unsigned int, sel->r.height,
> > -			      crops[CCS_PAD_SINK]->height);
> > -
> > -	a = crops[CCS_PAD_SINK]->width
> > -		* CCS_LIM(sensor, SCALER_N_MIN) / sel->r.width;
> > -	b = crops[CCS_PAD_SINK]->height
> > -		* CCS_LIM(sensor, SCALER_N_MIN) / sel->r.height;
> > -	max_m = crops[CCS_PAD_SINK]->width
> > -		* CCS_LIM(sensor, SCALER_N_MIN)
> > +	sel->r.width = min_t(unsigned int, sel->r.width, sink_crop->width);
> > +	sel->r.height = min_t(unsigned int, sel->r.height, sink_crop->height);
> > +
> > +	a = sink_crop->width * CCS_LIM(sensor, SCALER_N_MIN) / sel->r.width;
> > +	b = sink_crop->height * CCS_LIM(sensor, SCALER_N_MIN) / sel->r.height;
> > +	max_m = sink_crop->width * CCS_LIM(sensor, SCALER_N_MIN)
> >  		/ CCS_LIM(sensor, MIN_X_OUTPUT_SIZE);
> >  
> >  	a = clamp(a, CCS_LIM(sensor, SCALER_M_MIN),
> > @@ -2622,10 +2603,10 @@ static void ccs_set_compose_scaler(struct v4l2_subdev *subdev,
> >  	for (i = 0; i < ntry; i++) {
> >  		int this = scaling_goodness(
> >  			subdev,
> > -			crops[CCS_PAD_SINK]->width
> > +			sink_crop->width
> >  			/ try[i] * CCS_LIM(sensor, SCALER_N_MIN),
> >  			sel->r.width,
> > -			crops[CCS_PAD_SINK]->height,
> > +			sink_crop->height,
> >  			sel->r.height,
> >  			sel->flags);
> >  
> > @@ -2642,12 +2623,10 @@ static void ccs_set_compose_scaler(struct v4l2_subdev *subdev,
> >  			continue;
> >  
> >  		this = scaling_goodness(
> > -			subdev, crops[CCS_PAD_SINK]->width
> > -			/ try[i]
> > +			subdev, sink_crop->width / try[i]
> >  			* CCS_LIM(sensor, SCALER_N_MIN),
> >  			sel->r.width,
> > -			crops[CCS_PAD_SINK]->height
> > -			/ try[i]
> > +			sink_crop->height / try[i]
> >  			* CCS_LIM(sensor, SCALER_N_MIN),
> >  			sel->r.height,
> >  			sel->flags);
> > @@ -2660,17 +2639,15 @@ static void ccs_set_compose_scaler(struct v4l2_subdev *subdev,
> >  	}
> >  
> >  	sel->r.width =
> > -		(crops[CCS_PAD_SINK]->width
> > -		 / scale_m
> > +		(sink_crop->width / scale_m
> >  		 * CCS_LIM(sensor, SCALER_N_MIN)) & ~1;
> 
> 	sel->r.width = (sink_crop->width / scale_m
> 		* CCS_LIM(sensor, SCALER_N_MIN)) & ~1;
> 
> >  	if (mode == SMIAPP_SCALING_MODE_BOTH)
> >  		sel->r.height =
> > -			(crops[CCS_PAD_SINK]->height
> > -			 / scale_m
> > +			(sink_crop->height / scale_m
> >  			 * CCS_LIM(sensor, SCALER_N_MIN))
> >  			& ~1;
> >  	else
> > -		sel->r.height = crops[CCS_PAD_SINK]->height;
> > +		sel->r.height = sink_crop->height;
> >  
> >  	if (sel->which == V4L2_SUBDEV_FORMAT_ACTIVE) {
> >  		sensor->scale_m = scale_m;
> > @@ -2684,17 +2661,20 @@ static int ccs_set_compose(struct v4l2_subdev *subdev,
> >  {
> >  	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
> >  	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
> > -	struct v4l2_rect *comp, *crops[CCS_PADS];
> > +	struct v4l2_rect *comp, *sink_crop;
> 
> sink_crop can be const.
> 
> >  
> > -	ccs_get_crop_compose(subdev, sd_state, crops, &comp);
> > +	sink_crop = v4l2_subdev_state_get_crop(sd_state, CCS_PAD_SINK,
> > +					       CCS_STREAM_PIXEL);
> > +	comp = v4l2_subdev_state_get_compose(sd_state, ssd->sink_pad,
> > +					     CCS_STREAM_PIXEL);
> >  
> >  	sel->r.top = 0;
> >  	sel->r.left = 0;
> >  
> >  	if (ssd == sensor->binner)
> > -		ccs_set_compose_binner(subdev, sd_state, sel, crops, comp);
> > +		ccs_set_compose_binner(subdev, sd_state, sel, sink_crop, comp);
> >  	else
> > -		ccs_set_compose_scaler(subdev, sd_state, sel, crops, comp);
> > +		ccs_set_compose_scaler(subdev, sd_state, sel, sink_crop, comp);
> >  
> >  	*comp = sel->r;
> >  	ccs_propagate(subdev, sd_state, sel->which, V4L2_SEL_TGT_COMPOSE);
> > @@ -2755,9 +2735,12 @@ static int ccs_set_crop(struct v4l2_subdev *subdev,
> >  {
> >  	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
> >  	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
> > -	struct v4l2_rect src_size = { 0 }, *crops[CCS_PADS], *comp;
> > +	struct v4l2_rect src_size = { 0 }, *crop, *comp;
> 
> comp can be const.
> 
> >  
> > -	ccs_get_crop_compose(subdev, sd_state, crops, &comp);
> > +	crop = v4l2_subdev_state_get_crop(sd_state, sel->pad,
> > +					  CCS_STREAM_PIXEL);
> > +	comp = v4l2_subdev_state_get_compose(sd_state, ssd->sink_pad,
> > +					     CCS_STREAM_PIXEL);
> >  
> >  	if (sel->pad == ssd->sink_pad) {
> >  		struct v4l2_mbus_framefmt *mfmt =
> > @@ -2781,7 +2764,7 @@ static int ccs_set_crop(struct v4l2_subdev *subdev,
> >  	sel->r.left = min_t(int, sel->r.left, src_size.width - sel->r.width);
> >  	sel->r.top = min_t(int, sel->r.top, src_size.height - sel->r.height);
> >  
> > -	*crops[sel->pad] = sel->r;
> > +	*crop = sel->r;
> >  
> >  	if (ssd != sensor->pixel_array && sel->pad == CCS_PAD_SINK)
> >  		ccs_propagate(subdev, sd_state, sel->which, V4L2_SEL_TGT_CROP);
> > @@ -2806,14 +2789,17 @@ static int ccs_get_selection(struct v4l2_subdev *subdev,
> >  {
> >  	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
> >  	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
> > -	struct v4l2_rect *comp, *crops[CCS_PADS];
> > +	struct v4l2_rect *crop, *comp;
> 
> const
> 
> >  	int ret;
> >  
> >  	ret = ccs_sel_supported(subdev, sel);
> >  	if (ret)
> >  		return ret;
> >  
> > -	ccs_get_crop_compose(subdev, sd_state, crops, &comp);
> > +	crop = v4l2_subdev_state_get_crop(sd_state, sel->pad,
> > +					  CCS_STREAM_PIXEL);
> > +	comp = v4l2_subdev_state_get_compose(sd_state, ssd->sink_pad,
> > +					     CCS_STREAM_PIXEL);
> >  
> >  	switch (sel->target) {
> >  	case V4L2_SEL_TGT_CROP_BOUNDS:
> > @@ -2835,7 +2821,7 @@ static int ccs_get_selection(struct v4l2_subdev *subdev,
> >  		break;
> >  	case V4L2_SEL_TGT_CROP:
> >  	case V4L2_SEL_TGT_COMPOSE_BOUNDS:
> > -		sel->r = *crops[sel->pad];
> > +		sel->r = *crop;
> >  		break;
> >  	case V4L2_SEL_TGT_COMPOSE:
> >  		sel->r = *comp;
> 

-- 
Regards,

Sakari Ailus

Re: [PATCH v8 30/38] media: Documentation: ccs: Document routing

[Sakari Ailus]

Hi Laurent,

On Thu, Mar 21, 2024 at 07:43:06PM +0200, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.

Thank you for the review!

> 
> On Wed, Mar 13, 2024 at 09:25:08AM +0200, Sakari Ailus wrote:
> > Document which routes are available for the CCS driver (source) sub-device
> > and what configuration are possible.
> > 
> > Also update copyright.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  .../userspace-api/media/drivers/ccs.rst       | 34 ++++++++++++++++++-
> >  .../media/v4l/subdev-formats.rst              |  2 ++
> >  2 files changed, 35 insertions(+), 1 deletion(-)
> > 
> > diff --git a/Documentation/userspace-api/media/drivers/ccs.rst b/Documentation/userspace-api/media/drivers/ccs.rst
> > index 03015b33d5ab..bc2804ec663b 100644
> > --- a/Documentation/userspace-api/media/drivers/ccs.rst
> > +++ b/Documentation/userspace-api/media/drivers/ccs.rst
> > @@ -111,4 +111,36 @@ than in the centre.
> >  Shading correction needs to be enabled for luminance correction level to have an
> >  effect.
> >  
> > -**Copyright** |copy| 2020 Intel Corporation
> > +.. _media-ccs-routes:
> > +
> > +Routes
> > +------
> > +
> > +The CCS driver implements one or two :ref:`routes <subdev-routing>` in
> > +its source sub-device (scaler sub-device if exists for the device, otherwise
> > +binner) depending on whether the sensor supports embedded data. (All CCS
> > +compliant sensors do but the CCS driver supports preceding standards that did
> > +not require embedded data support, too.)
> 
> s/too.)/too).

Hmm. The entire sentence is in parentheses. Why should the period not be?

> 
> > +
> > +The first route of the CCS source sub-device is for pixel data (internal pad
> > +1/stream 0 -> pad 0/stream 0) and the second one is for embedded data (internal
> 
> I thought the source subdev had one external sink pad for image data,
> one internal sink pad for embedded data, and one source pad.

Hmm. I guess I was thinking of IMX219 when writing this. ;)

The text should have been:

The first route of the CCS source sub-device is for pixel data (sink pad
0/stream 0 -> source pad 1/stream 0) and the second one is for embedded data
(internal sink pad 2/stream 0 -> source pad 1/stream 1).


> 
> > +pad 2/stream 0 -> pad 0/stream 1).
> > +
> > +Embedded data
> > +~~~~~~~~~~~~~
> > +
> > +MIPI CCS supports generation of camera sensor embedded data. The media bus code
> > +used for this format is :ref:`MEDIA_BUS_FMT_CCS_EMBEDDDED
> > +<MEDIA-BUS-FMT-CCS-EMBEDDED>`.
> 
> Maybe state that this is the format on the internal ED sink pad.

Yes.

> 
> > +
> > +The bit depth of the CCS pixel data affects how the sensor will output the
> > +embedded data, adding padding to align with CSI-2 bus :ref:`Data units
> > +<media-glossary-data-unit>` for that particular bit depth. This is indicated by
> > +the generic metadata format on the sensor's source sub-device's source pad.
> > +
> > +Embedded data for bit depths greater than or equal to 16 may support more dense
> > +packing or legacy single metadata byte per data unit, or both of these,
> > +depending on the device. The supported embedded data formats can be enumerated
> > +and configured on stream 1 of the source pad (1) of the CCS source sub-device.
> > +
> > +**Copyright** |copy| 2020, 2023 Intel Corporation
> 
> 2024 ?

Indeed.

> 
> > diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > index c8f982411e70..ca4da6a400ff 100644
> > --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > @@ -8592,3 +8592,5 @@ levels above.
> >  This mbus code are only used for "2-byte simplified tagged data format" (code
> >  0xa) but their use may be extended further in the future, to cover other CCS
> >  embedded data format codes.
> > +
> > +Also see :ref:`CCS driver documentation <media-ccs-routes>`.
> 

-- 
Kind regards,

Sakari Ailus

Re: [PATCH v8 28/38] media: ccs: Compute scaling configuration from sub-device state

[Sakari Ailus]

Hi Laurent,

On Thu, Mar 21, 2024 at 07:50:04PM +0200, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.
> 
> On Wed, Mar 13, 2024 at 09:25:06AM +0200, Sakari Ailus wrote:
> > Compute scaling configuration from sub-device state instead of storing it
> > to the driver's device context struct.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  drivers/media/i2c/ccs/ccs-core.c | 60 ++++++++++++++++++++++----------
> >  drivers/media/i2c/ccs/ccs.h      |  3 --
> >  2 files changed, 41 insertions(+), 22 deletions(-)
> > 
> > diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
> > index 3b80c54453cc..a147dbb9f362 100644
> > --- a/drivers/media/i2c/ccs/ccs-core.c
> > +++ b/drivers/media/i2c/ccs/ccs-core.c
> > @@ -547,19 +547,52 @@ ccs_get_binning(struct ccs_sensor *sensor, u8 *binning_mode, u8 *binh, u8 *binv)
> >  		*binv = binner_sink_crop->height / binner_sink_comp->height;
> >  }
> >  
> > +static void ccs_get_scaling(struct ccs_sensor *sensor,
> > +			    u8 *scaling_mode, u8 *scale_m)
> > +{
> > +	struct v4l2_subdev_state *scaler_state =
> > +		v4l2_subdev_get_locked_active_state(&sensor->scaler->sd);
> 
> Have you double-checked that the scaler state is locked in all code
> paths that can lead to this ? The function is called at probe time, with
> a manual lock of sensor->mutex. That makes me a bit uncomfortable, I
> wonder if it wouldn't be better to pass the locked scaler state to the
> function explicitly, to let the callers guarantee the locking
> requirements.

Given that there's a single mutex and v4l2_subdev_get_locked_active_state()
also uses lockdep_assert_held(), I'm not really concerned.

> 
> > +	struct v4l2_rect *scaler_sink_crop =
> 
> const
> 
> I would also drop the scaler_ prefix here and for the next variable.

Works for me.

> 
> > +		v4l2_subdev_state_get_crop(scaler_state, CCS_PAD_SINK,
> > +					   CCS_STREAM_PIXEL);
> > +	struct v4l2_rect *scaler_sink_comp =
> > +		v4l2_subdev_state_get_compose(scaler_state, CCS_PAD_SINK,
> > +					      CCS_STREAM_PIXEL);
> > +
> > +	if (scale_m)
> > +		*scale_m = scaler_sink_crop->width *
> > +			CCS_LIM(sensor, SCALER_N_MIN) /
> > +			scaler_sink_comp->width;
> > +
> > +	if (scaling_mode) {
> > +		if (scaler_sink_crop->width == scaler_sink_comp->width)
> > +			*scaling_mode = CCS_SCALING_MODE_NO_SCALING;
> > +		else if (scaler_sink_crop->height == scaler_sink_comp->height)
> > +			*scaling_mode = CCS_SCALING_MODE_HORIZONTAL;
> > +		else
> > +			*scaling_mode = SMIAPP_SCALING_MODE_BOTH;
> > +	}
> > +}
> > +
> >  static int ccs_pll_update(struct ccs_sensor *sensor)
> >  {
> >  	struct ccs_pll *pll = &sensor->pll;
> >  	u8 binh, binv;
> > +	u8 scale_m;
> >  	int rval;
> >  
> >  	ccs_get_binning(sensor, NULL, &binh, &binv);
> >  
> > +	if (sensor->scaler)
> > +		ccs_get_scaling(sensor, NULL, &scale_m);
> > +	else
> > +		scale_m = CCS_LIM(sensor, SCALER_N_MIN);
> > +
> >  	pll->binning_horizontal = binh;
> >  	pll->binning_vertical = binv;
> >  	pll->link_freq =
> >  		sensor->link_freq->qmenu_int[sensor->link_freq->val];
> > -	pll->scale_m = sensor->scale_m;
> > +	pll->scale_m = scale_m;
> >  	pll->bits_per_pixel = sensor->csi_format->compressed;
> >  
> >  	rval = ccs_pll_try(sensor, pll);
> > @@ -1202,7 +1235,7 @@ static int ccs_get_mbus_formats(struct ccs_sensor *sensor)
> >  	/* Figure out which BPP values can be used with which formats. */
> >  	pll->binning_horizontal = 1;
> >  	pll->binning_vertical = 1;
> > -	pll->scale_m = sensor->scale_m;
> > +	pll->scale_m = CCS_LIM(sensor, SCALER_N_MIN);
> >  
> >  	for (i = 0; i < ARRAY_SIZE(ccs_csi_data_formats); i++) {
> >  		sensor->compressed_min_bpp =
> > @@ -1950,11 +1983,15 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
> >  	/* Scaling */
> >  	if (CCS_LIM(sensor, SCALING_CAPABILITY)
> >  	    != CCS_SCALING_CAPABILITY_NONE) {
> > -		rval = ccs_write(sensor, SCALING_MODE, sensor->scaling_mode);
> > +		u8 scaling_mode, scale_m;
> > +
> > +		ccs_get_scaling(sensor, &scaling_mode, &scale_m);
> > +
> > +		rval = ccs_write(sensor, SCALING_MODE, scaling_mode);
> >  		if (rval < 0)
> >  			goto err_pm_put;
> >  
> > -		rval = ccs_write(sensor, SCALE_M, sensor->scale_m);
> > +		rval = ccs_write(sensor, SCALE_M, scale_m);
> >  		if (rval < 0)
> >  			goto err_pm_put;
> >  	}
> > @@ -2270,7 +2307,6 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
> >  			  struct v4l2_subdev_state *sd_state, int which,
> >  			  int target)
> >  {
> > -	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
> >  	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
> >  	struct v4l2_rect *comp, *crop;
> >  	struct v4l2_mbus_framefmt *fmt;
> > @@ -2283,13 +2319,6 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
> >  						  CCS_STREAM_PIXEL);
> >  		comp->width = crop->width;
> >  		comp->height = crop->height;
> > -		if (which == V4L2_SUBDEV_FORMAT_ACTIVE) {
> > -			if (ssd == sensor->scaler) {
> > -				sensor->scale_m = CCS_LIM(sensor, SCALER_N_MIN);
> > -				sensor->scaling_mode =
> > -					CCS_SCALING_MODE_NO_SCALING;
> > -			}
> > -		}
> >  		fallthrough;
> >  	case V4L2_SEL_TGT_COMPOSE:
> >  		crop = v4l2_subdev_state_get_crop(sd_state, CCS_PAD_SRC,
> > @@ -2674,11 +2703,6 @@ static void ccs_set_compose_scaler(struct v4l2_subdev *subdev,
> >  			& ~1;
> >  	else
> >  		sel->r.height = sink_crop->height;
> > -
> > -	if (sel->which == V4L2_SUBDEV_FORMAT_ACTIVE) {
> > -		sensor->scale_m = scale_m;
> > -		sensor->scaling_mode = mode;
> > -	}
> >  }
> >  /* We're only called on source pads. This function sets scaling. */
> >  static int ccs_set_compose(struct v4l2_subdev *subdev,
> > @@ -3785,8 +3809,6 @@ static int ccs_probe(struct i2c_client *client)
> >  	sensor->pixel_array = &sensor->ssds[sensor->ssds_used];
> >  	sensor->ssds_used++;
> >  
> > -	sensor->scale_m = CCS_LIM(sensor, SCALER_N_MIN);
> > -
> >  	/* prepare PLL configuration input values */
> >  	sensor->pll.bus_type = CCS_PLL_BUS_TYPE_CSI2_DPHY;
> >  	sensor->pll.csi2.lanes = sensor->hwcfg.lanes;
> > diff --git a/drivers/media/i2c/ccs/ccs.h b/drivers/media/i2c/ccs/ccs.h
> > index e6fc00a9fa11..d33014f2710b 100644
> > --- a/drivers/media/i2c/ccs/ccs.h
> > +++ b/drivers/media/i2c/ccs/ccs.h
> > @@ -237,9 +237,6 @@ struct ccs_sensor {
> >  	u32 embedded_mbus_code;
> >  	u8 emb_data_ctrl;
> >  
> > -	u8 scale_m;
> > -	u8 scaling_mode;
> > -
> >  	u8 frame_skip;
> >  	u16 embedded_start; /* embedded data start line */
> >  	u16 embedded_end;
> 

-- 
Regards,

Sakari Ailus

Re: [PATCH v8 27/38] media: ccs: Compute binning configuration from sub-device state

[Sakari Ailus]

Hi Laurent,

On Thu, Mar 21, 2024 at 07:57:07PM +0200, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.
> 
> On Wed, Mar 13, 2024 at 09:25:05AM +0200, Sakari Ailus wrote:
> > Calculate binning configuration from sub-device state so the state related
> > configuration can be removed from the driver's device context struct.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  drivers/media/i2c/ccs/ccs-core.c | 72 ++++++++++++++++++++++----------
> >  drivers/media/i2c/ccs/ccs.h      |  3 --
> >  2 files changed, 49 insertions(+), 26 deletions(-)
> > 
> > diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
> > index a8c48abd2e30..3b80c54453cc 100644
> > --- a/drivers/media/i2c/ccs/ccs-core.c
> > +++ b/drivers/media/i2c/ccs/ccs-core.c
> > @@ -511,13 +511,52 @@ static int ccs_pll_try(struct ccs_sensor *sensor, struct ccs_pll *pll)
> >  	return ccs_pll_calculate(&client->dev, &lim, pll);
> >  }
> >  
> > +static void
> > +ccs_get_binning(struct ccs_sensor *sensor, u8 *binning_mode, u8 *binh, u8 *binv)
> > +{
> > +	struct v4l2_subdev_state *binner_state =
> > +		v4l2_subdev_get_locked_active_state(&sensor->binner->sd);
> > +	struct v4l2_rect *binner_sink_crop =
> > +		v4l2_subdev_state_get_crop(binner_state, CCS_PAD_SINK,
> > +					   CCS_STREAM_PIXEL);
> > +	struct v4l2_rect *binner_sink_comp =
> > +		v4l2_subdev_state_get_compose(binner_state, CCS_PAD_SINK,
> > +					      CCS_STREAM_PIXEL);
> 
> I like this patch very much :-) The comments I made to 28/28 apply there
> too.

I agree.

> 
> > +
> > +	if (binner_sink_crop->width == binner_sink_comp->width &&
> > +	    binner_sink_crop->height == binner_sink_comp->height) {
> > +		if (binning_mode)
> > +			*binning_mode = 0;
> > +
> > +		if (binh)
> > +			*binh = 1;
> > +
> > +		if (binv)
> > +			*binv = 1;
> > +
> > +		return;
> > +	}
> > +
> > +	if (binning_mode)
> > +		*binning_mode = 1;
> > +
> > +	if (binh)
> > +		*binh = binner_sink_crop->width / binner_sink_comp->width;
> > +
> > +	if (binv)
> > +		*binv = binner_sink_crop->height / binner_sink_comp->height;
> 
> binh and binv are never NULL. How about the following ?
> 
> 	if (binning_mode)
> 		*binning_mode = sink_crop->width != sink_comp->width ||
> 				sink_crop->height != sink_comp->height;

The binning mode should use CCS register macros which for some reason
appear to be absent on the binning registers. The others have them.

> 
> 	*binh = sink_crop->width / sink_comp->width;
> 	*binv = sink_crop->height / sink_comp->height;

Yes.

> 
> > +}
> > +
> >  static int ccs_pll_update(struct ccs_sensor *sensor)
> >  {
> >  	struct ccs_pll *pll = &sensor->pll;
> > +	u8 binh, binv;
> >  	int rval;
> >  
> > -	pll->binning_horizontal = sensor->binning_horizontal;
> > -	pll->binning_vertical = sensor->binning_vertical;
> > +	ccs_get_binning(sensor, NULL, &binh, &binv);
> > +
> > +	pll->binning_horizontal = binh;
> > +	pll->binning_vertical = binv;
> >  	pll->link_freq =
> >  		sensor->link_freq->qmenu_int[sensor->link_freq->val];
> >  	pll->scale_m = sensor->scale_m;
> > @@ -1241,8 +1280,11 @@ static void ccs_update_blanking(struct ccs_sensor *sensor,
> >  	struct v4l2_ctrl *hblank = sensor->hblank;
> >  	u16 min_fll, max_fll, min_llp, max_llp, min_lbp;
> >  	int min, max;
> > +	u8 binh, binv;
> > +
> > +	ccs_get_binning(sensor, NULL, &binh, &binv);
> >  
> > -	if (sensor->binning_vertical > 1 || sensor->binning_horizontal > 1) {
> > +	if (binv > 1 || binh > 1) {
> >  		min_fll = CCS_LIM(sensor, MIN_FRAME_LENGTH_LINES_BIN);
> >  		max_fll = CCS_LIM(sensor, MAX_FRAME_LENGTH_LINES_BIN);
> >  		min_llp = CCS_LIM(sensor, MIN_LINE_LENGTH_PCK_BIN);
> > @@ -1813,7 +1855,7 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
> >  		v4l2_subdev_state_get_crop(src_state, CCS_PAD_SRC,
> >  					   CCS_STREAM_PIXEL);
> >  	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
> > -	unsigned int binning_mode;
> > +	u8 binning_mode, binh, binv;
> >  	int rval;
> >  
> >  	if (pad != CCS_PAD_SRC)
> > @@ -1835,19 +1877,12 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
> >  		goto err_pm_put;
> >  
> >  	/* Binning configuration */
> > -	if (sensor->binning_horizontal == 1 &&
> > -	    sensor->binning_vertical == 1) {
> > -		binning_mode = 0;
> > -	} else {
> > -		u8 binning_type =
> > -			(sensor->binning_horizontal << 4)
> > -			| sensor->binning_vertical;
> > +	ccs_get_binning(sensor,	&binning_mode, &binh, &binv);
> >  
> > -		rval = ccs_write(sensor, BINNING_TYPE, binning_type);
> > +	if (binning_mode) {
> > +		rval = ccs_write(sensor, BINNING_TYPE, (binh << 4) | binv);
> >  		if (rval < 0)
> >  			goto err_pm_put;
> > -
> > -		binning_mode = 1;
> >  	}
> >  	rval = ccs_write(sensor, BINNING_MODE, binning_mode);
> >  	if (rval < 0)
> > @@ -2253,9 +2288,6 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
> >  				sensor->scale_m = CCS_LIM(sensor, SCALER_N_MIN);
> >  				sensor->scaling_mode =
> >  					CCS_SCALING_MODE_NO_SCALING;
> > -			} else if (ssd == sensor->binner) {
> > -				sensor->binning_horizontal = 1;
> > -				sensor->binning_vertical = 1;
> >  			}
> >  		}
> >  		fallthrough;
> > @@ -2529,10 +2561,6 @@ static void ccs_set_compose_binner(struct v4l2_subdev *subdev,
> >  			best = this;
> >  		}
> >  	}
> > -	if (sel->which == V4L2_SUBDEV_FORMAT_ACTIVE) {
> > -		sensor->binning_vertical = binv;
> > -		sensor->binning_horizontal = binh;
> > -	}
> >  
> >  	sel->r.width = (sink_crop->width / binh) & ~1;
> >  	sel->r.height = (sink_crop->height / binv) & ~1;
> > @@ -3721,8 +3749,6 @@ static int ccs_probe(struct i2c_client *client)
> >  				sensor->binning_subtypes[i].vertical);
> >  		}
> >  	}
> > -	sensor->binning_horizontal = 1;
> > -	sensor->binning_vertical = 1;
> >  
> >  	if (device_create_file(&client->dev, &dev_attr_ident) != 0) {
> >  		dev_err(&client->dev, "sysfs ident entry creation failed\n");
> > diff --git a/drivers/media/i2c/ccs/ccs.h b/drivers/media/i2c/ccs/ccs.h
> > index a2a461cda338..e6fc00a9fa11 100644
> > --- a/drivers/media/i2c/ccs/ccs.h
> > +++ b/drivers/media/i2c/ccs/ccs.h
> > @@ -237,9 +237,6 @@ struct ccs_sensor {
> >  	u32 embedded_mbus_code;
> >  	u8 emb_data_ctrl;
> >  
> > -	u8 binning_horizontal;
> > -	u8 binning_vertical;
> > -
> >  	u8 scale_m;
> >  	u8 scaling_mode;
> >  
> 

-- 
Regards,

Sakari Ailus

Re: [PATCH v8 16/38] media: v4l: subdev: Allow a larger number of routes than there's room for

[Sakari Ailus]

On Wed, Mar 20, 2024 at 03:53:19AM +0200, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.
> 
> On Wed, Mar 13, 2024 at 09:24:54AM +0200, Sakari Ailus wrote:
> > On VIDIOC_SUBDEV_[GS]_ROUTING, only return as many routes back to the user
> > as there's room. Do not consider it an error if more routes existed.
> > Simply inform the user there are more routes.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  .../media/v4l/vidioc-subdev-g-routing.rst              |  4 ----
> >  drivers/media/v4l2-core/v4l2-subdev.c                  | 10 +++-------
> >  2 files changed, 3 insertions(+), 11 deletions(-)
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > index 6eb6a59570dc..08b8d17cef3f 100644
> > --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > @@ -147,10 +147,6 @@ On success 0 is returned, on error -1 and the ``errno`` variable is set
> >  appropriately. The generic error codes are described at the
> >  :ref:`Generic Error Codes <gen-errors>` chapter.
> >  
> > -ENOSPC
> > -   The application provided ``num_routes`` is not big enough to contain
> > -   all the available routes the subdevice exposes.
> > -
> 
> I wonder if this patch should be squashed with the previous two. You
> update the documentation in 14/38 already.

I'll do that.

-- 
Sakari Ailus

Re: [PATCH v8 23/38] media: uapi: ccs: Add media bus code for MIPI CCS embedded data

[Laurent Pinchart]

Hi Sakari,

On Sun, Apr 14, 2024 at 10:48:49AM +0000, Sakari Ailus wrote:
> On Fri, Apr 12, 2024 at 10:07:46PM +0300, Laurent Pinchart wrote:
> > On Thu, Apr 11, 2024 at 09:04:47AM +0000, Sakari Ailus wrote:
> > > On Thu, Mar 21, 2024 at 06:49:14PM +0200, Laurent Pinchart wrote:
> > > > On Wed, Mar 13, 2024 at 09:25:01AM +0200, Sakari Ailus wrote:
> > > > > Add new MIPI CCS embedded data media bus code
> > > > > (MEDIA_BUS_FMT_CCS_EMBEDDED).
> > > > > 
> > > > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > > > ---
> > > > >  .../media/v4l/subdev-formats.rst              | 28 +++++++++++++++++++
> > > > >  include/uapi/linux/media-bus-format.h         |  3 ++
> > > > >  2 files changed, 31 insertions(+)
> > > > > 
> > > > > diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > > > > index cbd475f7cae9..c8f982411e70 100644
> > > > > --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > > > > +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > > > > @@ -8564,3 +8564,31 @@ and finally the bit number in subscript. "X" indicates a padding bit.
> > > > >        - X
> > > > >        - X
> > > > >        - X
> > > > > +
> > > > > +.. _MEDIA-BUS-FMT-CCS-EMBEDDED:
> > > > > +
> > > > > +MIPI CCS Embedded Data Formats
> > > > > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > > > +
> > > > > +`MIPI CCS <https://www.mipi.org/specifications/camera-command-set>`_ defines a
> > > > > +metadata format for sensor embedded data, which is used to store the register
> > > > > +configuration used for capturing a given frame. The format is defined in the CCS
> > > > > +specification. The media bus code for this format is
> > > > > +``MEDIA_BUS_FMT_CCS_EMBEDDED``.
> > > > > +
> > > > > +The CCS embedded data format definition includes three levels:
> > > > > +
> > > > > +1. Padding within CSI-2 bus :ref:`Data unit <media-glossary-data-unit>` as
> > > > 
> > > > s/Data unit/Data Unit/
> > > 
> > > Yes.
> > > 
> > > > > +   documented in the MIPI CCS specification.
> > > > > +
> > > > > +2. The tagged data format as documented in the MIPI CCS specification.
> > > > > +
> > > > > +3. Register addresses and register documentation as documented in the MIPI CCS
> > > > > +   specification.
> > > > > +
> > > > > +The format definition shall be used only by devices that fulfill all three
> > > > > +levels above.
> > > > 
> > > > Hmmmm... Do we need to mandate level 3 ? There are lots of sensors that
> > > > comply with the first two levels but have their own register set. Would
> > > > you like a sensor-specific embedded data format for each of them ? If
> > > > so, how would we document it without essentially copying the datasheet ?
> > > 
> > > You could refer to the datasheet if it's publicly available.
> > 
> > What if it's not ? How would you expect documentation to be organized ?
> > Will we then end up adding one media bus code per sensor ?
> 
> Probably. The user space needs a way to find out what to expect. If a
> sensor produces its own custom metadata format, it needs to be documented.
> 
> There's also documentation for OV2740 embedded data in the set.

The OV2740 is a different case than what I had in mind. It uses level 1
of the CCS embedded data only, without using the CCS tagged format. The
documentation you've included in this patch series looks good to me.

For sensors that are compatible with CCS embedded data level 2, I'm not
sure how to proceed. Those sensors send (register, value) pairs in their
embedded data, but the register set is not CCS-compatible. Even when
documentation is available, it's hard to verify that the sensor will
never output the value of undocumented registers, as the registers
included in the embedded data may vary depending on the sensor
configuration.

How would you proceed in such cases ? I can try to write a sample
documentation patch, but I don't know what you expect. I'm thinking
about the IMX219 sensor for instance, but any sensor compatible with CCS
embedded data level 2 without a public datasheet is a good example.

-- 
Regards,

Laurent Pinchart