[PATCH v9 00/46] Generic line based metadata support, internal pads

[PATCH v9 00/46] Generic line based metadata support, internal pads

[Sakari Ailus]

Hi folks,

Here are a few patches to add support generic, line based metadata as well
as internal pads. While the amount of code is not very large, to the
contrary it is quite small actually IMO, I presume what this is about and
why it is being proposed requires some explaining.

Metadata mbus codes and formats have existed for some time in V4L2. They
however have been only used by drivers that produce the data itself and
effectively this metadata has always been statistics of some sort (at
least when it comes to ISPs). What is different here is that we intend to
add support for metadata originating from camera sensors.

Camera sensors produce different kinds of metadata, embedded data (usually
register address--value pairs used to capture the frame, in a more or less
sensor specific format), histograms (in a very sensor specific format),
dark pixels etc. The number of these formats is probably going to be about
as large as image data formats if not larger, as the image data formats
are much better standardised but a smaller subset of them will be
supported by V4L2, at least initially but possibly much more in the long
run.

Having this many device specific formats would be a major problem for all
the other drivers along that pipeline (not to mention the users of those
drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2
receiver drivers that have DMA: the poor driver developer would not only
need to know camera sensor specific formats but to choose the specific
packing of that format suitable for the DMA used by the hardware. It is
unlikely many of these would ever get tested while being present on the
driver API. Also adding new sensors with new embedded data formats would
involve updating all bridge and CSI-2 receiver drivers. I don't expect
this to be a workable approach.

Instead what I'm proposing is to use specific metadata formats on the
sensor devices only, on internal pads (more about those soon) of the
sensors, only visible in the UAPI, and then generic mbus formats along the
pipeline and finally generic V4L2 metadata formats on the DMAs (specific
to bit depth and packing). This would unsnarl the two, defining what data
there is (specific mbus code) and how that is transported and packed
(generic mbus codes and V4L2 formats).

The user space would be required to "know" the path of that data from the
sensor's internal pad to the V4L2 video node. I do not see this as these
devices require at least some knowledge of the pipeline, i.e. hardware at
hand. Separating what the data means and how it is packed may even be
beneficial: it allows separating code that interprets the data (sensor
internal mbus code) from the code that accesses it (packing).

These formats are in practice line based, meaning that there may be
padding at the end of the line, depending on the bus as well as the DMA.
If non-line based formats are needed, it is always possible to set the
"height" field to 1.

The internal sink pads are an alternative to source routes [1]. The source
routes were not universally liked and I do have to say I like re-using
existing interface concepts (pads and everything you can do with pads,
including access format, selections etc.) wherever it makes sense, instead
of duplicating functionality.

Effectively internal sink pads behave mostly just like sink pads, but they
describe a flow of data that originates from a sub-device instead of
arriving to a sub-device. The SUBDEV_S_ROUTING IOCTLs are used to enable
and disable routes from internal sink pads to sub-device's source pads.
The subdev format IOCTLs are usable, too, so one can find which subdev
format is available on given internal sink pad.

I've also pushed these here and I'll keep updating the branch, I've also
included untested OV2740 patches:

<URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=metadata>

Questions and comments are most welcome.

Preliminary media-ctl and yavta patches can be found here:

<URL:https://git.retiisi.eu/?p=~sailus/yavta.git;a=shortlog;h=refs/heads/metadata>
<URL:https://git.retiisi.eu/?p=~sailus/v4l-utils.git;a=shortlog;h=refs/heads/metadata>

I have used IMX219 as an example on routing in a sensor driver in this
version.

The patches are on my master branch
<URL:https://git.linuxtv.org/sailus/media_tree.git/log/>.

[1] <URL:https://lore.kernel.org/linux-media/20220831141357.1396081-20-tomi.valkeinen@ideasonboard.com/>

since v8:

- Move the patch adding internal pad flag past the routing API reworks, as
  well as a few other patches, in order to separate the patches to those
  that could still be merged for v6.10 (routing changes) and those that
  couldn't (sensor API related). The patch on the edge is "media: uapi:
  v4l: subdev: Enable streams API".

- Include Laurent's two patches to address crop API issues wrt. streams.

- Add two patches to prepare for CCS driver rework (media: ccs: Move
  ccs_pm_get_init function up and media: ccs: Rename out label of
  ccs_start_streaming).

- Address issues in the ov2740 driver patches (as well as the driver
  itself), 4 more patches towards the end of the set.

- Improved generic metadata format names, align with other existing
  formats.

- Improved ov2740 embedded data documentation.

- Reworked streams and camera sensor documentation based on Laurent's
  comments mainly. In particular, the contradictory concept of internal
  source pads no longer should exist in the patches.

- Fixed pad numbering in the CCS example.

- Fixed S_ROUTING behaviour when len_routes is too small and when
  S_ROUTING isn't implemented by the driver.

- Reorder sections in meta-formats.rst alphabetically.

- Add a note per struct fields that certain struct v4l2_subdev_format are
  zero for metadata mbus codes.

- CCS driver patch cleanups.

- CCS driver metadata width fix for space-efficient embedded data at 16
  bpp and over.

- Postpone CCS frame descriptor quirk for now.

- Use MIPI_CSI2_DT_USER_DEFINED(0) instead of a numerical value for
  compressed data datatype.

since v7:

- Add embedded data support for the ov2740 driver.

- Add three patches on top, to add an IMMUTABLE flag to source streams
  when they cannot be disabled.

- Improved documentation of len_routes and num_routes arguments of
  [GS]_ROUTING.

- Remove one inclusion of twice-included media/v4l2-fwnode.h in
  drivers/media/i2c/ccs/ccs-core.c .

- Add missing forward declaration of ccs_internal_ops in
  drivers/media/i2c/ccs/ccs-core.c .

since v6:

- Improve embedded data UAPI documentation on camera sensors.

- Improve wording of stream glossary entry.

- Improve internal pad flag documentation.

- Fix definition of "data unit" and remove an extra "only" in INTERNAL pad
  flag description (1st patch).

- Use IMX219 driver in examples consistently.

- Remove the CSI-2 to parallel bridge from the example to simplify the
  example.

- Minor rewording of some parts of the routing examples.

- Rebase on unified sub-device state information access functions:
  <URL:https://lore.kernel.org/linux-media/20231027095913.1010187-1-sakari.ailus@linux.intel.com/T/#t>

- In CCS driver, do not maintain current active configuration in driver's
  device context struct (apart from mbus codes). Rely on sub-device state
  locking and clean up the code. (Multiple patches towards the end of the
  set.)

- Arrange the CCS patches early in the set towards the end of the set.

- Move the patch enabling streams API to the end of the set.

- Rework IOCTL argument copying condition for [GS]_ROUTING).

- Handle copying back routes in S_ROUTING, do not rely on G_ROUTING
  IOCTL implementation.

- Rebase on metadata preparation patchset v6:
  <URL:https://lore.kernel.org/linux-media/20231106121805.1266696-1-sakari.ailus@linux.intel.com/T/#t>.

since v5:

- Rebase on new set of preparation patches.

- Switch CCS driver from s_stream to enable_streams/disable_streams. Keep
  streaming state information --- the sensor remains in streaming state if
  any of the streams is enabled.

- Fix setting mbus code on embedded data in get_frame_desc() op in the CCS
  driver.

since v4:

- Add a patch to acquire two sub-device states that may use the same lock.

- Add a patch for CCS driver to remove ccs_get_crop_compose() helper.

- Add a patch for CCS driver moving acquiring and releasing the mutex to
  the s_stream callback.

- Add a patch for CCS driver to rely on sub-device state locking using a
  single driver-provided lock.

- Fixed calculating minimum number of routes in copying the routes
  (thanks, Laurent).

- Moved a label in S_ROUTING handling to make Clang happy (hopefully).

- Fixed setting emb_data_ctrl register for CCS embedded data support.

- Rebase on Laurent's cleanup patches.

- Wrap a few long lines.

- Write in embedded data documentation sensor drivers generally don't
  allow configuring it.

since v3:

- Separate preparation patches from this set.

- Add a definition for "Data unit", a pixel that is not image data and use
  it instead in format documentation.

- Fix more numbered lists in dev-subdev.rst.

- Remove a redundant definition for V4L2_META_FMT_GENERIC_CSI2_2_24 ---
  V4L2_META_FMT_GENERIC_CSI2_12 can be used instead.

- Use "X" instead of "p" to denote padding in format documentation.

- Use IMX219 in examples instead of CCS.

- Document that the generic V4L2 CSI-2 metadata formats use padding
  defined in CSI-2 spec and packing defined in CCS spec.

- Add patches to align [GS]_ROUTING behaviour with V4L2. This means mainly
  returning configured routes as part of S_ROUTING as well. "len_routes"
  field is added to denote the length of the array and having more routes
  than fits in the array is no longer an error. Also added more reserved
  fields.

- Added trivial support for S_ROUTING (via G_ROUTING implementation) for
  use in drivers with static-only routes.

- Added helper functions to obtain mbus format as well as crop and compose
  rectangles that are streams-independent.

- Added a patch to define generic CSI-2 long packet types.

- Removed MEDIA_BUS_FMT_IS_META() macro. It didn't seem useful in the end.

- Use a single CCS embedded data format. The bit depth can be selected
  using the meta stream on the source pad.

- Fix mbus code numbers (there were holes due to removed redundant
  formats).

- Fix generic mbus code documentation (byte was being used instead of
  bit).

- Fix spelling of "length".

- Added a patch to remove v4l2_subdev_enable_streams_api that disables
  streams API. This should be merged once libcamera support for streams
  works nicely.

- Don't use strings in printing frame descriptor flags.

- Warn on string truncation in printing frame descriptor.

since v2:

- Add a better example, with formats.

- Add CCS static data media bus codes.

- Added an example demonstrating the use of internal pads. --- Is the
  level of detail enough for the purpose?

- Improved documentation.

- Added a macro to tell whether a format is a metadata format.
  (Documentation could be added.)

- A small ReST syntax fix in the same section.

- Drop leftovers of a patch checking for the INTERNAL_SOURCE flag.

since v1:

- Make the new pad flag just "INTERNAL", requiring either SINK or SOURCE
  pad flag to accompany it. Removed the union in struct v4l2_subdev_route.

- Add the term "stream" to MC glossary.

- Improved and fixed documentation (according to comments).

- Note these formats are little endian.

- Remove 1X8 from the names of the mbus codes. These formats have generally
  8 bits per pixel.

- Fix mbus code numbering (had holes in RFC).

- Add new metadata fields to debug prints.

- Fix a minor documentation build issue.

Laurent Pinchart (2):
  media: v4l2-subdev: Fix stream handling for crop API
  media: v4l2-subdev: Clearly document that the crop API won't be
    extended

Sakari Ailus (44):
  media: Documentation: Add "stream" into glossary
  media: uapi: Add generic serial metadata mbus formats
  media: uapi: Document which mbus format fields are valid for metadata
  media: uapi: v4l: Add generic 8-bit metadata format definitions
  media: v4l: Support line-based metadata capture
  media: Documentation: Additional streams generally don't harm capture
  media: Documentation: Document embedded data guidelines for camera
    sensors
  media: Documentation: v4l: Document internal sink pads
  media: Documentation: Document S_ROUTING behaviour
  media: v4l: subdev: Add a function to lock two sub-device states, use
    it
  media: v4l: subdev: Move G_ROUTING handling below S_ROUTING
  media: v4l: subdev: Copy argument back to user also for S_ROUTING
  media: v4l: subdev: Add len_routes field to struct v4l2_subdev_routing
  media: v4l: subdev: Return routes set using S_ROUTING
  media: v4l: subdev: Add trivial set_routing support
  media: ccs: No need to set streaming to false in power off
  media: ccs: Move ccs_pm_get_init function up
  media: ccs: Rename out label of ccs_start_streaming
  media: ccs: Use {enable,disable}_streams operations
  media: ccs: Track streaming state
  media: ccs: Move ccs_validate_csi_data_format up
  media: ccs: Support frame descriptors
  media: uapi: v4l: subdev: Enable streams API
  media: mc: Add INTERNAL pad flag
  media: uapi: ccs: Add media bus code for MIPI CCS embedded data
  media: Documentation: Document non-CCS use of CCS embedded data format
  media: Documentation: ccs: Document routing
  media: ccs: Add support for embedded data stream
  media: ccs: Remove ccs_get_crop_compose helper
  media: ccs: Rely on sub-device state locking
  media: ccs: Compute binning configuration from sub-device state
  media: ccs: Compute scaling configuration from sub-device state
  media: ccs: Remove which parameter from ccs_propagate
  media: uapi: Add media bus code for ov2740 embedded data
  media: ov2740: Fix LINK_FREQ and PIXEL_RATE control value reporting
  media: ov2740: Remove shorthand variables
  media: ov2740: Switch to {enable,disable}_streams
  media: ov2740: Track streaming state
  media: ov2740: Add support for embedded data
  media: ov2740: Add generic sensor fwnode properties as controls
  media: ov2740: Add support for G_SELECTION IOCTL
  media: v4l: Add V4L2_SUBDEV_ROUTE_FL_IMMUTABLE sub-device routing flag
  media: ccs: Add IMMUTABLE route flag
  media: ov2740: Add IMMUTABLE route flag

 .../media/drivers/camera-sensor.rst           |   32 +
 .../userspace-api/media/drivers/ccs.rst       |   38 +-
 .../userspace-api/media/glossary.rst          |   15 +
 .../media/mediactl/media-types.rst            |    9 +
 .../userspace-api/media/v4l/dev-meta.rst      |   21 +
 .../userspace-api/media/v4l/dev-subdev.rst    |  179 ++-
 .../userspace-api/media/v4l/meta-formats.rst  |    3 +-
 .../media/v4l/metafmt-generic.rst             |  328 +++++
 .../media/v4l/subdev-formats.rst              |  374 +++++-
 .../media/v4l/vidioc-enum-fmt.rst             |    7 +
 .../media/v4l/vidioc-subdev-g-crop.rst        |    6 +-
 .../media/v4l/vidioc-subdev-g-routing.rst     |   60 +-
 .../media/videodev2.h.rst.exceptions          |    1 +
 drivers/media/i2c/ccs/ccs-core.c              | 1050 +++++++++++------
 drivers/media/i2c/ccs/ccs.h                   |   27 +-
 drivers/media/i2c/ov2740.c                    |  304 +++--
 drivers/media/mc/mc-entity.c                  |   15 +-
 drivers/media/v4l2-core/v4l2-ioctl.c          |   25 +-
 drivers/media/v4l2-core/v4l2-subdev.c         |  118 +-
 include/media/v4l2-subdev.h                   |   42 +
 include/uapi/linux/media-bus-format.h         |   13 +
 include/uapi/linux/media.h                    |    1 +
 include/uapi/linux/v4l2-mediabus.h            |   18 +-
 include/uapi/linux/v4l2-subdev.h              |   18 +-
 include/uapi/linux/videodev2.h                |   18 +
 25 files changed, 2183 insertions(+), 539 deletions(-)
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst

-- 
2.39.2

[PATCH v9 01/46] media: v4l2-subdev: Fix stream handling for crop API

[Sakari Ailus]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

When support for streams was added to the V4L2 subdev API, the
v4l2_subdev_crop structure was extended with a stream field, but the
field was not handled in the core code that translates the
VIDIOC_SUBDEV_[GS]_CROP ioctls to the selection API. Fix it.

Fixes: 2f91e10ee6fd ("media: subdev: add stream based configuration")
Reviewed-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 4c6198c48dd6..45836f0a2b0a 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -732,6 +732,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 		memset(&sel, 0, sizeof(sel));
 		sel.which = crop->which;
 		sel.pad = crop->pad;
+		sel.stream = crop->stream;
 		sel.target = V4L2_SEL_TGT_CROP;
 
 		rval = v4l2_subdev_call(
@@ -756,6 +757,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 		memset(&sel, 0, sizeof(sel));
 		sel.which = crop->which;
 		sel.pad = crop->pad;
+		sel.stream = crop->stream;
 		sel.target = V4L2_SEL_TGT_CROP;
 		sel.r = crop->rect;
 
-- 
2.39.2

[PATCH v9 02/46] media: v4l2-subdev: Clearly document that the crop API won't be extended

[Sakari Ailus]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

The V4L2 subdev crop API has been marked as obsolete, deprecated by the
selection API. Despite this, it has recently been extended with streams
support. In hindsight this was a mistake. Make sure it doesn't happen
again by clearly documenting that no new extensions will be accepted.

Suggested-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Acked-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../userspace-api/media/v4l/vidioc-subdev-g-crop.rst        | 6 +++---
 include/uapi/linux/v4l2-subdev.h                            | 4 ++++
 2 files changed, 7 insertions(+), 3 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
index 92d933631fda..88a748103a71 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
@@ -37,9 +37,9 @@ Description
 
 .. note::
 
-    This is an :ref:`obsolete` interface and may be removed
-    in the future. It is superseded by
-    :ref:`the selection API <VIDIOC_SUBDEV_G_SELECTION>`.
+    This is an :ref:`obsolete` interface and may be removed in the future. It is
+    superseded by :ref:`the selection API <VIDIOC_SUBDEV_G_SELECTION>`. No new
+    extensions to the :c:type:`v4l2_subdev_crop` structure will be accepted.
 
 To retrieve the current crop rectangle applications set the ``pad``
 field of a struct :c:type:`v4l2_subdev_crop` to the
diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
index 7048c51581c6..81a24bd38003 100644
--- a/include/uapi/linux/v4l2-subdev.h
+++ b/include/uapi/linux/v4l2-subdev.h
@@ -50,6 +50,10 @@ struct v4l2_subdev_format {
  * @rect: pad crop rectangle boundaries
  * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
+ *
+ * The subdev crop API is an obsolete interface and may be removed in the
+ * future. It is superseded by the selection API. No new extensions to this
+ * structure will be accepted.
  */
 struct v4l2_subdev_crop {
 	__u32 which;
-- 
2.39.2

[PATCH v9 03/46] media: Documentation: Add "stream" into glossary

[Sakari Ailus]

Add term "stream" to the glossary of the Media subsystem documentation.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 Documentation/userspace-api/media/glossary.rst | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/Documentation/userspace-api/media/glossary.rst b/Documentation/userspace-api/media/glossary.rst
index 96a360edbf3b..ef0ab601b5bf 100644
--- a/Documentation/userspace-api/media/glossary.rst
+++ b/Documentation/userspace-api/media/glossary.rst
@@ -173,6 +173,12 @@ Glossary
 	An integrated circuit that integrates all components of a computer
 	or other electronic systems.
 
+_media-glossary-stream:
+    Stream
+	A distinct flow of data (image data or metadata) from an initial source
+	to a final sink. The initial source may be e.g. an image sensor and the
+	final sink e.g. a memory buffer.
+
     V4L2 API
 	**V4L2 userspace API**
 
-- 
2.39.2

[PATCH v9 04/46] media: uapi: Add generic serial metadata mbus formats

[Sakari Ailus]

Add generic serial metadata mbus formats. These formats describe data
width and packing but not the content itself. The reason for specifying
such formats is that the formats as such are fairly device specific but
they are still handled by CSI-2 receiver drivers that should not be aware
of device specific formats. What makes generic metadata formats possible
is that these formats are parsed by software only, after capturing the
data to system memory.

Also add a definition for "Data Unit" to cover what is essentially a pixel
but is not image data.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
---
 .../userspace-api/media/glossary.rst          |   9 +
 .../media/v4l/subdev-formats.rst              | 255 ++++++++++++++++++
 include/uapi/linux/media-bus-format.h         |   9 +
 3 files changed, 273 insertions(+)

diff --git a/Documentation/userspace-api/media/glossary.rst b/Documentation/userspace-api/media/glossary.rst
index ef0ab601b5bf..84dbded08b47 100644
--- a/Documentation/userspace-api/media/glossary.rst
+++ b/Documentation/userspace-api/media/glossary.rst
@@ -25,6 +25,15 @@ Glossary
 
 	See :ref:`cec`.
 
+.. _media-glossary-data-unit:
+
+    Data Unit
+
+	Unit of data transported by a bus. On parallel buses, the data unit
+	consists of one or more related samples while on serial buses the data
+	unit is logical. If the data unit is image data, it may also be called a
+	pixel.
+
     Device Driver
 	Part of the Linux Kernel that implements support for a hardware
 	component.
diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
index eb3cd20b0cf2..d9a5ee954cdd 100644
--- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
+++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
@@ -8306,3 +8306,258 @@ The following table lists the existing metadata formats.
 	both sides of the link and the bus format is a fixed
 	metadata format that is not configurable from userspace.
 	Width and height will be set to 0 for this format.
+
+Generic Serial Metadata Formats
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Generic serial metadata formats are used on serial buses where the actual data
+content is more or less device specific but the data is transmitted and received
+by multiple devices that do not process the data in any way, simply writing
+it to system memory for processing in software at the end of the pipeline.
+
+"b" in an array cell signifies a byte of data, followed by the number of the bit
+and finally the bit number in subscript. "x" indicates a padding bit.
+
+.. _media-bus-format-generic-meta:
+
+.. cssclass: longtable
+
+.. flat-table:: Generic Serial Metadata Formats
+    :header-rows:  2
+    :stub-columns: 0
+
+    * - Identifier
+      - Code
+      -
+      - :cspan:`23` Data organization within bus ``Data Unit
+	<media-glossary-data-unit>``
+    * -
+      -
+      - Bit
+      - 23
+      - 22
+      - 21
+      - 20
+      - 19
+      - 18
+      - 17
+      - 16
+      - 15
+      - 14
+      - 13
+      - 12
+      - 11
+      - 10
+      - 9
+      - 8
+      - 7
+      - 6
+      - 5
+      - 4
+      - 3
+      - 2
+      - 1
+      - 0
+    * .. _MEDIA-BUS-FMT-META-8:
+
+      - MEDIA_BUS_FMT_META_8
+      - 0x8001
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+    * .. _MEDIA-BUS-FMT-META-10:
+
+      - MEDIA_BUS_FMT_META_10
+      - 0x8002
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - x
+      - x
+    * .. _MEDIA-BUS-FMT-META-12:
+
+      - MEDIA_BUS_FMT_META_12
+      - 0x8003
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - x
+      - x
+      - x
+      - x
+    * .. _MEDIA-BUS-FMT-META-14:
+
+      - MEDIA_BUS_FMT_META_14
+      - 0x8004
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - x
+      - x
+      - x
+      - x
+      - x
+      - x
+    * .. _MEDIA-BUS-FMT-META-16:
+
+      - MEDIA_BUS_FMT_META_16
+      - 0x8005
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - x
+      - x
+      - x
+      - x
+      - x
+      - x
+      - x
+      - x
+    * .. _MEDIA-BUS-FMT-META-20:
+
+      - MEDIA_BUS_FMT_META_20
+      - 0x8006
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - x
+      - x
+      - x
+      - x
+      - x
+      - x
+      - x
+      - x
+      - x
+      - x
+      - x
+      - x
+    * .. _MEDIA-BUS-FMT-META-24:
+
+      - MEDIA_BUS_FMT_META_24
+      - 0x8007
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - x
+      - x
+      - x
+      - x
+      - x
+      - x
+      - x
+      - x
+      - x
+      - x
+      - x
+      - x
+      - x
+      - x
+      - x
+      - x
diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
index f05f747e444d..d4c1d991014b 100644
--- a/include/uapi/linux/media-bus-format.h
+++ b/include/uapi/linux/media-bus-format.h
@@ -174,4 +174,13 @@
  */
 #define MEDIA_BUS_FMT_METADATA_FIXED		0x7001
 
+/* Generic line based metadata formats for serial buses. Next is 0x8008. */
+#define MEDIA_BUS_FMT_META_8			0x8001
+#define MEDIA_BUS_FMT_META_10			0x8002
+#define MEDIA_BUS_FMT_META_12			0x8003
+#define MEDIA_BUS_FMT_META_14			0x8004
+#define MEDIA_BUS_FMT_META_16			0x8005
+#define MEDIA_BUS_FMT_META_20			0x8006
+#define MEDIA_BUS_FMT_META_24			0x8007
+
 #endif /* __LINUX_MEDIA_BUS_FORMAT_H */
-- 
2.39.2

[PATCH v9 05/46] media: uapi: Document which mbus format fields are valid for metadata

[Sakari Ailus]

Now that metadata mbus formats have been added, it is necessary to define
which fields in struct v4l2_mbus_format are applicable to them (not many).

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 .../userspace-api/media/v4l/subdev-formats.rst | 15 ++++++++-------
 include/uapi/linux/v4l2-mediabus.h             | 18 ++++++++++++------
 2 files changed, 20 insertions(+), 13 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
index d9a5ee954cdd..0547f2733ee3 100644
--- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
+++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
@@ -33,7 +33,7 @@ Media Bus Formats
     * - __u32
       - ``field``
       - Field order, from enum :c:type:`v4l2_field`. See
-	:ref:`field-order` for details.
+	:ref:`field-order` for details. Zero on metadata mbus codes.
     * - __u32
       - ``colorspace``
       - Image colorspace, from enum :c:type:`v4l2_colorspace`.
@@ -45,7 +45,7 @@ Media Bus Formats
 	conversion is supported by setting the flag
 	V4L2_SUBDEV_MBUS_CODE_CSC_COLORSPACE in the corresponding struct
 	:c:type:`v4l2_subdev_mbus_code_enum` during enumeration.
-	See :ref:`v4l2-subdev-mbus-code-flags`.
+	See :ref:`v4l2-subdev-mbus-code-flags`. Zero on metadata mbus codes.
     * - union {
       - (anonymous)
     * - __u16
@@ -61,7 +61,7 @@ Media Bus Formats
 	that ycbcr_enc conversion is supported by setting the flag
 	V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC in the corresponding struct
 	:c:type:`v4l2_subdev_mbus_code_enum` during enumeration.
-	See :ref:`v4l2-subdev-mbus-code-flags`.
+	See :ref:`v4l2-subdev-mbus-code-flags`. Zero on metadata mbus codes.
     * - __u16
       - ``hsv_enc``
       - HSV encoding, from enum :c:type:`v4l2_hsv_encoding`.
@@ -75,7 +75,7 @@ Media Bus Formats
 	that hsv_enc conversion is supported by setting the flag
 	V4L2_SUBDEV_MBUS_CODE_CSC_HSV_ENC in the corresponding struct
 	:c:type:`v4l2_subdev_mbus_code_enum` during enumeration.
-	See :ref:`v4l2-subdev-mbus-code-flags`
+	See :ref:`v4l2-subdev-mbus-code-flags`. Zero on metadata mbus codes.
     * - }
       -
     * - __u16
@@ -90,8 +90,8 @@ Media Bus Formats
 	The driver indicates that quantization conversion is supported by
 	setting the flag V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION in the
 	corresponding struct :c:type:`v4l2_subdev_mbus_code_enum`
-	during enumeration. See :ref:`v4l2-subdev-mbus-code-flags`.
-
+	during enumeration. See :ref:`v4l2-subdev-mbus-code-flags`. Zero on
+	metadata mbus codes.
     * - __u16
       - ``xfer_func``
       - Transfer function, from enum :c:type:`v4l2_xfer_func`.
@@ -104,7 +104,8 @@ Media Bus Formats
 	The driver indicates that the transfer function conversion is supported by
 	setting the flag V4L2_SUBDEV_MBUS_CODE_CSC_XFER_FUNC in the
 	corresponding struct :c:type:`v4l2_subdev_mbus_code_enum`
-	during enumeration. See :ref:`v4l2-subdev-mbus-code-flags`.
+	during enumeration. See :ref:`v4l2-subdev-mbus-code-flags`. Zero on
+	metadata mbus codes.
     * - __u16
       - ``flags``
       - flags See:  :ref:v4l2-mbus-framefmt-flags
diff --git a/include/uapi/linux/v4l2-mediabus.h b/include/uapi/linux/v4l2-mediabus.h
index 6b07b73473b5..de1d6161bf62 100644
--- a/include/uapi/linux/v4l2-mediabus.h
+++ b/include/uapi/linux/v4l2-mediabus.h
@@ -19,12 +19,18 @@
  * @width:	image width
  * @height:	image height
  * @code:	data format code (from enum v4l2_mbus_pixelcode)
- * @field:	used interlacing type (from enum v4l2_field)
- * @colorspace:	colorspace of the data (from enum v4l2_colorspace)
- * @ycbcr_enc:	YCbCr encoding of the data (from enum v4l2_ycbcr_encoding)
- * @hsv_enc:	HSV encoding of the data (from enum v4l2_hsv_encoding)
- * @quantization: quantization of the data (from enum v4l2_quantization)
- * @xfer_func:  transfer function of the data (from enum v4l2_xfer_func)
+ * @field:	used interlacing type (from enum v4l2_field), zero on metadata
+ *		mbus codes
+ * @colorspace:	colorspace of the data (from enum v4l2_colorspace), zero on
+ *		metadata mbus codes
+ * @ycbcr_enc:	YCbCr encoding of the data (from enum v4l2_ycbcr_encoding), zero
+ *		on metadata mbus codes
+ * @hsv_enc:	HSV encoding of the data (from enum v4l2_hsv_encoding), zero on
+ *		metadata mbus codes
+ * @quantization: quantization of the data (from enum v4l2_quantization), zero
+ *		on metadata mbus codes
+ * @xfer_func:  transfer function of the data (from enum v4l2_xfer_func), zero
+ *		on metadata mbus codes
  * @flags:	flags (V4L2_MBUS_FRAMEFMT_*)
  * @reserved:  reserved bytes that can be later used
  */
-- 
2.39.2

[PATCH v9 06/46] media: uapi: v4l: Add generic 8-bit metadata format definitions

[Sakari Ailus]

Generic 8-bit metadata formats define the in-memory data layout but not
the format of the data itself. The reasoning for having such formats is to
allow CSI-2 receiver drivers to receive and DMA drivers to write the data
to memory without knowing a large number of device-specific formats.

These formats may be used only in conjunction with a Media controller
pipeline where the internal pad of the source sub-device defines the
specific format of the data (using an mbus code).

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../userspace-api/media/v4l/dev-subdev.rst    |   2 +
 .../userspace-api/media/v4l/meta-formats.rst  |   3 +-
 .../media/v4l/metafmt-generic.rst             | 328 ++++++++++++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c          |   7 +
 include/uapi/linux/videodev2.h                |   8 +
 5 files changed, 347 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst

diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index 43988516acdd..f375b820ab68 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -506,6 +506,8 @@ source pads.
 
     subdev-formats
 
+.. _subdev-routing:
+
 Streams, multiplexed media pads and internal routing
 ----------------------------------------------------
 
diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
index 0bb61fc5bc00..c23aac823d2c 100644
--- a/Documentation/userspace-api/media/v4l/meta-formats.rst
+++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
@@ -13,9 +13,10 @@ These formats are used for the :ref:`metadata` interface only.
     :maxdepth: 1
 
     metafmt-d4xx
+    metafmt-generic
     metafmt-intel-ipu3
     metafmt-rkisp1
     metafmt-uvc
+    metafmt-vivid
     metafmt-vsp1-hgo
     metafmt-vsp1-hgt
-    metafmt-vivid
diff --git a/Documentation/userspace-api/media/v4l/metafmt-generic.rst b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
new file mode 100644
index 000000000000..cba34be64dfe
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
@@ -0,0 +1,328 @@
+.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
+
+********************************************************************************************************************************************************************************************************************************************************************************
+V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O')
+********************************************************************************************************************************************************************************************************************************************************************************
+
+
+Generic line-based metadata formats
+
+
+Description
+===========
+
+These generic line-based metadata formats define the memory layout of the data
+without defining the format or meaning of the metadata itself. These formats may
+only be used with a Media Controller pipeline where the more specific format is
+reported by an :ref:`internal sink pad <MEDIA-PAD-FL-INTERNAL>` of the source
+sub-device. See also :ref:`source routes <subdev-routing>`.
+
+.. _v4l2-meta-fmt-generic-8:
+
+V4L2_META_FMT_GENERIC_8
+-----------------------
+
+The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format.
+
+This format is also used on CSI-2 for both 8 bits per ``Data Unit
+<media-glossary-data-unit>`` as well as for 16 bits per Data Unit when two bytes
+of metadata are packed into one 16-bit Data Unit.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_8.**
+Each cell is one byte. "M" denotes a byte of metadata.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|
+
+.. flat-table:: Sample 4x2 Metadata Frame
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+    * - start + 4:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+
+.. _v4l2-meta-fmt-generic-csi2-10:
+
+V4L2_META_FMT_GENERIC_CSI2_10
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_10 contains 8-bit generic metadata packed in 10-bit
+Data Units, with one padding byte after every four bytes of metadata. This
+format is typically used by CSI-2 receivers with a source that transmits
+MEDIA_BUS_FMT_META_10 and the CSI-2 receiver writes the received data to memory
+as-is.
+
+The packing of the data follows the MIPI CSI-2 specification and the padding of
+the data is defined in the MIPI CCS specification.
+
+This format is also used in conjunction with 20 bits per ``Data Unit
+<media-glossary-data-unit>`` formats that pack two bytes of metadata into one
+Data Unit.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
+Each cell is one byte. "M" denotes a byte of metadata and "x" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table:: Sample 4x2 Metadata Frame
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - x
+    * - start + 5:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - x
+
+.. _v4l2-meta-fmt-generic-csi2-12:
+
+V4L2_META_FMT_GENERIC_CSI2_12
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_12 contains 8-bit generic metadata packed in 12-bit
+Data Units, with two padding bytes after every four bytes of metadata. This
+format is typically used by CSI-2 receivers with a source that transmits
+MEDIA_BUS_FMT_META_12 and the CSI-2 receiver writes the received data to memory
+as-is.
+
+The packing of the data follows the MIPI CSI-2 specification and the padding of
+the data is defined in the MIPI CCS specification.
+
+This format is also used in conjunction with 24 bits per ``Data Unit
+<media-glossary-data-unit>`` formats that pack two bytes of metadata into one
+Data Unit.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_12.**
+Each cell is one byte. "M" denotes a byte of metadata and "x" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table:: Sample 4x2 Metadata Frame
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - x
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - x
+    * - start + 6:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - x
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - x
+
+.. _v4l2-meta-fmt-generic-csi2-14:
+
+V4L2_META_FMT_GENERIC_CSI2_14
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_14 contains 8-bit generic metadata packed in 14-bit
+Data Units, with three padding bytes after every four bytes of metadata. This
+format is typically used by CSI-2 receivers with a source that transmits
+MEDIA_BUS_FMT_META_14 and the CSI-2 receiver writes the received data to memory
+as-is.
+
+The packing of the data follows the MIPI CSI-2 specification and the padding of
+the data is defined in the MIPI CCS specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_14.**
+Each cell is one byte. "M" denotes a byte of metadata and "x" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table:: Sample 4x2 Metadata Frame
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - x
+      - x
+      - x
+    * - start + 7:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - x
+      - x
+      - x
+
+.. _v4l2-meta-fmt-generic-csi2-16:
+
+V4L2_META_FMT_GENERIC_CSI2_16
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_16 contains 8-bit generic metadata packed in 16-bit
+Data Units, with one padding byte after every byte of metadata. This format is
+typically used by CSI-2 receivers with a source that transmits
+MEDIA_BUS_FMT_META_16 and the CSI-2 receiver writes the received data to memory
+as-is.
+
+The packing of the data follows the MIPI CSI-2 specification and the padding of
+the data is defined in the MIPI CCS specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_16.**
+Each cell is one byte. "M" denotes a byte of metadata and "x" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table:: Sample 4x2 Metadata Frame
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - x
+      - M\ :sub:`10`
+      - x
+      - M\ :sub:`20`
+      - x
+      - M\ :sub:`30`
+      - x
+    * - start + 8:
+      - M\ :sub:`01`
+      - x
+      - M\ :sub:`11`
+      - x
+      - M\ :sub:`21`
+      - x
+      - M\ :sub:`31`
+      - x
+
+.. _v4l2-meta-fmt-generic-csi2-20:
+
+V4L2_META_FMT_GENERIC_CSI2_20
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_20 contains 8-bit generic metadata packed in 20-bit
+Data Units, with alternating one or two padding bytes after every byte of
+metadata. This format is typically used by CSI-2 receivers with a source that
+transmits MEDIA_BUS_FMT_META_20 and the CSI-2 receiver writes the received data
+to memory as-is.
+
+The packing of the data follows the MIPI CSI-2 specification and the padding of
+the data is defined in the MIPI CCS specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_20.**
+Each cell is one byte. "M" denotes a byte of metadata and "x" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table:: Sample 4x2 Metadata Frame
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - x
+      - M\ :sub:`10`
+      - x
+      - x
+      - M\ :sub:`20`
+      - x
+      - M\ :sub:`30`
+      - x
+      - x
+    * - start + 10:
+      - M\ :sub:`01`
+      - x
+      - M\ :sub:`11`
+      - x
+      - x
+      - M\ :sub:`21`
+      - x
+      - M\ :sub:`31`
+      - x
+      - x
+
+.. _v4l2-meta-fmt-generic-csi2-24:
+
+V4L2_META_FMT_GENERIC_CSI2_24
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_24 contains 8-bit generic metadata packed in 24-bit
+Data Units, with two padding bytes after every byte of metadata. This format is
+typically used by CSI-2 receivers with a source that transmits
+MEDIA_BUS_FMT_META_24 and the CSI-2 receiver writes the received data to memory
+as-is.
+
+The packing of the data follows the MIPI CSI-2 specification and the padding of
+the data is defined in the MIPI CCS specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_24.**
+Each cell is one byte. "M" denotes a byte of metadata and "x" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table:: Sample 4x2 Metadata Frame
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - x
+      - x
+      - M\ :sub:`10`
+      - x
+      - x
+      - M\ :sub:`20`
+      - x
+      - x
+      - M\ :sub:`30`
+      - x
+      - x
+    * - start + 12:
+      - M\ :sub:`01`
+      - x
+      - x
+      - M\ :sub:`11`
+      - x
+      - x
+      - M\ :sub:`21`
+      - x
+      - x
+      - M\ :sub:`31`
+      - x
+      - x
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index 5aeff5519407..ae2dca7f2817 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -1460,6 +1460,13 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
 	case V4L2_PIX_FMT_Y210:		descr = "10-bit YUYV Packed"; break;
 	case V4L2_PIX_FMT_Y212:		descr = "12-bit YUYV Packed"; break;
 	case V4L2_PIX_FMT_Y216:		descr = "16-bit YUYV Packed"; break;
+	case V4L2_META_FMT_GENERIC_8:	descr = "8-bit Generic Metadata"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_10:	descr = "8-bit Generic Meta, 10b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_12:	descr = "8-bit Generic Meta, 12b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_14:	descr = "8-bit Generic Meta, 14b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_16:	descr = "8-bit Generic Meta, 16b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_20:	descr = "8-bit Generic Meta, 20b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_24:	descr = "8-bit Generic Meta, 24b CSI-2"; break;
 
 	default:
 		/* Compressed formats */
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 2663213b76a4..c7cf20b5da67 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -839,6 +839,14 @@ struct v4l2_pix_format {
 #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
 #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
 
+#define V4L2_META_FMT_GENERIC_8		v4l2_fourcc('M', 'E', 'T', '8') /* Generic 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_10	v4l2_fourcc('M', 'C', '1', 'A') /* 10-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_12	v4l2_fourcc('M', 'C', '1', 'C') /* 12-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_14	v4l2_fourcc('M', 'C', '1', 'E') /* 14-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_16	v4l2_fourcc('M', 'C', '1', 'G') /* 16-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_20	v4l2_fourcc('M', 'C', '1', 'K') /* 20-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_24	v4l2_fourcc('M', 'C', '1', 'O') /* 24-bit CSI-2 packed 8-bit metadata */
+
 /* priv field value to indicates that subsequent fields are valid. */
 #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
 
-- 
2.39.2

[PATCH v9 07/46] media: v4l: Support line-based metadata capture

[Sakari Ailus]

Many camera sensors, among other devices, transmit embedded data and image
data for each CSI-2 frame. This embedded data typically contains register
configuration of the sensor that has been used to capture the image data
of the same frame.

The embedded data is received by the CSI-2 receiver and has the same
properties as the image data, including that it is line based: it has
width, height and bytesperline (stride).

Add these fields to struct v4l2_meta_format and document them.

Also add V4L2_FMT_FLAG_META_LINE_BASED to tell a given format is
line-based i.e. these fields of struct v4l2_meta_format are valid for it.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../userspace-api/media/v4l/dev-meta.rst      | 21 +++++++++++++++++++
 .../media/v4l/vidioc-enum-fmt.rst             |  7 +++++++
 .../media/videodev2.h.rst.exceptions          |  1 +
 drivers/media/v4l2-core/v4l2-ioctl.c          |  5 +++--
 include/uapi/linux/videodev2.h                | 10 +++++++++
 5 files changed, 42 insertions(+), 2 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/dev-meta.rst b/Documentation/userspace-api/media/v4l/dev-meta.rst
index 0e7e1ee1471a..5eee9ab60395 100644
--- a/Documentation/userspace-api/media/v4l/dev-meta.rst
+++ b/Documentation/userspace-api/media/v4l/dev-meta.rst
@@ -47,6 +47,12 @@ member of the ``fmt`` union as needed per the desired operation. Both drivers
 and applications must set the remainder of the :c:type:`v4l2_format` structure
 to 0.
 
+Devices that capture metadata by line have the struct v4l2_fmtdesc
+``V4L2_FMT_FLAG_META_LINE_BASED`` flag set for :c:func:`VIDIOC_ENUM_FMT`. Such
+devices can typically also :ref:`capture image data <capture>`. This primarily
+involves devices that receive the data from a different devices such as a camera
+sensor.
+
 .. c:type:: v4l2_meta_format
 
 .. tabularcolumns:: |p{1.4cm}|p{2.4cm}|p{13.5cm}|
@@ -65,3 +71,18 @@ to 0.
       - ``buffersize``
       - Maximum buffer size in bytes required for data. The value is set by the
         driver.
+    * - __u32
+      - ``width``
+      - Width of a line of metadata in Data Units. Valid when
+	:c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set,
+	otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
+    * - __u32
+      - ``height``
+      - Number of rows of metadata. Valid when :c:type`v4l2_fmtdesc` flag
+	``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
+	:c:func:`VIDIOC_ENUM_FMT`.
+    * - __u32
+      - ``bytesperline``
+      - Offset in bytes between the beginning of two consecutive lines. Valid
+	when :c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is
+	set, otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
diff --git a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
index 000c154b0f98..a439be1b15d1 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
@@ -227,6 +227,13 @@ the ``mbus_code`` field is handled differently:
 	The application can ask to configure the quantization of the capture
 	device when calling the :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl with
 	:ref:`V4L2_PIX_FMT_FLAG_SET_CSC <v4l2-pix-fmt-flag-set-csc>` set.
+    * - ``V4L2_FMT_FLAG_META_LINE_BASED``
+      - 0x0200
+      - The metadata format is line-based. In this case the ``width``,
+	``height`` and ``bytesperline`` fields of :c:type:`v4l2_meta_format` are
+	valid. The buffer consists of ``height`` lines, each having ``width``
+	Data Units of data and offset (in bytes) between the beginning of each
+	two consecutive lines is ``bytesperline``.
 
 Return Value
 ============
diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b/Documentation/userspace-api/media/videodev2.h.rst.exceptions
index 3e58aac4ef0b..bdc628e8c1d6 100644
--- a/Documentation/userspace-api/media/videodev2.h.rst.exceptions
+++ b/Documentation/userspace-api/media/videodev2.h.rst.exceptions
@@ -215,6 +215,7 @@ replace define V4L2_FMT_FLAG_CSC_XFER_FUNC fmtdesc-flags
 replace define V4L2_FMT_FLAG_CSC_YCBCR_ENC fmtdesc-flags
 replace define V4L2_FMT_FLAG_CSC_HSV_ENC fmtdesc-flags
 replace define V4L2_FMT_FLAG_CSC_QUANTIZATION fmtdesc-flags
+replace define V4L2_FMT_FLAG_META_LINE_BASED fmtdesc-flags
 
 # V4L2 timecode types
 replace define V4L2_TC_TYPE_24FPS timecode-type
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index ae2dca7f2817..2cfc9106857a 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -343,8 +343,9 @@ static void v4l_print_format(const void *arg, bool write_only)
 	case V4L2_BUF_TYPE_META_OUTPUT:
 		meta = &p->fmt.meta;
 		pixelformat = meta->dataformat;
-		pr_cont(", dataformat=%p4cc, buffersize=%u\n",
-			&pixelformat, meta->buffersize);
+		pr_cont(", dataformat=%p4cc, buffersize=%u, width=%u, height=%u, bytesperline=%u\n",
+			&pixelformat, meta->buffersize, meta->width,
+			meta->height, meta->bytesperline);
 		break;
 	}
 }
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index c7cf20b5da67..37112dfebd0c 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -877,6 +877,7 @@ struct v4l2_fmtdesc {
 #define V4L2_FMT_FLAG_CSC_YCBCR_ENC		0x0080
 #define V4L2_FMT_FLAG_CSC_HSV_ENC		V4L2_FMT_FLAG_CSC_YCBCR_ENC
 #define V4L2_FMT_FLAG_CSC_QUANTIZATION		0x0100
+#define V4L2_FMT_FLAG_META_LINE_BASED		0x0200
 
 	/* Frame Size and frame rate enumeration */
 /*
@@ -2424,10 +2425,19 @@ struct v4l2_sdr_format {
  * struct v4l2_meta_format - metadata format definition
  * @dataformat:		little endian four character code (fourcc)
  * @buffersize:		maximum size in bytes required for data
+ * @width:		number of data units of data per line (valid for line
+ *			based formats only, see format documentation)
+ * @height:		number of lines of data per buffer (valid for line based
+ *			formats only)
+ * @bytesperline:	offset between the beginnings of two adjacent lines in
+ *			bytes (valid for line based formats only)
  */
 struct v4l2_meta_format {
 	__u32				dataformat;
 	__u32				buffersize;
+	__u32				width;
+	__u32				height;
+	__u32				bytesperline;
 } __attribute__ ((packed));
 
 /**
-- 
2.39.2

[PATCH v9 08/46] media: Documentation: Additional streams generally don't harm capture

[Sakari Ailus]

Having extra streams on the source end of the link that cannot be captured
by the sink sub-device generally are not an issue, at least not on CSI-2
bus. Still document that there may be hardware-specific limitations. For
example on parallel bus this might not work on all cases.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 Documentation/userspace-api/media/v4l/dev-subdev.rst | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index f375b820ab68..b76e02e54512 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -529,9 +529,9 @@ the its sink pad and allows to route them individually to one of its source
 pads.
 
 Subdevice drivers that support multiplexed streams are compatible with
-non-multiplexed subdev drivers, but, of course, require a routing configuration
-where the link between those two types of drivers contains only a single
-stream.
+non-multiplexed subdev drivers. However, if the driver at the sink end of a link
+does not support streams, then only stream 0 of source end may be captured.
+There may be additional limitations specific to the sink device.
 
 Understanding streams
 ^^^^^^^^^^^^^^^^^^^^^
-- 
2.39.2

[PATCH v9 09/46] media: Documentation: Document embedded data guidelines for camera sensors

[Sakari Ailus]

Document how embedded data support should be implemented for camera
sensors, and when and how CCS embedded data format should be referenced.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Julien Massot <julien.massot@collabora.com>
---
 .../media/drivers/camera-sensor.rst           | 21 +++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/Documentation/userspace-api/media/drivers/camera-sensor.rst b/Documentation/userspace-api/media/drivers/camera-sensor.rst
index 919a50e8b9d9..9f3b0da3ad0d 100644
--- a/Documentation/userspace-api/media/drivers/camera-sensor.rst
+++ b/Documentation/userspace-api/media/drivers/camera-sensor.rst
@@ -102,3 +102,24 @@ register programming sequences shall initialize the :ref:`V4L2_CID_HFLIP
 values programmed by the register sequences. The default values of these
 controls shall be 0 (disabled). Especially these controls shall not be inverted,
 independently of the sensor's mounting rotation.
+
+Embedded data
+-------------
+
+Many sensors, mostly raw sensors, support embedded data which is used to convey
+the sensor configuration for the captured frame back to the host. While CSI-2 is
+the most common data interface used by such sensors, embedded data can be
+available on other interfaces as well.
+
+Such sensors expose two internal sink pads (pads that have both the
+``MEDIA_PAD_FL_SINK <MEDIA-PAD-FL-SINK>`` and ``MEDIA_PAD_FL_INTERNAL
+<MEDIA-PAD-FL-INTERNAL>`` flags set) to model the source of the image and
+embedded data streams. Both of these pads produces a single stream, and the
+sub-device routes those streams to the external (source) pad. If the sub-device
+driver supports disabling embedded data, this can be done by disabling the
+embedded data route via the ``VIDIOC_SUBDEV_S_ROUTING`` IOCTL.
+
+In general, changing the embedded data format from the driver-configured values
+is not supported. The height of the metadata is device-specific and the width
+is that (or less of that) of the image width, as configured on the pixel data
+stream.
-- 
2.39.2

[PATCH v9 10/46] media: Documentation: v4l: Document internal sink pads

[Sakari Ailus]

Document internal sink pads, pads that have both SINK and INTERNAL flags
set. Use the IMX219 camera sensor as an example.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by Julien Massot <julien.massot@collabora.com>
---
 .../userspace-api/media/v4l/dev-subdev.rst    | 145 ++++++++++++++++++
 1 file changed, 145 insertions(+)

diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index b76e02e54512..d30dcb9e2537 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -553,6 +553,27 @@ A stream at a specific point in the media pipeline is identified by the
 sub-device and a (pad, stream) pair. For sub-devices that do not support
 multiplexed streams the 'stream' field is always 0.
 
+.. _v4l2-subdev-internal-source-pads:
+
+Internal sink pads and routing
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Cases where a single sub-device source pad is traversed by multiple streams, one
+or more of which originate from within the sub-device itself, are special as
+there is no external sink pad for such routes. In those cases, the sources of
+the internally generated streams are represented by internal sink pads, which
+are sink pads that have the :ref:`MEDIA_PAD_FL_INTERNAL <MEDIA-PAD-FL-INTERNAL>`
+pad flag set.
+
+Internal pads have all the properties of an external pad, including formats and
+selections. The format in this case is the source format of the stream. An
+internal pad always has a single stream only (0).
+
+Routes from an internal sink pad to an external source pad are typically not
+modifiable but they can be activated and deactivated using the
+:ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE <v4l2-subdev-routing-flags>` flag, depending
+on driver capabilities.
+
 Interaction between routes, streams, formats and selections
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -668,3 +689,127 @@ To configure this pipeline, the userspace must take the following steps:
    the configurations along the stream towards the receiver, using
    :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each
    stream endpoint in each sub-device.
+
+Internal pads setup example
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A simple example of a multiplexed stream setup might be as follows:
+
+- An IMX219 camera sensor source sub-device, with one source pad (0), one
+  internal sink pad (1) as the source of the image data and an internal sink
+  pad (2) as the source of the embedded data. There are two routes, one from the
+  internal sink pad 1 to the source 0 (image data) and another from the internal
+  sink pad 2 to the source pad 0 (embedded data). Both streams are always
+  active, i.e. there is no need to separately enable the embedded data
+  stream. The sensor uses the CSI-2 interface.
+
+- A CSI-2 receiver in the SoC. The receiver has a single sink pad (pad 0),
+  connected to the sensor, and two source pads (pads 1 and 2), to the DMA
+  engine. The receiver demultiplexes the incoming streams to the source pads.
+
+- DMA engines in the SoC, one for each stream. Each DMA engine is connected to a
+  single source pad of the receiver.
+
+The sensor and the receiver are modelled as V4L2 sub-devices, exposed to
+userspace via /dev/v4l-subdevX device nodes. The DMA engines are modelled as
+V4L2 devices, exposed to userspace via /dev/videoX nodes.
+
+To configure this pipeline, the userspace must take the following steps:
+
+1) Set up media links between entities: enable the links from the sensor to the
+   receiver and from the receiver to the DMA engines. This step does not differ
+   from normal non-multiplexed media controller setup.
+
+2) Configure routing
+
+.. flat-table:: Camera sensor. There are no configurable routes.
+    :header-rows: 1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 1/0
+      - 0/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from the sink pad
+    * - 2/0
+      - 0/1
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from the internal sink pad
+
+.. flat-table:: Receiver routing table. Typically both routes need to be
+		explicitly set.
+    :header-rows:  1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0
+      - 1/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from camera sensor
+    * - 0/1
+      - 2/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from camera sensor
+
+The options available in sensor's routing configuration are dictated by hardware
+capabilities: typically camera sensors always produce an image data stream while
+it may be possible to enable and disable the embedded data stream.
+
+3) Configure formats and selections
+
+   This example assumes that the formats are propagated from sink pad to the
+   source pad as-is. The tables contain fields of both struct v4l2_subdev_format
+   and struct v4l2_mbus_framefmt.
+
+.. flat-table:: Formats set on the sub-devices. Bold values are set, others are
+                static or propagated. The order is aligned with configured
+                routes.
+    :header-rows: 1
+    :fill-cells:
+
+    * - Sub-device
+      - Pad/Stream
+      - Width
+      - Height
+      - Code
+    * - :rspan:`3` IMX219
+      - 1/0
+      - 3296
+      - 2480
+      - MEDIA_BUS_FMT_SRGGB10
+    * - 0/0
+      - **3296**
+      - **2480**
+      - **MEDIA_BUS_FMT_SRGGB10**
+    * - 2/0
+      - 3296
+      - 2
+      - MEDIA_BUS_FMT_IMX219_EMBEDDED
+    * - 1/1
+      - 3296
+      - 2
+      - MEDIA_BUS_FMT_META_10
+    * - :rspan:`3` CSI-2 receiver
+      - 0/0
+      - **3296**
+      - **2480**
+      - **MEDIA_BUS_FMT_SRGGB10**
+    * - 1/0
+      - 3296
+      - 2480
+      - MEDIA_BUS_FMT_SRGGB10
+    * - 0/1
+      - **3296**
+      - **2**
+      - **MEDIA_BUS_FMT_META_10**
+    * - 2/0
+      - 3296
+      - 2
+      - MEDIA_BUS_FMT_META_10
+
+The embedded data format does not need to be configured as the format is
+dictated by the pixel data format in this case.
-- 
2.39.2

[PATCH v9 11/46] media: Documentation: Document S_ROUTING behaviour

[Sakari Ailus]

Document S_ROUTING behaviour for different devices.

Generally in devices that produce streams the streams are static and some
can be enabled and disabled, whereas in devices that just transport them
or write them to memory, more configurability is allowed. Document this.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Julien Massot <julien.massot@collabora.com>
---
 .../userspace-api/media/v4l/dev-subdev.rst    | 24 +++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index d30dcb9e2537..de8dfd4f11a5 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -593,6 +593,30 @@ Any configurations of a stream within a pad, such as format or selections,
 are independent of similar configurations on other streams. This is
 subject to change in the future.
 
+Device types and routing setup
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Different kinds of sub-devices have differing behaviour for route activation,
+depending on the hardware. In all cases, however, only routes that have the
+``V4L2_SUBDEV_STREAM_FL_ACTIVE`` flag set are active.
+
+Devices generating the streams may allow enabling and disabling some of the
+routes or the configuration is fixed. If the routes can be disabled, not
+declaring the routes (or declaring them without
+``VIDIOC_SUBDEV_STREAM_FL_ACTIVE`` flag set) in ``VIDIOC_SUBDEV_S_ROUTING`` will
+disable the routes while the sub-device driver retains the streams and their
+format and selection configuration. The ``VIDIOC_SUBDEV_S_ROUTING`` will still
+return such routes back to the user in the routes array, with the
+``V4L2_SUBDEV_STREAM_FL_ACTIVE`` flag unset.
+
+Devices transporting the streams almost always have more configurability with
+respect to routing. Typically any route between the sub-device's sink and source
+pads is possible, and multiple routes (usually up to certain limited number) may
+be active simultaneously. For such devices, no routes are created by the driver
+and user-created routes are fully replaced when ``VIDIOC_SUBDEV_S_ROUTING`` is
+called on the sub-device. Such newly created routes have the device's default
+configuration for format and selection rectangles.
+
 Configuring streams
 ^^^^^^^^^^^^^^^^^^^
 
-- 
2.39.2

[PATCH v9 12/46] media: v4l: subdev: Add a function to lock two sub-device states, use it

[Sakari Ailus]

Add two new functions, v4l2_subdev_lock_states() and
v4l2_subdev_unclock_states(), to acquire and release the state of two
sub-devices. They differ from calling v4l2_subdev_{un,}lock_state() so
that if the two states share the same lock, the lock is acquired only
once.

Also use the new functions in v4l2_subdev_link_validate().

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Julien Massot <julien.massot@collabora.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 12 +++-----
 include/media/v4l2-subdev.h           | 40 +++++++++++++++++++++++++++
 2 files changed, 44 insertions(+), 8 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 45836f0a2b0a..4b1b310b5d68 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1402,17 +1402,13 @@ int v4l2_subdev_link_validate(struct media_link *link)
 
 	states_locked = sink_state && source_state;
 
-	if (states_locked) {
-		v4l2_subdev_lock_state(sink_state);
-		v4l2_subdev_lock_state(source_state);
-	}
+	if (states_locked)
+		v4l2_subdev_lock_states(sink_state, source_state);
 
 	ret = v4l2_subdev_link_validate_locked(link, states_locked);
 
-	if (states_locked) {
-		v4l2_subdev_unlock_state(sink_state);
-		v4l2_subdev_unlock_state(source_state);
-	}
+	if (states_locked)
+		v4l2_subdev_unlock_states(sink_state, source_state);
 
 	return ret;
 }
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index a9e6b8146279..9cce48365975 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1724,6 +1724,46 @@ static inline void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state)
 	mutex_unlock(state->lock);
 }
 
+/**
+ * v4l2_subdev_lock_states - Lock two sub-device states
+ * @state1: One subdevice state
+ * @state2: The other subdevice state
+ *
+ * Locks the state of two sub-devices.
+ *
+ * The states must be unlocked with v4l2_subdev_unlock_states() after use.
+ *
+ * This differs from calling v4l2_subdev_lock_state() on both states so that if
+ * the states share the same lock, the lock is acquired only once (so no
+ * deadlock occurs). The caller is responsible for ensuring the locks will
+ * always be acquired in the same order.
+ */
+static inline void v4l2_subdev_lock_states(struct v4l2_subdev_state *state1,
+					   struct v4l2_subdev_state *state2)
+{
+	mutex_lock(state1->lock);
+	if (state1->lock != state2->lock)
+		mutex_lock(state2->lock);
+}
+
+/**
+ * v4l2_subdev_unlock_states() - Unlock two sub-device states
+ * @state1: One subdevice state
+ * @state2: The other subdevice state
+ *
+ * Unlocks the state of two sub-devices.
+ *
+ * This differs from calling v4l2_subdev_unlock_state() on both states so that
+ * if the states share the same lock, the lock is released only once.
+ */
+static inline void v4l2_subdev_unlock_states(struct v4l2_subdev_state *state1,
+					     struct v4l2_subdev_state *state2)
+{
+	mutex_unlock(state1->lock);
+	if (state1->lock != state2->lock)
+		mutex_unlock(state2->lock);
+}
+
 /**
  * v4l2_subdev_get_unlocked_active_state() - Checks that the active subdev state
  *					     is unlocked and returns it
-- 
2.39.2

[PATCH v9 13/46] media: v4l: subdev: Move G_ROUTING handling below S_ROUTING

[Sakari Ailus]

Move G_ROUTING IOCTL handling below that of S_ROUTING. G_ROUTING
implementation will soon needed in handling S_ROUTING as well.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Julien Massot <julien.massot@collabora.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 54 +++++++++++++--------------
 1 file changed, 27 insertions(+), 27 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 4b1b310b5d68..2ba11e5343f0 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -911,33 +911,6 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 	case VIDIOC_SUBDEV_QUERYSTD:
 		return v4l2_subdev_call(sd, video, querystd, arg);
 
-	case VIDIOC_SUBDEV_G_ROUTING: {
-		struct v4l2_subdev_routing *routing = arg;
-		struct v4l2_subdev_krouting *krouting;
-
-		if (!v4l2_subdev_enable_streams_api)
-			return -ENOIOCTLCMD;
-
-		if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS))
-			return -ENOIOCTLCMD;
-
-		memset(routing->reserved, 0, sizeof(routing->reserved));
-
-		krouting = &state->routing;
-
-		if (routing->num_routes < krouting->num_routes) {
-			routing->num_routes = krouting->num_routes;
-			return -ENOSPC;
-		}
-
-		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
-		       krouting->routes,
-		       krouting->num_routes * sizeof(*krouting->routes));
-		routing->num_routes = krouting->num_routes;
-
-		return 0;
-	}
-
 	case VIDIOC_SUBDEV_S_ROUTING: {
 		struct v4l2_subdev_routing *routing = arg;
 		struct v4l2_subdev_route *routes =
@@ -986,6 +959,33 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 					routing->which, &krouting);
 	}
 
+	case VIDIOC_SUBDEV_G_ROUTING: {
+		struct v4l2_subdev_routing *routing = arg;
+		struct v4l2_subdev_krouting *krouting;
+
+		if (!v4l2_subdev_enable_streams_api)
+			return -ENOIOCTLCMD;
+
+		if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS))
+			return -ENOIOCTLCMD;
+
+		memset(routing->reserved, 0, sizeof(routing->reserved));
+
+		krouting = &state->routing;
+
+		if (routing->num_routes < krouting->num_routes) {
+			routing->num_routes = krouting->num_routes;
+			return -ENOSPC;
+		}
+
+		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
+		       krouting->routes,
+		       krouting->num_routes * sizeof(*krouting->routes));
+		routing->num_routes = krouting->num_routes;
+
+		return 0;
+	}
+
 	case VIDIOC_SUBDEV_G_CLIENT_CAP: {
 		struct v4l2_subdev_client_capability *client_cap = arg;
 
-- 
2.39.2

[PATCH v9 14/46] media: v4l: subdev: Copy argument back to user also for S_ROUTING

[Sakari Ailus]

As the user needs to know what went wrong for S_ROUTING, copy array
arguments back to the user.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Julien Massot <julien.massot@collabora.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-ioctl.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index 2cfc9106857a..1863ecae9ec9 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -3445,11 +3445,14 @@ video_usercopy(struct file *file, unsigned int orig_cmd, unsigned long arg,
 	 * FIXME: subdev IOCTLS are partially handled here and partially in
 	 * v4l2-subdev.c and the 'always_copy' flag can only be set for IOCTLS
 	 * defined here as part of the 'v4l2_ioctls' array. As
-	 * VIDIOC_SUBDEV_G_ROUTING needs to return results to applications even
-	 * in case of failure, but it is not defined here as part of the
+	 * VIDIOC_SUBDEV_[GS]_ROUTING needs to return results to applications
+	 * even in case of failure, but it is not defined here as part of the
 	 * 'v4l2_ioctls' array, insert an ad-hoc check to address that.
 	 */
-	if (err < 0 && !always_copy && cmd != VIDIOC_SUBDEV_G_ROUTING)
+	if (cmd == VIDIOC_SUBDEV_G_ROUTING || cmd == VIDIOC_SUBDEV_S_ROUTING)
+		always_copy = true;
+
+	if (err < 0 && !always_copy)
 		goto out;
 
 	if (has_array_args) {
-- 
2.39.2

[PATCH v9 15/46] media: v4l: subdev: Add len_routes field to struct v4l2_subdev_routing

[Sakari Ailus]

The len_routes field is used to tell the size of the routes array in
struct v4l2_subdev_routing. This way the number of routes returned from
S_ROUTING IOCTL may be larger than the number of routes provided, in case
there are more routes returned by the driver.

Note that this uAPI is still disabled in the code, so this change can
safely be done. Anyone who manually patched the code to enable this uAPI
must update their code.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../media/v4l/vidioc-subdev-g-routing.rst     | 50 +++++++++++++------
 drivers/media/v4l2-core/v4l2-ioctl.c          |  4 +-
 drivers/media/v4l2-core/v4l2-subdev.c         | 12 ++---
 include/media/v4l2-subdev.h                   |  2 +
 include/uapi/linux/v4l2-subdev.h              |  9 ++--
 5 files changed, 52 insertions(+), 25 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
index 26b5004bfe6d..27eb4c82a0e1 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
@@ -43,23 +43,42 @@ The routing configuration determines the flows of data inside an entity.
 Drivers report their current routing tables using the
 ``VIDIOC_SUBDEV_G_ROUTING`` ioctl and application may enable or disable routes
 with the ``VIDIOC_SUBDEV_S_ROUTING`` ioctl, by adding or removing routes and
-setting or clearing flags of the  ``flags`` field of a
-struct :c:type:`v4l2_subdev_route`.
+setting or clearing flags of the ``flags`` field of a struct
+:c:type:`v4l2_subdev_route`.
 
-All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called. This
-means that the userspace must reconfigure all streams after calling the ioctl
-with e.g. ``VIDIOC_SUBDEV_S_FMT``.
+All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called.
+This means that the userspace must reconfigure all stream formats and selections
+after calling the ioctl with e.g. ``VIDIOC_SUBDEV_S_FMT``.
 
 Only subdevices which have both sink and source pads can support routing.
 
-When inspecting routes through ``VIDIOC_SUBDEV_G_ROUTING`` and the application
-provided ``num_routes`` is not big enough to contain all the available routes
-the subdevice exposes, drivers return the ENOSPC error code and adjust the
-value of the ``num_routes`` field. Application should then reserve enough memory
-for all the route entries and call ``VIDIOC_SUBDEV_G_ROUTING`` again.
-
-On a successful ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the
-``num_routes`` field to reflect the actual number of routes returned.
+The ``len_routes`` field indicates the number of routes that can fit in the
+``routes`` array allocated by userspace. It is set by applications for both
+ioctls to indicate how many routes the kernel can return, and is never modified
+by the kernel.
+
+The ``num_routes`` field, when returned from the kernel on both IOCTLs,
+indicates the number of routes in the subdevice routing table and when calling
+``VIDIOC_SUBDEV_S_ROUTING``, it is set by userspace to the number of routes that
+the application stored in the ``routes`` array. The value returned by the kernel
+may be smaller or larger than the value of ``num_routes`` set by the application
+for ``VIDIOC_SUBDEV_S_ROUTING``, as drivers may adjust the requested routing
+table.
+
+The kernel can return a ``num_routes`` value larger than ``len_routes`` from
+both ioctls. This indicates thare are more routes in the routing table than fits
+the ``routes`` array. In this case, the ``routes`` array is filled by the kernel
+with the first ``len_routes`` entries of the subdevice routing table. This is
+not considered to be an error, and the ioctl call succeeds. If the applications
+wants to retrieve the missing routes, it can issue a new
+``VIDIOC_SUBDEV_G_ROUTING`` call with a large enough ``routes`` array.
+
+indicate there are more routes than fits to the ``routes`` array. In this
+case first ``len_routes`` were returned back to the userspace in the
+``routes`` array. This is not considered as an error.
+
+Also ``VIDIOC_SUBDEV_S_ROUTING`` may return more route than the user provided in
+``num_routes`` field due to e.g. hardware properties.
 
 .. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
 
@@ -74,6 +93,9 @@ On a successful ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the
       - ``which``
       - Routing table to be accessed, from enum
         :ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
+    * - __u32
+      - ``len_routes``
+      - The length of the array (as in memory reserved for the array)
     * - struct :c:type:`v4l2_subdev_route`
       - ``routes[]``
       - Array of struct :c:type:`v4l2_subdev_route` entries
@@ -81,7 +103,7 @@ On a successful ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the
       - ``num_routes``
       - Number of entries of the routes array
     * - __u32
-      - ``reserved``\ [5]
+      - ``reserved``\ [11]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index 1863ecae9ec9..f30f61c008c7 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -3185,13 +3185,13 @@ static int check_array_args(unsigned int cmd, void *parg, size_t *array_size,
 	case VIDIOC_SUBDEV_S_ROUTING: {
 		struct v4l2_subdev_routing *routing = parg;
 
-		if (routing->num_routes > 256)
+		if (routing->len_routes > 256)
 			return -E2BIG;
 
 		*user_ptr = u64_to_user_ptr(routing->routes);
 		*kernel_ptr = (void **)&routing->routes;
 		*array_size = sizeof(struct v4l2_subdev_route)
-			    * routing->num_routes;
+			    * routing->len_routes;
 		ret = 1;
 		break;
 	}
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 2ba11e5343f0..904378007a79 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -927,6 +927,9 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 		if (routing->which != V4L2_SUBDEV_FORMAT_TRY && ro_subdev)
 			return -EPERM;
 
+		if (routing->num_routes > routing->len_routes)
+			return -EINVAL;
+
 		memset(routing->reserved, 0, sizeof(routing->reserved));
 
 		for (i = 0; i < routing->num_routes; ++i) {
@@ -953,6 +956,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 		}
 
 		krouting.num_routes = routing->num_routes;
+		krouting.len_routes = routing->len_routes;
 		krouting.routes = routes;
 
 		return v4l2_subdev_call(sd, pad, set_routing, state,
@@ -973,14 +977,10 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 
 		krouting = &state->routing;
 
-		if (routing->num_routes < krouting->num_routes) {
-			routing->num_routes = krouting->num_routes;
-			return -ENOSPC;
-		}
-
 		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
 		       krouting->routes,
-		       krouting->num_routes * sizeof(*krouting->routes));
+		       min(krouting->num_routes, routing->len_routes) *
+		       sizeof(*krouting->routes));
 		routing->num_routes = krouting->num_routes;
 
 		return 0;
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 9cce48365975..1df6b963a1c9 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -728,12 +728,14 @@ struct v4l2_subdev_stream_configs {
 /**
  * struct v4l2_subdev_krouting - subdev routing table
  *
+ * @len_routes: length of routes array, in routes
  * @num_routes: number of routes
  * @routes: &struct v4l2_subdev_route
  *
  * This structure contains the routing table for a subdev.
  */
 struct v4l2_subdev_krouting {
+	unsigned int len_routes;
 	unsigned int num_routes;
 	struct v4l2_subdev_route *routes;
 };
diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
index 81a24bd38003..6a39128d0606 100644
--- a/include/uapi/linux/v4l2-subdev.h
+++ b/include/uapi/linux/v4l2-subdev.h
@@ -228,15 +228,18 @@ struct v4l2_subdev_route {
  * struct v4l2_subdev_routing - Subdev routing information
  *
  * @which: configuration type (from enum v4l2_subdev_format_whence)
- * @num_routes: the total number of routes in the routes array
+ * @len_routes: the length of the routes array, in routes
  * @routes: pointer to the routes array
+ * @num_routes: the total number of routes, possibly more than fits in the
+ *		routes array
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_routing {
 	__u32 which;
-	__u32 num_routes;
+	__u32 len_routes;
 	__u64 routes;
-	__u32 reserved[6];
+	__u32 num_routes;
+	__u32 reserved[11];
 };
 
 /*
-- 
2.39.2

[PATCH v9 16/46] media: v4l: subdev: Return routes set using S_ROUTING

[Sakari Ailus]

Return the routes set using S_ROUTING back to the user. Also reflect this
in documentation.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../media/v4l/vidioc-subdev-g-routing.rst            |  7 ++-----
 drivers/media/v4l2-core/v4l2-subdev.c                | 12 +++++++++++-
 2 files changed, 13 insertions(+), 6 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
index 27eb4c82a0e1..88df7bf80b00 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
@@ -44,7 +44,8 @@ Drivers report their current routing tables using the
 ``VIDIOC_SUBDEV_G_ROUTING`` ioctl and application may enable or disable routes
 with the ``VIDIOC_SUBDEV_S_ROUTING`` ioctl, by adding or removing routes and
 setting or clearing flags of the ``flags`` field of a struct
-:c:type:`v4l2_subdev_route`.
+:c:type:`v4l2_subdev_route`. Similarly to ``VIDIOC_SUBDEV_G_ROUTING``, also
+``VIDIOC_SUBDEV_S_ROUTING`` returns the routes back to the user.
 
 All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called.
 This means that the userspace must reconfigure all stream formats and selections
@@ -157,10 +158,6 @@ On success 0 is returned, on error -1 and the ``errno`` variable is set
 appropriately. The generic error codes are described at the
 :ref:`Generic Error Codes <gen-errors>` chapter.
 
-ENOSPC
-   The application provided ``num_routes`` is not big enough to contain
-   all the available routes the subdevice exposes.
-
 EINVAL
    The sink or source pad identifiers reference a non-existing pad or reference
    pads of different types (ie. the sink_pad identifiers refers to a source
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 904378007a79..36b2f78cd551 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -959,8 +959,18 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 		krouting.len_routes = routing->len_routes;
 		krouting.routes = routes;
 
-		return v4l2_subdev_call(sd, pad, set_routing, state,
+		rval = v4l2_subdev_call(sd, pad, set_routing, state,
 					routing->which, &krouting);
+		if (rval < 0)
+			return rval;
+
+		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
+		       state->routing.routes,
+		       min(state->routing.num_routes, routing->len_routes) *
+		       sizeof(*state->routing.routes));
+		routing->num_routes = state->routing.num_routes;
+
+		return 0;
 	}
 
 	case VIDIOC_SUBDEV_G_ROUTING: {
-- 
2.39.2

[PATCH v9 17/46] media: v4l: subdev: Add trivial set_routing support

[Sakari Ailus]

Add trivial S_ROUTING IOCTL support for drivers where routing is static.
Essentially this means returning the same information G_ROUTING call would
have done.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Co-developed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Julien Massot <julien.massot@collabora.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 36b2f78cd551..e68f16a7c041 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -955,6 +955,20 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 				return -EINVAL;
 		}
 
+		/*
+		 * If the driver doesn't support setting routing, just return
+		 * the routing table.
+		 */
+		if (!v4l2_subdev_has_op(sd, pad, set_routing)) {
+			memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
+			       state->routing.routes,
+			       min(state->routing.num_routes, routing->len_routes) *
+			       sizeof(*state->routing.routes));
+			routing->num_routes = state->routing.num_routes;
+
+			return 0;
+		}
+
 		krouting.num_routes = routing->num_routes;
 		krouting.len_routes = routing->len_routes;
 		krouting.routes = routes;
-- 
2.39.2

[PATCH v9 18/46] media: ccs: No need to set streaming to false in power off

[Sakari Ailus]

Streaming will have been stopped before the sensor is powered off, and so
sensor->streaming is also false already. Do not set it as part of the
runtime suspend callback.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 drivers/media/i2c/ccs/ccs-core.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
index e1ae0f9fad43..7e5474e38732 100644
--- a/drivers/media/i2c/ccs/ccs-core.c
+++ b/drivers/media/i2c/ccs/ccs-core.c
@@ -1710,7 +1710,6 @@ static int ccs_power_off(struct device *dev)
 	usleep_range(5000, 5000);
 	regulator_bulk_disable(ARRAY_SIZE(ccs_regulators),
 			       sensor->regulators);
-	sensor->streaming = false;
 
 	return 0;
 }
-- 
2.39.2

[PATCH v9 19/46] media: ccs: Move ccs_pm_get_init function up

[Sakari Ailus]

Prepare for using ccs_pm_get_init from locations earlier than its the
current place.

Also add a missing newline while at it.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ccs/ccs-core.c | 73 ++++++++++++++++----------------
 1 file changed, 37 insertions(+), 36 deletions(-)

diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
index 7e5474e38732..d7bc9418eabb 100644
--- a/drivers/media/i2c/ccs/ccs-core.c
+++ b/drivers/media/i2c/ccs/ccs-core.c
@@ -1718,6 +1718,43 @@ static int ccs_power_off(struct device *dev)
  * Video stream management
  */
 
+static int ccs_pm_get_init(struct ccs_sensor *sensor)
+{
+	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
+	int rval;
+
+	/*
+	 * It can't use pm_runtime_resume_and_get() here, as the driver
+	 * relies at the returned value to detect if the device was already
+	 * active or not.
+	 */
+	rval = pm_runtime_get_sync(&client->dev);
+	if (rval < 0)
+		goto error;
+
+	/* Device was already active, so don't set controls */
+	if (rval == 1 && !sensor->handler_setup_needed)
+		return 0;
+
+	sensor->handler_setup_needed = false;
+
+	/* Restore V4L2 controls to the previously suspended device */
+	rval = v4l2_ctrl_handler_setup(&sensor->pixel_array->ctrl_handler);
+	if (rval)
+		goto error;
+
+	rval = v4l2_ctrl_handler_setup(&sensor->src->ctrl_handler);
+	if (rval)
+		goto error;
+
+	/* Keep PM runtime usage_count incremented on success */
+	return 0;
+
+error:
+	pm_runtime_put(&client->dev);
+	return rval;
+}
+
 static int ccs_start_streaming(struct ccs_sensor *sensor)
 {
 	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
@@ -1872,42 +1909,6 @@ static int ccs_stop_streaming(struct ccs_sensor *sensor)
  * V4L2 subdev video operations
  */
 
-static int ccs_pm_get_init(struct ccs_sensor *sensor)
-{
-	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
-	int rval;
-
-	/*
-	 * It can't use pm_runtime_resume_and_get() here, as the driver
-	 * relies at the returned value to detect if the device was already
-	 * active or not.
-	 */
-	rval = pm_runtime_get_sync(&client->dev);
-	if (rval < 0)
-		goto error;
-
-	/* Device was already active, so don't set controls */
-	if (rval == 1 && !sensor->handler_setup_needed)
-		return 0;
-
-	sensor->handler_setup_needed = false;
-
-	/* Restore V4L2 controls to the previously suspended device */
-	rval = v4l2_ctrl_handler_setup(&sensor->pixel_array->ctrl_handler);
-	if (rval)
-		goto error;
-
-	rval = v4l2_ctrl_handler_setup(&sensor->src->ctrl_handler);
-	if (rval)
-		goto error;
-
-	/* Keep PM runtime usage_count incremented on success */
-	return 0;
-error:
-	pm_runtime_put(&client->dev);
-	return rval;
-}
-
 static int ccs_set_stream(struct v4l2_subdev *subdev, int enable)
 {
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
-- 
2.39.2

[PATCH v9 20/46] media: ccs: Rename out label of ccs_start_streaming

[Sakari Ailus]

In preparation for upcoming changes in the function, rename the out label
as err_pm_put. The purpose of the label is changed to match its name in
the next patch.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ccs/ccs-core.c | 38 ++++++++++++++++----------------
 1 file changed, 19 insertions(+), 19 deletions(-)

diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
index d7bc9418eabb..d14e90f8568a 100644
--- a/drivers/media/i2c/ccs/ccs-core.c
+++ b/drivers/media/i2c/ccs/ccs-core.c
@@ -1767,7 +1767,7 @@ static int ccs_start_streaming(struct ccs_sensor *sensor)
 			 (sensor->csi_format->width << 8) |
 			 sensor->csi_format->compressed);
 	if (rval)
-		goto out;
+		goto err_pm_put;
 
 	/* Binning configuration */
 	if (sensor->binning_horizontal == 1 &&
@@ -1780,38 +1780,38 @@ static int ccs_start_streaming(struct ccs_sensor *sensor)
 
 		rval = ccs_write(sensor, BINNING_TYPE, binning_type);
 		if (rval < 0)
-			goto out;
+			goto err_pm_put;
 
 		binning_mode = 1;
 	}
 	rval = ccs_write(sensor, BINNING_MODE, binning_mode);
 	if (rval < 0)
-		goto out;
+		goto err_pm_put;
 
 	/* Set up PLL */
 	rval = ccs_pll_configure(sensor);
 	if (rval)
-		goto out;
+		goto err_pm_put;
 
 	/* Analog crop start coordinates */
 	rval = ccs_write(sensor, X_ADDR_START, sensor->pa_src.left);
 	if (rval < 0)
-		goto out;
+		goto err_pm_put;
 
 	rval = ccs_write(sensor, Y_ADDR_START, sensor->pa_src.top);
 	if (rval < 0)
-		goto out;
+		goto err_pm_put;
 
 	/* Analog crop end coordinates */
 	rval = ccs_write(sensor, X_ADDR_END,
 			 sensor->pa_src.left + sensor->pa_src.width - 1);
 	if (rval < 0)
-		goto out;
+		goto err_pm_put;
 
 	rval = ccs_write(sensor, Y_ADDR_END,
 			 sensor->pa_src.top + sensor->pa_src.height - 1);
 	if (rval < 0)
-		goto out;
+		goto err_pm_put;
 
 	/*
 	 * Output from pixel array, including blanking, is set using
@@ -1824,22 +1824,22 @@ static int ccs_start_streaming(struct ccs_sensor *sensor)
 		rval = ccs_write(sensor, DIGITAL_CROP_X_OFFSET,
 				 sensor->scaler_sink.left);
 		if (rval < 0)
-			goto out;
+			goto err_pm_put;
 
 		rval = ccs_write(sensor, DIGITAL_CROP_Y_OFFSET,
 				 sensor->scaler_sink.top);
 		if (rval < 0)
-			goto out;
+			goto err_pm_put;
 
 		rval = ccs_write(sensor, DIGITAL_CROP_IMAGE_WIDTH,
 				 sensor->scaler_sink.width);
 		if (rval < 0)
-			goto out;
+			goto err_pm_put;
 
 		rval = ccs_write(sensor, DIGITAL_CROP_IMAGE_HEIGHT,
 				 sensor->scaler_sink.height);
 		if (rval < 0)
-			goto out;
+			goto err_pm_put;
 	}
 
 	/* Scaling */
@@ -1847,20 +1847,20 @@ static int ccs_start_streaming(struct ccs_sensor *sensor)
 	    != CCS_SCALING_CAPABILITY_NONE) {
 		rval = ccs_write(sensor, SCALING_MODE, sensor->scaling_mode);
 		if (rval < 0)
-			goto out;
+			goto err_pm_put;
 
 		rval = ccs_write(sensor, SCALE_M, sensor->scale_m);
 		if (rval < 0)
-			goto out;
+			goto err_pm_put;
 	}
 
 	/* Output size from sensor */
 	rval = ccs_write(sensor, X_OUTPUT_SIZE, sensor->src_src.width);
 	if (rval < 0)
-		goto out;
+		goto err_pm_put;
 	rval = ccs_write(sensor, Y_OUTPUT_SIZE, sensor->src_src.height);
 	if (rval < 0)
-		goto out;
+		goto err_pm_put;
 
 	if (CCS_LIM(sensor, FLASH_MODE_CAPABILITY) &
 	    (CCS_FLASH_MODE_CAPABILITY_SINGLE_STROBE |
@@ -1869,18 +1869,18 @@ static int ccs_start_streaming(struct ccs_sensor *sensor)
 	    sensor->hwcfg.strobe_setup->trigger != 0) {
 		rval = ccs_setup_flash_strobe(sensor);
 		if (rval)
-			goto out;
+			goto err_pm_put;
 	}
 
 	rval = ccs_call_quirk(sensor, pre_streamon);
 	if (rval) {
 		dev_err(&client->dev, "pre_streamon quirks failed\n");
-		goto out;
+		goto err_pm_put;
 	}
 
 	rval = ccs_write(sensor, MODE_SELECT, CCS_MODE_SELECT_STREAMING);
 
-out:
+err_pm_put:
 	mutex_unlock(&sensor->mutex);
 
 	return rval;
-- 
2.39.2

[PATCH v9 21/46] media: ccs: Use {enable,disable}_streams operations

[Sakari Ailus]

Switch from s_stream() video op to enable_streams() and disable_streams()
pad operations. They are preferred and required for streams support.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ccs/ccs-core.c | 83 ++++++++++++++------------------
 1 file changed, 36 insertions(+), 47 deletions(-)

diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
index d14e90f8568a..aaa6bf8495e1 100644
--- a/drivers/media/i2c/ccs/ccs-core.c
+++ b/drivers/media/i2c/ccs/ccs-core.c
@@ -1715,7 +1715,7 @@ static int ccs_power_off(struct device *dev)
 }
 
 /* -----------------------------------------------------------------------------
- * Video stream management
+ * V4L2 subdev video operations
  */
 
 static int ccs_pm_get_init(struct ccs_sensor *sensor)
@@ -1739,11 +1739,11 @@ static int ccs_pm_get_init(struct ccs_sensor *sensor)
 	sensor->handler_setup_needed = false;
 
 	/* Restore V4L2 controls to the previously suspended device */
-	rval = v4l2_ctrl_handler_setup(&sensor->pixel_array->ctrl_handler);
+	rval = __v4l2_ctrl_handler_setup(&sensor->pixel_array->ctrl_handler);
 	if (rval)
 		goto error;
 
-	rval = v4l2_ctrl_handler_setup(&sensor->src->ctrl_handler);
+	rval = __v4l2_ctrl_handler_setup(&sensor->src->ctrl_handler);
 	if (rval)
 		goto error;
 
@@ -1755,13 +1755,21 @@ static int ccs_pm_get_init(struct ccs_sensor *sensor)
 	return rval;
 }
 
-static int ccs_start_streaming(struct ccs_sensor *sensor)
+static int ccs_enable_streams(struct v4l2_subdev *subdev,
+			      struct v4l2_subdev_state *state, u32 pad,
+			      u64 streams_mask)
 {
+	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
 	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
 	unsigned int binning_mode;
 	int rval;
 
-	mutex_lock(&sensor->mutex);
+	if (pad != CCS_PAD_SRC)
+		return -EINVAL;
+
+	rval = ccs_pm_get_init(sensor);
+	if (rval)
+		return rval;
 
 	rval = ccs_write(sensor, CSI_DATA_FORMAT,
 			 (sensor->csi_format->width << 8) |
@@ -1880,64 +1888,41 @@ static int ccs_start_streaming(struct ccs_sensor *sensor)
 
 	rval = ccs_write(sensor, MODE_SELECT, CCS_MODE_SELECT_STREAMING);
 
-err_pm_put:
-	mutex_unlock(&sensor->mutex);
-
-	return rval;
-}
-
-static int ccs_stop_streaming(struct ccs_sensor *sensor)
-{
-	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
-	int rval;
+	sensor->streaming = true;
 
-	mutex_lock(&sensor->mutex);
-	rval = ccs_write(sensor, MODE_SELECT, CCS_MODE_SELECT_SOFTWARE_STANDBY);
-	if (rval)
-		goto out;
+	return 0;
 
-	rval = ccs_call_quirk(sensor, post_streamoff);
-	if (rval)
-		dev_err(&client->dev, "post_streamoff quirks failed\n");
+err_pm_put:
+	pm_runtime_mark_last_busy(&client->dev);
+	pm_runtime_put_autosuspend(&client->dev);
 
-out:
-	mutex_unlock(&sensor->mutex);
 	return rval;
 }
 
-/* -----------------------------------------------------------------------------
- * V4L2 subdev video operations
- */
-
-static int ccs_set_stream(struct v4l2_subdev *subdev, int enable)
+static int ccs_disable_streams(struct v4l2_subdev *subdev,
+			       struct v4l2_subdev_state *state, u32 pad,
+			       u64 streams_mask)
 {
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
 	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
 	int rval;
 
-	if (!enable) {
-		ccs_stop_streaming(sensor);
-		sensor->streaming = false;
-		pm_runtime_mark_last_busy(&client->dev);
-		pm_runtime_put_autosuspend(&client->dev);
-
-		return 0;
-	}
+	if (pad != CCS_PAD_SRC)
+		return -EINVAL;
 
-	rval = ccs_pm_get_init(sensor);
+	rval = ccs_write(sensor, MODE_SELECT, CCS_MODE_SELECT_SOFTWARE_STANDBY);
 	if (rval)
 		return rval;
 
-	sensor->streaming = true;
+	rval = ccs_call_quirk(sensor, post_streamoff);
+	if (rval)
+		dev_err(&client->dev, "post_streamoff quirks failed\n");
 
-	rval = ccs_start_streaming(sensor);
-	if (rval < 0) {
-		sensor->streaming = false;
-		pm_runtime_mark_last_busy(&client->dev);
-		pm_runtime_put_autosuspend(&client->dev);
-	}
+	sensor->streaming = false;
+	pm_runtime_mark_last_busy(&client->dev);
+	pm_runtime_put_autosuspend(&client->dev);
 
-	return rval;
+	return 0;
 }
 
 static int ccs_pre_streamon(struct v4l2_subdev *subdev, u32 flags)
@@ -1963,7 +1948,9 @@ static int ccs_pre_streamon(struct v4l2_subdev *subdev, u32 flags)
 		}
 	}
 
+	mutex_lock(&sensor->mutex);
 	rval = ccs_pm_get_init(sensor);
+	mutex_unlock(&sensor->mutex);
 	if (rval)
 		return rval;
 
@@ -3047,7 +3034,7 @@ static int ccs_init_state(struct v4l2_subdev *sd,
 }
 
 static const struct v4l2_subdev_video_ops ccs_video_ops = {
-	.s_stream = ccs_set_stream,
+	.s_stream = v4l2_subdev_s_stream_helper,
 	.pre_streamon = ccs_pre_streamon,
 	.post_streamoff = ccs_post_streamoff,
 };
@@ -3058,6 +3045,8 @@ static const struct v4l2_subdev_pad_ops ccs_pad_ops = {
 	.set_fmt = ccs_set_format,
 	.get_selection = ccs_get_selection,
 	.set_selection = ccs_set_selection,
+	.enable_streams = ccs_enable_streams,
+	.disable_streams = ccs_disable_streams,
 };
 
 static const struct v4l2_subdev_sensor_ops ccs_sensor_ops = {
-- 
2.39.2

[PATCH v9 22/46] media: ccs: Track streaming state

[Sakari Ailus]

With enable_streams and disable_streams, the driver for a device where
streams are not independently started and stopped needs to maintain state
information on streams that have been requested to be started. Do that
now.

In the future, a helper function in the framework is a desirable way to do
this instead.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Julien Massot <julien.massot@collabora.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 drivers/media/i2c/ccs/ccs-core.c | 13 ++++++++++---
 drivers/media/i2c/ccs/ccs.h      |  2 +-
 2 files changed, 11 insertions(+), 4 deletions(-)

diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
index aaa6bf8495e1..c39ee4394534 100644
--- a/drivers/media/i2c/ccs/ccs-core.c
+++ b/drivers/media/i2c/ccs/ccs-core.c
@@ -1767,6 +1767,11 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
 	if (pad != CCS_PAD_SRC)
 		return -EINVAL;
 
+	if (sensor->streaming) {
+		sensor->streaming |= streams_mask;
+		return 0;
+	}
+
 	rval = ccs_pm_get_init(sensor);
 	if (rval)
 		return rval;
@@ -1888,7 +1893,7 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
 
 	rval = ccs_write(sensor, MODE_SELECT, CCS_MODE_SELECT_STREAMING);
 
-	sensor->streaming = true;
+	sensor->streaming |= streams_mask;
 
 	return 0;
 
@@ -1910,6 +1915,10 @@ static int ccs_disable_streams(struct v4l2_subdev *subdev,
 	if (pad != CCS_PAD_SRC)
 		return -EINVAL;
 
+	sensor->streaming &= ~streams_mask;
+	if (sensor->streaming)
+		return 0;
+
 	rval = ccs_write(sensor, MODE_SELECT, CCS_MODE_SELECT_SOFTWARE_STANDBY);
 	if (rval)
 		return rval;
@@ -1918,7 +1927,6 @@ static int ccs_disable_streams(struct v4l2_subdev *subdev,
 	if (rval)
 		dev_err(&client->dev, "post_streamoff quirks failed\n");
 
-	sensor->streaming = false;
 	pm_runtime_mark_last_busy(&client->dev);
 	pm_runtime_put_autosuspend(&client->dev);
 
@@ -3526,7 +3534,6 @@ static int ccs_probe(struct i2c_client *client)
 		goto out_cleanup;
 	}
 
-	sensor->streaming = false;
 	sensor->dev_init_done = true;
 	sensor->handler_setup_needed = true;
 
diff --git a/drivers/media/i2c/ccs/ccs.h b/drivers/media/i2c/ccs/ccs.h
index 096573845a10..4725e6eca8d0 100644
--- a/drivers/media/i2c/ccs/ccs.h
+++ b/drivers/media/i2c/ccs/ccs.h
@@ -236,7 +236,7 @@ struct ccs_sensor {
 	u16 image_start; /* image data start line */
 	u16 visible_pixel_start; /* start pixel of the visible image */
 
-	bool streaming;
+	u8 streaming;
 	bool dev_init_done;
 	bool handler_setup_needed;
 	u8 compressed_min_bpp;
-- 
2.39.2

[PATCH v9 23/46] media: ccs: Move ccs_validate_csi_data_format up

[Sakari Ailus]

ccs_validate_csi_data_format() will soon be needed elsewhere, above its
current location. Move it up.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Julien Massot <julien.massot@collabora.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 drivers/media/i2c/ccs/ccs-core.c | 28 ++++++++++++++--------------
 1 file changed, 14 insertions(+), 14 deletions(-)

diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
index c39ee4394534..a711233f6fbf 100644
--- a/drivers/media/i2c/ccs/ccs-core.c
+++ b/drivers/media/i2c/ccs/ccs-core.c
@@ -1980,6 +1980,20 @@ static int ccs_post_streamoff(struct v4l2_subdev *subdev)
 	return pm_runtime_put(&client->dev);
 }
 
+static const struct ccs_csi_data_format
+*ccs_validate_csi_data_format(struct ccs_sensor *sensor, u32 code)
+{
+	unsigned int i;
+
+	for (i = 0; i < ARRAY_SIZE(ccs_csi_data_formats); i++) {
+		if (sensor->mbus_frame_fmts & (1 << i) &&
+		    ccs_csi_data_formats[i].code == code)
+			return &ccs_csi_data_formats[i];
+	}
+
+	return sensor->csi_format;
+}
+
 static int ccs_enum_mbus_code(struct v4l2_subdev *subdev,
 			      struct v4l2_subdev_state *sd_state,
 			      struct v4l2_subdev_mbus_code_enum *code)
@@ -2115,20 +2129,6 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
 	}
 }
 
-static const struct ccs_csi_data_format
-*ccs_validate_csi_data_format(struct ccs_sensor *sensor, u32 code)
-{
-	unsigned int i;
-
-	for (i = 0; i < ARRAY_SIZE(ccs_csi_data_formats); i++) {
-		if (sensor->mbus_frame_fmts & (1 << i) &&
-		    ccs_csi_data_formats[i].code == code)
-			return &ccs_csi_data_formats[i];
-	}
-
-	return sensor->csi_format;
-}
-
 static int ccs_set_format_source(struct v4l2_subdev *subdev,
 				 struct v4l2_subdev_state *sd_state,
 				 struct v4l2_subdev_format *fmt)
-- 
2.39.2

[PATCH v9 24/46] media: ccs: Support frame descriptors

[Sakari Ailus]

Provide information on the frame layout using frame descriptors.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Julien Massot <julien.massot@collabora.com>
---
 drivers/media/i2c/ccs/ccs-core.c | 57 ++++++++++++++++++++++++++++++++
 drivers/media/i2c/ccs/ccs.h      |  4 +++
 2 files changed, 61 insertions(+)

diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
index a711233f6fbf..3ca2415fca3b 100644
--- a/drivers/media/i2c/ccs/ccs-core.c
+++ b/drivers/media/i2c/ccs/ccs-core.c
@@ -25,6 +25,7 @@
 #include <linux/slab.h>
 #include <linux/smiapp.h>
 #include <linux/v4l2-mediabus.h>
+#include <media/mipi-csi2.h>
 #include <media/v4l2-cci.h>
 #include <media/v4l2-device.h>
 #include <media/v4l2-fwnode.h>
@@ -245,6 +246,33 @@ static int ccs_read_all_limits(struct ccs_sensor *sensor)
 	return ret;
 }
 
+static u8 ccs_mipi_csi2_data_type(unsigned int bpp)
+{
+	switch (bpp) {
+	case 6:
+		return MIPI_CSI2_DT_RAW6;
+	case 7:
+		return MIPI_CSI2_DT_RAW7;
+	case 8:
+		return MIPI_CSI2_DT_RAW8;
+	case 10:
+		return MIPI_CSI2_DT_RAW10;
+	case 12:
+		return MIPI_CSI2_DT_RAW12;
+	case 14:
+		return MIPI_CSI2_DT_RAW14;
+	case 16:
+		return MIPI_CSI2_DT_RAW16;
+	case 20:
+		return MIPI_CSI2_DT_RAW20;
+	case 24:
+		return MIPI_CSI2_DT_RAW24;
+	default:
+		WARN_ON(1);
+		return 0;
+	}
+}
+
 static int ccs_read_frame_fmt(struct ccs_sensor *sensor)
 {
 	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
@@ -2633,6 +2661,34 @@ static int ccs_set_selection(struct v4l2_subdev *subdev,
 	return ret;
 }
 
+static int ccs_get_frame_desc(struct v4l2_subdev *subdev, unsigned int pad,
+				 struct v4l2_mbus_frame_desc *desc)
+{
+	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
+	struct v4l2_mbus_frame_desc_entry *entry = desc->entry;
+
+	switch (sensor->hwcfg.csi_signalling_mode) {
+	case CCS_CSI_SIGNALING_MODE_CSI_2_DPHY:
+	case CCS_CSI_SIGNALING_MODE_CSI_2_CPHY:
+		desc->type = V4L2_MBUS_FRAME_DESC_TYPE_CSI2;
+		break;
+	default:
+		/* FIXME: CCP2 support */
+		return -EINVAL;
+	}
+
+	entry->pixelcode = sensor->csi_format->code;
+	entry->stream = CCS_STREAM_PIXEL;
+	entry->bus.csi2.dt =
+		sensor->csi_format->width == sensor->csi_format->compressed ?
+		ccs_mipi_csi2_data_type(sensor->csi_format->width) :
+		CCS_DEFAULT_COMPRESSED_DT;
+	entry++;
+	desc->num_entries++;
+
+	return 0;
+}
+
 static int ccs_get_skip_frames(struct v4l2_subdev *subdev, u32 *frames)
 {
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
@@ -3055,6 +3111,7 @@ static const struct v4l2_subdev_pad_ops ccs_pad_ops = {
 	.set_selection = ccs_set_selection,
 	.enable_streams = ccs_enable_streams,
 	.disable_streams = ccs_disable_streams,
+	.get_frame_desc = ccs_get_frame_desc,
 };
 
 static const struct v4l2_subdev_sensor_ops ccs_sensor_ops = {
diff --git a/drivers/media/i2c/ccs/ccs.h b/drivers/media/i2c/ccs/ccs.h
index 4725e6eca8d0..90b442a3d53e 100644
--- a/drivers/media/i2c/ccs/ccs.h
+++ b/drivers/media/i2c/ccs/ccs.h
@@ -46,6 +46,8 @@
 
 #define CCS_COLOUR_COMPONENTS		4
 
+#define CCS_DEFAULT_COMPRESSED_DT	MIPI_CSI2_DT_USER_DEFINED(0)
+
 #define SMIAPP_NAME			"smiapp"
 #define CCS_NAME			"ccs"
 
@@ -175,6 +177,8 @@ struct ccs_csi_data_format {
 #define CCS_PAD_SRC			1
 #define CCS_PADS			2
 
+#define CCS_STREAM_PIXEL		0
+
 struct ccs_binning_subtype {
 	u8 horizontal:4;
 	u8 vertical:4;
-- 
2.39.2

[PATCH v9 25/46] media: uapi: v4l: subdev: Enable streams API

[Sakari Ailus]

Remove v4l2_subdev_enable_streams_api variable that was used to easily
enable streams API for development, and conditions that use the variable.

This patch enables the streams API for V4L2 sub-device interface which
allows transporting multiple streams on a single MC link.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 30 ---------------------------
 1 file changed, 30 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index e68f16a7c041..caf312e5e0d9 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -26,15 +26,6 @@
 #include <media/v4l2-fh.h>
 #include <media/v4l2-ioctl.h>
 
-#if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
-/*
- * The Streams API is an experimental feature. To use the Streams API, set
- * 'v4l2_subdev_enable_streams_api' to 1 below.
- */
-
-static bool v4l2_subdev_enable_streams_api;
-#endif
-
 /*
  * Maximum stream ID is 63 for now, as we use u64 bitmask to represent a set
  * of streams.
@@ -570,13 +561,6 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 				       V4L2_SUBDEV_CLIENT_CAP_STREAMS;
 	int rval;
 
-	/*
-	 * If the streams API is not enabled, remove V4L2_SUBDEV_CAP_STREAMS.
-	 * Remove this when the API is no longer experimental.
-	 */
-	if (!v4l2_subdev_enable_streams_api)
-		streams_subdev = false;
-
 	switch (cmd) {
 	case VIDIOC_SUBDEV_QUERYCAP: {
 		struct v4l2_subdev_capability *cap = arg;
@@ -918,9 +902,6 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 		struct v4l2_subdev_krouting krouting = {};
 		unsigned int i;
 
-		if (!v4l2_subdev_enable_streams_api)
-			return -ENOIOCTLCMD;
-
 		if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS))
 			return -ENOIOCTLCMD;
 
@@ -991,9 +972,6 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 		struct v4l2_subdev_routing *routing = arg;
 		struct v4l2_subdev_krouting *krouting;
 
-		if (!v4l2_subdev_enable_streams_api)
-			return -ENOIOCTLCMD;
-
 		if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS))
 			return -ENOIOCTLCMD;
 
@@ -1021,14 +999,6 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 	case VIDIOC_SUBDEV_S_CLIENT_CAP: {
 		struct v4l2_subdev_client_capability *client_cap = arg;
 
-		/*
-		 * Clear V4L2_SUBDEV_CLIENT_CAP_STREAMS if streams API is not
-		 * enabled. Remove this when streams API is no longer
-		 * experimental.
-		 */
-		if (!v4l2_subdev_enable_streams_api)
-			client_cap->capabilities &= ~V4L2_SUBDEV_CLIENT_CAP_STREAMS;
-
 		/* Filter out unsupported capabilities */
 		client_cap->capabilities &= (V4L2_SUBDEV_CLIENT_CAP_STREAMS |
 					     V4L2_SUBDEV_CLIENT_CAP_INTERVAL_USES_WHICH);
-- 
2.39.2

[PATCH v9 26/46] media: mc: Add INTERNAL pad flag

[Sakari Ailus]

Internal sink pads will be used as routing endpoints in V4L2 [GS]_ROUTING
IOCTLs, to indicate that the stream begins in the entity. Internal sink
pads are pads that have both SINK and INTERNAL flags set.

Also prevent creating links to pads that have been flagged as internal and
initialising SOURCE pads with INTERNAL flag set.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
---
 .../userspace-api/media/mediactl/media-types.rst  |  9 +++++++++
 drivers/media/mc/mc-entity.c                      | 15 ++++++++++++---
 include/uapi/linux/media.h                        |  1 +
 3 files changed, 22 insertions(+), 3 deletions(-)

diff --git a/Documentation/userspace-api/media/mediactl/media-types.rst b/Documentation/userspace-api/media/mediactl/media-types.rst
index 6332e8395263..200c37a1da26 100644
--- a/Documentation/userspace-api/media/mediactl/media-types.rst
+++ b/Documentation/userspace-api/media/mediactl/media-types.rst
@@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
 .. _MEDIA-PAD-FL-SINK:
 .. _MEDIA-PAD-FL-SOURCE:
 .. _MEDIA-PAD-FL-MUST-CONNECT:
+.. _MEDIA-PAD-FL-INTERNAL:
 
 .. flat-table:: Media pad flags
     :header-rows:  0
@@ -381,6 +382,14 @@ Types and flags used to represent the media graph elements
 	  enabled links even when this flag isn't set; the absence of the flag
 	  doesn't imply there is none.
 
+    *  -  ``MEDIA_PAD_FL_INTERNAL``
+       -  The internal flag indicates an internal pad that has no external
+	  connections. As they are internal to entities, internal pads shall not
+	  be connected with links.
+
+	  The internal flag may currently be present only in a sink pad where it
+	  indicates that the :ref:``stream <media-glossary-stream>`` originates
+	  from within the entity.
 
 One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
 must be set for every pad.
diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 0e28b9a7936e..7747aa16f37e 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -209,11 +209,16 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 		mutex_lock(&mdev->graph_mutex);
 
 	media_entity_for_each_pad(entity, iter) {
+		const u32 pad_flags = iter->flags & (MEDIA_PAD_FL_SINK |
+						     MEDIA_PAD_FL_SOURCE |
+						     MEDIA_PAD_FL_INTERNAL);
+
 		iter->entity = entity;
 		iter->index = i++;
 
-		if (hweight32(iter->flags & (MEDIA_PAD_FL_SINK |
-					     MEDIA_PAD_FL_SOURCE)) != 1) {
+		if (pad_flags != MEDIA_PAD_FL_SINK &&
+		    pad_flags != (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL) &&
+		    pad_flags != MEDIA_PAD_FL_SOURCE) {
 			ret = -EINVAL;
 			break;
 		}
@@ -1112,7 +1117,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
 
 	for (i = 0; i < entity->num_pads; i++) {
 		if ((entity->pads[i].flags &
-		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
+		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
+		      MEDIA_PAD_FL_INTERNAL)) != pad_type)
 			continue;
 
 		if (entity->pads[i].sig_type == sig_type)
@@ -1142,6 +1148,9 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
 		return -EINVAL;
 	if (WARN_ON(!(sink->pads[sink_pad].flags & MEDIA_PAD_FL_SINK)))
 		return -EINVAL;
+	if (WARN_ON(source->pads[source_pad].flags & MEDIA_PAD_FL_INTERNAL) ||
+	    WARN_ON(sink->pads[sink_pad].flags & MEDIA_PAD_FL_INTERNAL))
+		return -EINVAL;
 
 	link = media_add_link(&source->links);
 	if (link == NULL)
diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
index 1c80b1d6bbaf..80cfd12a43fc 100644
--- a/include/uapi/linux/media.h
+++ b/include/uapi/linux/media.h
@@ -208,6 +208,7 @@ struct media_entity_desc {
 #define MEDIA_PAD_FL_SINK			(1U << 0)
 #define MEDIA_PAD_FL_SOURCE			(1U << 1)
 #define MEDIA_PAD_FL_MUST_CONNECT		(1U << 2)
+#define MEDIA_PAD_FL_INTERNAL			(1U << 3)
 
 struct media_pad_desc {
 	__u32 entity;		/* entity ID */
-- 
2.39.2

[PATCH v9 27/46] media: uapi: ccs: Add media bus code for MIPI CCS embedded data

[Sakari Ailus]

Add new MIPI CCS embedded data media bus code
(MEDIA_BUS_FMT_CCS_EMBEDDED).

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Julien Massot <julien.massot@collabora.com>
---
 .../media/v4l/subdev-formats.rst              | 32 +++++++++++++++++++
 include/uapi/linux/media-bus-format.h         |  3 ++
 2 files changed, 35 insertions(+)

diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
index 0547f2733ee3..fa181ce8f48c 100644
--- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
+++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
@@ -8316,6 +8316,10 @@ content is more or less device specific but the data is transmitted and received
 by multiple devices that do not process the data in any way, simply writing
 it to system memory for processing in software at the end of the pipeline.
 
+The exact format of the data generated by the device is reported on the internal
+source pad of the originating sub-device, using one of the more specific
+metadata formats such as MEDIA_BUS_FMT_CCS_EMBEDDED.
+
 "b" in an array cell signifies a byte of data, followed by the number of the bit
 and finally the bit number in subscript. "x" indicates a padding bit.
 
@@ -8562,3 +8566,31 @@ and finally the bit number in subscript. "x" indicates a padding bit.
       - x
       - x
       - x
+
+.. _MEDIA-BUS-FMT-CCS-EMBEDDED:
+
+MIPI CCS Embedded Data Formats
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+`MIPI CCS <https://www.mipi.org/specifications/camera-command-set>`_ defines a
+metadata format for sensor embedded data, which is used to store the register
+configuration used for capturing a given frame. The format is defined in the CCS
+specification. The media bus code for this format is
+``MEDIA_BUS_FMT_CCS_EMBEDDED``.
+
+The CCS embedded data format definition includes three levels:
+
+1. Padding within CSI-2 bus :ref:`Data Unit <media-glossary-data-unit>` as
+   documented in the MIPI CCS specification.
+
+2. The tagged data format as documented in the MIPI CCS specification.
+
+3. Register addresses and register documentation as documented in the MIPI CCS
+   specification.
+
+The format definition shall be used only by devices that fulfill all three
+levels above.
+
+This mbus code are only used for "2-byte simplified tagged data format" (code
+0xa) but their use may be extended further in the future, to cover other CCS
+embedded data format codes.
diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
index d4c1d991014b..03f7e9ab517b 100644
--- a/include/uapi/linux/media-bus-format.h
+++ b/include/uapi/linux/media-bus-format.h
@@ -183,4 +183,7 @@
 #define MEDIA_BUS_FMT_META_20			0x8006
 #define MEDIA_BUS_FMT_META_24			0x8007
 
+/* Specific metadata formats. Next is 0x9002. */
+#define MEDIA_BUS_FMT_CCS_EMBEDDED		0x9001
+
 #endif /* __LINUX_MEDIA_BUS_FORMAT_H */
-- 
2.39.2

[PATCH v9 28/46] media: Documentation: Document non-CCS use of CCS embedded data format

[Sakari Ailus]

The CCS embedded data format has multiple aspects (packing, encoding and
the rest, including register addresses and semantics). Explicitly allow
non-compliant embedded data to use the two former to reduce redundant
documentation.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../userspace-api/media/drivers/camera-sensor.rst     | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/Documentation/userspace-api/media/drivers/camera-sensor.rst b/Documentation/userspace-api/media/drivers/camera-sensor.rst
index 9f3b0da3ad0d..dc415b8f6c8e 100644
--- a/Documentation/userspace-api/media/drivers/camera-sensor.rst
+++ b/Documentation/userspace-api/media/drivers/camera-sensor.rst
@@ -123,3 +123,14 @@ In general, changing the embedded data format from the driver-configured values
 is not supported. The height of the metadata is device-specific and the width
 is that (or less of that) of the image width, as configured on the pixel data
 stream.
+
+CCS and non-CCS embedded data
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Embedded data which is fully compliant with CCS definitions uses ``CCS embedded
+data format <MEDIA-BUS-FMT-CCS-EMBEDDED>`` media bus code (level
+3). Device-specific embedded data compliant with either MIPI CCS embedded data
+levels 1 or 2 only shall not use CCS embedded data mbus code, but may refer to
+CCS embedded data documentation with the level of conformance specified and omit
+documenting these aspects of the format. The rest of the device-specific
+embedded data format is documented in the context of the data format itself.
-- 
2.39.2

[PATCH v9 29/46] media: Documentation: ccs: Document routing

[Sakari Ailus]

Document which routes are available for the CCS driver (source) sub-device
and what configuration are possible.

Also update copyright.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../userspace-api/media/drivers/ccs.rst       | 38 ++++++++++++++++++-
 .../media/v4l/subdev-formats.rst              |  2 +
 2 files changed, 39 insertions(+), 1 deletion(-)

diff --git a/Documentation/userspace-api/media/drivers/ccs.rst b/Documentation/userspace-api/media/drivers/ccs.rst
index 03015b33d5ab..53890ac54dab 100644
--- a/Documentation/userspace-api/media/drivers/ccs.rst
+++ b/Documentation/userspace-api/media/drivers/ccs.rst
@@ -111,4 +111,40 @@ than in the centre.
 Shading correction needs to be enabled for luminance correction level to have an
 effect.
 
-**Copyright** |copy| 2020 Intel Corporation
+.. _media-ccs-routes:
+
+Routes
+------
+
+The CCS driver implements one or two :ref:`routes <subdev-routing>` in
+its source sub-device (scaler sub-device if exists for the device, otherwise
+binner) depending on whether the sensor supports embedded data. (All CCS
+compliant sensors do but the CCS driver supports preceding standards that did
+not require embedded data support, too.)
+
+The first route of the CCS source sub-device is for pixel data (sink pad
+0/stream 0 -> source pad 1/stream 0) and the second one is for embedded data
+(internal sink pad 2/stream 0 -> source pad 1/stream 1).
+
+Embedded data
+~~~~~~~~~~~~~
+
+MIPI CCS supports generation of camera sensor embedded data. The media bus code
+used for this format on the internal sink pad is
+:ref:`MEDIA_BUS_FMT_CCS_EMBEDDDED <MEDIA-BUS-FMT-CCS-EMBEDDED>`.
+
+The bit depth of the CCS pixel data affects how the sensor will output the
+embedded data, adding padding to align with CSI-2 bus :ref:`Data Units
+<media-glossary-data-unit>` for that particular bit depth. This is indicated by
+the generic metadata format on the sensor's source sub-device's source pad.
+
+Devices supporting embedded data for bit depths greater than or equal to 16 may
+support more dense packing or legacy single metadata byte per data unit, or both
+of these. The supported embedded data formats can be enumerated and configured
+on stream 1 of the source pad (1) of the CCS source sub-device.
+
+The use of the denser packing results in embedded data lines being longer than
+the pixel data in data units since the data units are smaller. In bytes the
+embedded data lines are still not longer than the image data lines.
+
+**Copyright** |copy| 2020, 2023 Intel Corporation
diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
index fa181ce8f48c..a04756092238 100644
--- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
+++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
@@ -8594,3 +8594,5 @@ levels above.
 This mbus code are only used for "2-byte simplified tagged data format" (code
 0xa) but their use may be extended further in the future, to cover other CCS
 embedded data format codes.
+
+Also see :ref:`CCS driver documentation <media-ccs-routes>`.
-- 
2.39.2

[PATCH v9 30/46] media: ccs: Add support for embedded data stream

[Sakari Ailus]

Add support for embedded data stream, in UAPI and frame descriptor.

This patch adds also a new embedded data pad (2) to the source sub-device.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ccs/ccs-core.c | 372 +++++++++++++++++++++++++++++--
 drivers/media/i2c/ccs/ccs.h      |  18 +-
 2 files changed, 362 insertions(+), 28 deletions(-)

diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
index 3ca2415fca3b..ba629eafec43 100644
--- a/drivers/media/i2c/ccs/ccs-core.c
+++ b/drivers/media/i2c/ccs/ccs-core.c
@@ -1903,6 +1903,13 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
 	if (rval < 0)
 		goto err_pm_put;
 
+	/* Configure embedded data */
+	if (sensor->csi_format->compressed >= 16) {
+		rval = ccs_write(sensor, EMB_DATA_CTRL, sensor->emb_data_ctrl);
+		if (rval < 0)
+			goto err_pm_put;
+	}
+
 	if (CCS_LIM(sensor, FLASH_MODE_CAPABILITY) &
 	    (CCS_FLASH_MODE_CAPABILITY_SINGLE_STROBE |
 	     SMIAPP_FLASH_MODE_CAPABILITY_MULTIPLE_STROBE) &&
@@ -2022,6 +2029,57 @@ static const struct ccs_csi_data_format
 	return sensor->csi_format;
 }
 
+#define CCS_EMBEDDED_CODE_DEPTH(depth, half_depth)			\
+	depth,								\
+	CCS_EMB_DATA_CAPABILITY_TWO_BYTES_PER_RAW##depth,		\
+	CCS_EMB_DATA_CAPABILITY_NO_ONE_BYTE_PER_RAW##depth,		\
+	CCS_EMB_DATA_CTRL_RAW##half_depth##_PACKING_FOR_RAW##depth,	\
+	MEDIA_BUS_FMT_META_##half_depth,				\
+	MEDIA_BUS_FMT_META_##depth,					\
+
+static const struct ccs_embedded_code {
+	u8 depth;
+	u8 cap_two_bytes_per_sample;
+	u8 cap_no_legacy;
+	u8 ctrl;
+	u32 code_two_bytes;
+	u32 code_legacy;
+} ccs_embedded_codes[] = {
+	{ CCS_EMBEDDED_CODE_DEPTH(16, 8) },
+	{ CCS_EMBEDDED_CODE_DEPTH(20, 10) },
+	{ CCS_EMBEDDED_CODE_DEPTH(24, 12) },
+};
+
+static const struct ccs_embedded_code *ccs_embedded_code(unsigned int bpp)
+{
+	unsigned int i;
+
+	for (i = 0; i < ARRAY_SIZE(ccs_embedded_codes); i++)
+		if (ccs_embedded_codes[i].depth == bpp)
+			return ccs_embedded_codes + i;
+
+	WARN_ON(1);
+
+	return ccs_embedded_codes;
+}
+
+static u32
+ccs_default_embedded_code(struct ccs_sensor *sensor,
+			  const struct ccs_embedded_code *embedded_code)
+{
+	if (CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
+	    BIT(embedded_code->cap_two_bytes_per_sample))
+		return embedded_code->code_two_bytes;
+
+	if (!(CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
+	      BIT(embedded_code->cap_no_legacy)))
+		return embedded_code->code_legacy;
+
+	WARN_ON(1);
+
+	return embedded_code->code_legacy;
+}
+
 static int ccs_enum_mbus_code(struct v4l2_subdev *subdev,
 			      struct v4l2_subdev_state *sd_state,
 			      struct v4l2_subdev_mbus_code_enum *code)
@@ -2037,6 +2095,69 @@ static int ccs_enum_mbus_code(struct v4l2_subdev *subdev,
 	dev_err(&client->dev, "subdev %s, pad %u, index %u\n",
 		subdev->name, code->pad, code->index);
 
+	if (subdev == &sensor->src->sd) {
+		if (code->pad == CCS_PAD_META) {
+			if (code->index)
+				goto out;
+
+			code->code = MEDIA_BUS_FMT_CCS_EMBEDDED;
+
+			rval = 0;
+			goto out;
+		}
+		if (code->stream == CCS_STREAM_META) {
+			struct v4l2_mbus_framefmt *pix_fmt =
+				v4l2_subdev_state_get_format(sd_state,
+							     CCS_PAD_SRC,
+							     CCS_STREAM_PIXEL);
+			const struct ccs_csi_data_format *csi_format =
+				ccs_validate_csi_data_format(sensor,
+							     pix_fmt->code);
+			unsigned int i = 0;
+			u32 codes[2];
+
+			switch (csi_format->compressed) {
+			case 8:
+				codes[i++] = MEDIA_BUS_FMT_META_8;
+				break;
+			case 10:
+				codes[i++] = MEDIA_BUS_FMT_META_10;
+				break;
+			case 12:
+				codes[i++] = MEDIA_BUS_FMT_META_12;
+				break;
+			case 14:
+				codes[i++] = MEDIA_BUS_FMT_META_14;
+				break;
+			case 16:
+			case 20:
+			case 24: {
+				const struct ccs_embedded_code *embedded_code =
+					ccs_embedded_code(csi_format->compressed);
+
+				if (CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
+				    BIT(embedded_code->cap_two_bytes_per_sample))
+					codes[i++] =
+						embedded_code->code_two_bytes;
+
+				if (!(CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
+				      BIT(embedded_code->cap_no_legacy)))
+					codes[i++] = embedded_code->code_legacy;
+				break;
+			}
+			default:
+				WARN_ON(1);
+			}
+
+			if (WARN_ON(i > ARRAY_SIZE(codes)) || code->index >= i)
+				goto out;
+
+			code->code = codes[code->index];
+			rval = 0;
+			goto out;
+		}
+	}
+
 	if (subdev != &sensor->src->sd || code->pad != CCS_PAD_SRC) {
 		if (code->index)
 			goto out;
@@ -2079,8 +2200,11 @@ static int __ccs_get_format(struct v4l2_subdev *subdev,
 			    struct v4l2_subdev_state *sd_state,
 			    struct v4l2_subdev_format *fmt)
 {
-	fmt->format = *v4l2_subdev_state_get_format(sd_state, fmt->pad);
-	fmt->format.code = __ccs_get_mbus_code(subdev, fmt->pad);
+	fmt->format = *v4l2_subdev_state_get_format(sd_state, fmt->pad,
+						    fmt->stream);
+
+	if (fmt->pad != CCS_PAD_META && fmt->stream != CCS_STREAM_META)
+		fmt->format.code = __ccs_get_mbus_code(subdev, fmt->pad);
 
 	return 0;
 }
@@ -2110,10 +2234,11 @@ static void ccs_get_crop_compose(struct v4l2_subdev *subdev,
 	if (crops)
 		for (i = 0; i < subdev->entity.num_pads; i++)
 			crops[i] =
-				v4l2_subdev_state_get_crop(sd_state, i);
+				v4l2_subdev_state_get_crop(sd_state, i,
+							   CCS_STREAM_PIXEL);
 	if (comps)
-		*comps = v4l2_subdev_state_get_compose(sd_state,
-						       ssd->sink_pad);
+		*comps = v4l2_subdev_state_get_compose(sd_state, ssd->sink_pad,
+						       CCS_STREAM_PIXEL);
 }
 
 /* Changes require propagation only on sink pad. */
@@ -2146,7 +2271,8 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
 		fallthrough;
 	case V4L2_SEL_TGT_COMPOSE:
 		*crops[CCS_PAD_SRC] = *comp;
-		fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC);
+		fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC,
+						   CCS_STREAM_PIXEL);
 		fmt->width = comp->width;
 		fmt->height = comp->height;
 		if (which == V4L2_SUBDEV_FORMAT_ACTIVE && ssd == sensor->src)
@@ -2210,6 +2336,83 @@ static int ccs_set_format_source(struct v4l2_subdev *subdev,
 	return ccs_pll_update(sensor);
 }
 
+static int ccs_set_format_meta(struct v4l2_subdev *subdev,
+			       struct v4l2_subdev_state *sd_state,
+			       struct v4l2_mbus_framefmt *fmt)
+{
+	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
+	const struct ccs_csi_data_format *csi_format;
+	struct v4l2_mbus_framefmt *pix_fmt;
+	struct v4l2_mbus_framefmt *meta_fmt;
+	struct v4l2_mbus_framefmt *meta_out_fmt;
+	u32 code;
+
+	pix_fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC,
+					       CCS_STREAM_PIXEL);
+	meta_fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_META, 0);
+	meta_out_fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC,
+						    CCS_STREAM_META);
+
+	code = fmt ? fmt->code : meta_out_fmt->code;
+
+	meta_out_fmt->width = meta_fmt->width = pix_fmt->width;
+	meta_out_fmt->height = meta_fmt->height =
+		sensor->embedded_end - sensor->embedded_start;
+	meta_fmt->code = MEDIA_BUS_FMT_CCS_EMBEDDED;
+
+	csi_format = ccs_validate_csi_data_format(sensor, pix_fmt->code);
+
+	switch (csi_format->compressed) {
+	case 8:
+		meta_out_fmt->code = MEDIA_BUS_FMT_META_8;
+		break;
+	case 10:
+		meta_out_fmt->code = MEDIA_BUS_FMT_META_10;
+		break;
+	case 12:
+		meta_out_fmt->code = MEDIA_BUS_FMT_META_12;
+		break;
+	case 14:
+		meta_out_fmt->code = MEDIA_BUS_FMT_META_14;
+		break;
+	case 16:
+	case 20:
+	case 24: {
+		const struct ccs_embedded_code *embedded_code;
+
+		embedded_code = ccs_embedded_code(csi_format->compressed);
+		meta_out_fmt->code =
+			ccs_default_embedded_code(sensor, embedded_code);
+
+		if (!(CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
+		      BIT(embedded_code->cap_no_legacy)) &&
+		    code == embedded_code->code_legacy) {
+			meta_out_fmt->code = embedded_code->code_legacy;
+			sensor->emb_data_ctrl = 0;
+		}
+
+		if (CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
+		    BIT(embedded_code->cap_two_bytes_per_sample) &&
+		    code == embedded_code->code_two_bytes) {
+			meta_out_fmt->code = embedded_code->code_two_bytes;
+			sensor->emb_data_ctrl = embedded_code->ctrl;
+			meta_fmt->width <<= 1;
+			meta_out_fmt->width <<= 1;
+		}
+
+		break;
+	}
+	default:
+		WARN_ON(1);
+		return 0;
+	}
+
+	if (fmt)
+		*fmt = *meta_out_fmt;
+
+	return 0;
+}
+
 static int ccs_set_format(struct v4l2_subdev *subdev,
 			  struct v4l2_subdev_state *sd_state,
 			  struct v4l2_subdev_format *fmt)
@@ -2218,12 +2421,25 @@ static int ccs_set_format(struct v4l2_subdev *subdev,
 	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
 	struct v4l2_rect *crops[CCS_PADS];
 
+	if (subdev == &sensor->src->sd && fmt->pad == CCS_PAD_META)
+		return ccs_get_format(subdev, sd_state, fmt);
+
 	mutex_lock(&sensor->mutex);
 
+	if (subdev == &sensor->src->sd && fmt->stream == CCS_STREAM_META) {
+		ccs_set_format_meta(subdev, sd_state, &fmt->format);
+
+		mutex_unlock(&sensor->mutex);
+
+		return 0;
+	}
+
 	if (fmt->pad == ssd->source_pad) {
 		int rval;
 
 		rval = ccs_set_format_source(subdev, sd_state, fmt);
+		if (sensor->embedded_start != sensor->embedded_end)
+			ccs_set_format_meta(subdev, sd_state, NULL);
 
 		mutex_unlock(&sensor->mutex);
 
@@ -2498,6 +2714,12 @@ static int ccs_sel_supported(struct v4l2_subdev *subdev,
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
 	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
 
+	if (sel->stream != CCS_STREAM_PIXEL)
+		return -EINVAL;
+
+	if (sel->pad == CCS_PAD_META)
+		return -EINVAL;
+
 	/* We only implement crop in three places. */
 	switch (sel->target) {
 	case V4L2_SEL_TGT_CROP:
@@ -2542,7 +2764,8 @@ static int ccs_set_crop(struct v4l2_subdev *subdev,
 
 	if (sel->pad == ssd->sink_pad) {
 		struct v4l2_mbus_framefmt *mfmt =
-			v4l2_subdev_state_get_format(sd_state, sel->pad);
+			v4l2_subdev_state_get_format(sd_state, sel->pad,
+						     CCS_STREAM_PIXEL);
 
 		src_size.width = mfmt->width;
 		src_size.height = mfmt->height;
@@ -2603,7 +2826,9 @@ static int ccs_get_selection(struct v4l2_subdev *subdev,
 		} else if (sel->pad == ssd->sink_pad) {
 			struct v4l2_mbus_framefmt *sink_fmt =
 				v4l2_subdev_state_get_format(sd_state,
-							     ssd->sink_pad);
+							     ssd->sink_pad,
+							     CCS_STREAM_PIXEL);
+
 			sel->r.top = sel->r.left = 0;
 			sel->r.width = sink_fmt->width;
 			sel->r.height = sink_fmt->height;
@@ -2686,6 +2911,14 @@ static int ccs_get_frame_desc(struct v4l2_subdev *subdev, unsigned int pad,
 	entry++;
 	desc->num_entries++;
 
+	if (sensor->embedded_start != sensor->embedded_end) {
+		entry->pixelcode = MEDIA_BUS_FMT_CCS_EMBEDDED;
+		entry->stream = CCS_STREAM_META;
+		entry->bus.csi2.dt = MIPI_CSI2_DT_EMBEDDED_8B;
+		entry++;
+		desc->num_entries++;
+	}
+
 	return 0;
 }
 
@@ -3004,6 +3237,8 @@ static void ccs_cleanup(struct ccs_sensor *sensor)
 	ccs_free_controls(sensor);
 }
 
+static const struct v4l2_subdev_internal_ops ccs_internal_ops;
+
 static int ccs_init_subdev(struct ccs_sensor *sensor,
 			   struct ccs_subdev *ssd, const char *name,
 			   unsigned short num_pads, u32 function,
@@ -3016,15 +3251,18 @@ static int ccs_init_subdev(struct ccs_sensor *sensor,
 	if (!ssd)
 		return 0;
 
-	if (ssd != sensor->src)
+	if (ssd != sensor->src) {
 		v4l2_subdev_init(&ssd->sd, &ccs_ops);
+		ssd->sd.internal_ops = &ccs_internal_ops;
+	}
 
 	ssd->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
 	ssd->sd.entity.function = function;
 	ssd->sensor = sensor;
 
 	ssd->npads = num_pads;
-	ssd->source_pad = num_pads - 1;
+	ssd->source_pad =
+		ssd == sensor->pixel_array ? CCS_PA_PAD_SRC : CCS_PAD_SRC;
 
 	v4l2_i2c_subdev_set_name(&ssd->sd, client, sensor->minfo.name, name);
 
@@ -3038,6 +3276,10 @@ static int ccs_init_subdev(struct ccs_sensor *sensor,
 		ssd->sd.owner = THIS_MODULE;
 		ssd->sd.dev = &client->dev;
 		v4l2_set_subdevdata(&ssd->sd, client);
+	} else {
+		ssd->sd.flags |= V4L2_SUBDEV_FL_STREAMS;
+		ssd->pads[CCS_PAD_META].flags =
+			MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL;
 	}
 
 	rval = media_entity_pads_init(&ssd->sd.entity, ssd->npads, ssd->pads);
@@ -3055,21 +3297,19 @@ static int ccs_init_subdev(struct ccs_sensor *sensor,
 	return 0;
 }
 
-static int ccs_init_state(struct v4l2_subdev *sd,
-			  struct v4l2_subdev_state *sd_state)
+static int __ccs_init_state(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_state *sd_state)
 {
 	struct ccs_subdev *ssd = to_ccs_subdev(sd);
 	struct ccs_sensor *sensor = ssd->sensor;
 	unsigned int pad = ssd == sensor->pixel_array ?
 		CCS_PA_PAD_SRC : CCS_PAD_SINK;
 	struct v4l2_mbus_framefmt *fmt =
-		v4l2_subdev_state_get_format(sd_state, pad);
+		v4l2_subdev_state_get_format(sd_state, pad, CCS_STREAM_PIXEL);
 	struct v4l2_rect *crop =
-		v4l2_subdev_state_get_crop(sd_state, pad);
+		v4l2_subdev_state_get_crop(sd_state, pad, CCS_STREAM_PIXEL);
 	bool is_active = !sd->active_state || sd->active_state == sd_state;
 
-	mutex_lock(&sensor->mutex);
-
 	ccs_get_native_size(ssd, crop);
 
 	fmt->width = crop->width;
@@ -3081,20 +3321,78 @@ static int ccs_init_state(struct v4l2_subdev *sd,
 		if (is_active)
 			sensor->pa_src = *crop;
 
-		mutex_unlock(&sensor->mutex);
 		return 0;
 	}
 
-	fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC);
+	fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC,
+					   CCS_STREAM_PIXEL);
 	fmt->code = ssd == sensor->src ?
 		sensor->csi_format->code : sensor->internal_csi_format->code;
 	fmt->field = V4L2_FIELD_NONE;
 
 	ccs_propagate(sd, sd_state, is_active, V4L2_SEL_TGT_CROP);
 
+	return 0;
+}
+
+static int ccs_init_state(struct v4l2_subdev *sd,
+			  struct v4l2_subdev_state *sd_state)
+{
+	struct ccs_subdev *ssd = to_ccs_subdev(sd);
+	struct ccs_sensor *sensor = ssd->sensor;
+	int rval;
+
+	mutex_lock(&sensor->mutex);
+	rval = __ccs_init_state(sd, sd_state);
 	mutex_unlock(&sensor->mutex);
 
-	return 0;
+	return rval;
+}
+
+static int ccs_src_init_state(struct v4l2_subdev *sd,
+			      struct v4l2_subdev_state *sd_state)
+{
+	struct v4l2_subdev_route routes[] = {
+		{
+			.sink_pad = CCS_PAD_SINK,
+			.source_pad = CCS_PAD_SRC,
+			.source_stream = CCS_STREAM_PIXEL,
+			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
+		}, {
+			.sink_pad = CCS_PAD_META,
+			.source_pad = CCS_PAD_SRC,
+			.source_stream = CCS_STREAM_META,
+			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
+		}
+	};
+	struct v4l2_subdev_krouting routing = {
+		.routes = routes,
+		.num_routes = 1,
+	};
+	struct ccs_subdev *ssd = to_ccs_subdev(sd);
+	struct ccs_sensor *sensor = ssd->sensor;
+	int rval;
+
+	mutex_lock(&sensor->mutex);
+
+	if (sensor->embedded_start != sensor->embedded_end)
+		routing.num_routes++;
+
+	rval = v4l2_subdev_set_routing(sd, sd_state, &routing);
+	if (rval)
+		goto out;
+
+	rval = __ccs_init_state(sd, sd_state);
+	if (rval)
+		goto out;
+
+	if (sensor->embedded_start != sensor->embedded_end)
+		ccs_set_format_meta(sd, sd_state, NULL);
+
+out:
+	mutex_unlock(&sensor->mutex);
+
+	return rval;
 }
 
 static const struct v4l2_subdev_video_ops ccs_video_ops = {
@@ -3109,6 +3407,14 @@ static const struct v4l2_subdev_pad_ops ccs_pad_ops = {
 	.set_fmt = ccs_set_format,
 	.get_selection = ccs_get_selection,
 	.set_selection = ccs_set_selection,
+};
+
+static const struct v4l2_subdev_pad_ops ccs_src_pad_ops = {
+	.enum_mbus_code = ccs_enum_mbus_code,
+	.get_fmt = ccs_get_format,
+	.set_fmt = ccs_set_format,
+	.get_selection = ccs_get_selection,
+	.set_selection = ccs_set_selection,
 	.enable_streams = ccs_enable_streams,
 	.disable_streams = ccs_disable_streams,
 	.get_frame_desc = ccs_get_frame_desc,
@@ -3125,12 +3431,22 @@ static const struct v4l2_subdev_ops ccs_ops = {
 	.sensor = &ccs_sensor_ops,
 };
 
+static const struct v4l2_subdev_ops ccs_src_ops = {
+	.video = &ccs_video_ops,
+	.pad = &ccs_src_pad_ops,
+	.sensor = &ccs_sensor_ops,
+};
+
 static const struct media_entity_operations ccs_entity_ops = {
 	.link_validate = v4l2_subdev_link_validate,
 };
 
-static const struct v4l2_subdev_internal_ops ccs_internal_src_ops = {
+static const struct v4l2_subdev_internal_ops ccs_internal_ops = {
 	.init_state = ccs_init_state,
+};
+
+static const struct v4l2_subdev_internal_ops ccs_src_internal_ops = {
+	.init_state = ccs_src_init_state,
 	.registered = ccs_registered,
 	.unregistered = ccs_unregistered,
 };
@@ -3281,8 +3597,8 @@ static int ccs_probe(struct i2c_client *client)
 
 	sensor->src = &sensor->ssds[sensor->ssds_used];
 
-	v4l2_i2c_subdev_init(&sensor->src->sd, client, &ccs_ops);
-	sensor->src->sd.internal_ops = &ccs_internal_src_ops;
+	v4l2_i2c_subdev_init(&sensor->src->sd, client, &ccs_src_ops);
+	sensor->src->sd.internal_ops = &ccs_src_internal_ops;
 
 	sensor->regulators = devm_kcalloc(&client->dev,
 					  ARRAY_SIZE(ccs_regulators),
@@ -3553,12 +3869,20 @@ static int ccs_probe(struct i2c_client *client)
 		goto out_cleanup;
 	}
 
-	rval = ccs_init_subdev(sensor, sensor->scaler, " scaler", 2,
+	rval = ccs_init_subdev(sensor, sensor->scaler, " scaler",
+			       sensor->ssds_used != CCS_SUBDEVS ?
+			       CCS_PADS_NOMETA :
+			       sensor->embedded_start == sensor->embedded_end ?
+			       CCS_PADS_NOMETA : CCS_PADS,
 			       MEDIA_ENT_F_PROC_VIDEO_SCALER,
 			       "ccs scaler mutex", &scaler_lock_key);
 	if (rval)
 		goto out_cleanup;
-	rval = ccs_init_subdev(sensor, sensor->binner, " binner", 2,
+	rval = ccs_init_subdev(sensor, sensor->binner, " binner",
+			       sensor->ssds_used == CCS_SUBDEVS ?
+			       CCS_PADS_NOMETA :
+			       sensor->embedded_start == sensor->embedded_end ?
+			       CCS_PADS_NOMETA : CCS_PADS,
 			       MEDIA_ENT_F_PROC_VIDEO_SCALER,
 			       "ccs binner mutex", &binner_lock_key);
 	if (rval)
diff --git a/drivers/media/i2c/ccs/ccs.h b/drivers/media/i2c/ccs/ccs.h
index 90b442a3d53e..477b2fb99aa0 100644
--- a/drivers/media/i2c/ccs/ccs.h
+++ b/drivers/media/i2c/ccs/ccs.h
@@ -173,11 +173,18 @@ struct ccs_csi_data_format {
 #define CCS_SUBDEVS			3
 
 #define CCS_PA_PAD_SRC			0
-#define CCS_PAD_SINK			0
-#define CCS_PAD_SRC			1
-#define CCS_PADS			2
+enum {
+	CCS_PAD_SINK,
+	CCS_PAD_SRC,
+	CCS_PAD_META,
+	CCS_PADS_NOMETA = CCS_PAD_META,
+	CCS_PADS,
+};
 
-#define CCS_STREAM_PIXEL		0
+enum {
+	CCS_STREAM_PIXEL,
+	CCS_STREAM_META,
+};
 
 struct ccs_binning_subtype {
 	u8 horizontal:4;
@@ -228,6 +235,9 @@ struct ccs_sensor {
 	int default_pixel_order;
 	struct ccs_data_container sdata, mdata;
 
+	u32 embedded_mbus_code;
+	u8 emb_data_ctrl;
+
 	u8 binning_horizontal;
 	u8 binning_vertical;
 
-- 
2.39.2

[PATCH v9 31/46] media: ccs: Remove ccs_get_crop_compose helper

[Sakari Ailus]

As it's now easier to obtain the necessary information on crop and compose
rectangles after moving to sub-device state, remove the
ccs_get_crop_compose helper.

Also remove the comp arguments of the compose goodness calculators and
make related local variables and function arguments const where
applicable.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ccs/ccs-core.c | 164 ++++++++++++++-----------------
 1 file changed, 72 insertions(+), 92 deletions(-)

diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
index ba629eafec43..299b5ac6538c 100644
--- a/drivers/media/i2c/ccs/ccs-core.c
+++ b/drivers/media/i2c/ccs/ccs-core.c
@@ -2223,24 +2223,6 @@ static int ccs_get_format(struct v4l2_subdev *subdev,
 	return rval;
 }
 
-static void ccs_get_crop_compose(struct v4l2_subdev *subdev,
-				 struct v4l2_subdev_state *sd_state,
-				 struct v4l2_rect **crops,
-				 struct v4l2_rect **comps)
-{
-	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
-	unsigned int i;
-
-	if (crops)
-		for (i = 0; i < subdev->entity.num_pads; i++)
-			crops[i] =
-				v4l2_subdev_state_get_crop(sd_state, i,
-							   CCS_STREAM_PIXEL);
-	if (comps)
-		*comps = v4l2_subdev_state_get_compose(sd_state, ssd->sink_pad,
-						       CCS_STREAM_PIXEL);
-}
-
 /* Changes require propagation only on sink pad. */
 static void ccs_propagate(struct v4l2_subdev *subdev,
 			  struct v4l2_subdev_state *sd_state, int which,
@@ -2248,15 +2230,17 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
 {
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
 	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
-	struct v4l2_rect *comp, *crops[CCS_PADS];
+	struct v4l2_rect *comp, *crop;
 	struct v4l2_mbus_framefmt *fmt;
 
-	ccs_get_crop_compose(subdev, sd_state, crops, &comp);
-
+	comp = v4l2_subdev_state_get_compose(sd_state, ssd->sink_pad,
+					     CCS_STREAM_PIXEL);
 	switch (target) {
 	case V4L2_SEL_TGT_CROP:
-		comp->width = crops[CCS_PAD_SINK]->width;
-		comp->height = crops[CCS_PAD_SINK]->height;
+		crop = v4l2_subdev_state_get_crop(sd_state, CCS_PAD_SINK,
+						  CCS_STREAM_PIXEL);
+		comp->width = crop->width;
+		comp->height = crop->height;
 		if (which == V4L2_SUBDEV_FORMAT_ACTIVE) {
 			if (ssd == sensor->scaler) {
 				sensor->scale_m = CCS_LIM(sensor, SCALER_N_MIN);
@@ -2270,13 +2254,15 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
 		}
 		fallthrough;
 	case V4L2_SEL_TGT_COMPOSE:
-		*crops[CCS_PAD_SRC] = *comp;
+		crop = v4l2_subdev_state_get_crop(sd_state, CCS_PAD_SRC,
+						  CCS_STREAM_PIXEL);
+		*crop = *comp;
 		fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC,
 						   CCS_STREAM_PIXEL);
 		fmt->width = comp->width;
 		fmt->height = comp->height;
 		if (which == V4L2_SUBDEV_FORMAT_ACTIVE && ssd == sensor->src)
-			sensor->src_src = *crops[CCS_PAD_SRC];
+			sensor->src_src = *crop;
 		break;
 	default:
 		WARN_ON_ONCE(1);
@@ -2419,7 +2405,7 @@ static int ccs_set_format(struct v4l2_subdev *subdev,
 {
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
 	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
-	struct v4l2_rect *crops[CCS_PADS];
+	struct v4l2_rect *crop;
 
 	if (subdev == &sensor->src->sd && fmt->pad == CCS_PAD_META)
 		return ccs_get_format(subdev, sd_state, fmt);
@@ -2461,12 +2447,13 @@ static int ccs_set_format(struct v4l2_subdev *subdev,
 		      CCS_LIM(sensor, MIN_Y_OUTPUT_SIZE),
 		      CCS_LIM(sensor, MAX_Y_OUTPUT_SIZE));
 
-	ccs_get_crop_compose(subdev, sd_state, crops, NULL);
+	crop = v4l2_subdev_state_get_crop(sd_state, ssd->sink_pad,
+					  CCS_STREAM_PIXEL);
 
-	crops[ssd->sink_pad]->left = 0;
-	crops[ssd->sink_pad]->top = 0;
-	crops[ssd->sink_pad]->width = fmt->format.width;
-	crops[ssd->sink_pad]->height = fmt->format.height;
+	crop->left = 0;
+	crop->top = 0;
+	crop->width = fmt->format.width;
+	crop->height = fmt->format.height;
 	ccs_propagate(subdev, sd_state, fmt->which, V4L2_SEL_TGT_CROP);
 
 	mutex_unlock(&sensor->mutex);
@@ -2521,26 +2508,23 @@ static int scaling_goodness(struct v4l2_subdev *subdev, int w, int ask_w,
 static void ccs_set_compose_binner(struct v4l2_subdev *subdev,
 				   struct v4l2_subdev_state *sd_state,
 				   struct v4l2_subdev_selection *sel,
-				   struct v4l2_rect **crops,
-				   struct v4l2_rect *comp)
+				   const struct v4l2_rect *sink_crop)
 {
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
 	unsigned int i;
 	unsigned int binh = 1, binv = 1;
-	int best = scaling_goodness(
-		subdev,
-		crops[CCS_PAD_SINK]->width, sel->r.width,
-		crops[CCS_PAD_SINK]->height, sel->r.height, sel->flags);
+	int best = scaling_goodness(subdev, sink_crop->width, sel->r.width,
+				    sink_crop->height, sel->r.height,
+				    sel->flags);
 
 	for (i = 0; i < sensor->nbinning_subtypes; i++) {
-		int this = scaling_goodness(
-			subdev,
-			crops[CCS_PAD_SINK]->width
-			/ sensor->binning_subtypes[i].horizontal,
-			sel->r.width,
-			crops[CCS_PAD_SINK]->height
-			/ sensor->binning_subtypes[i].vertical,
-			sel->r.height, sel->flags);
+		int this = scaling_goodness(subdev,
+					    sink_crop->width
+					    / sensor->binning_subtypes[i].horizontal,
+					    sel->r.width,
+					    sink_crop->height
+					    / sensor->binning_subtypes[i].vertical,
+					    sel->r.height, sel->flags);
 
 		if (this > best) {
 			binh = sensor->binning_subtypes[i].horizontal;
@@ -2553,8 +2537,8 @@ static void ccs_set_compose_binner(struct v4l2_subdev *subdev,
 		sensor->binning_horizontal = binh;
 	}
 
-	sel->r.width = (crops[CCS_PAD_SINK]->width / binh) & ~1;
-	sel->r.height = (crops[CCS_PAD_SINK]->height / binv) & ~1;
+	sel->r.width = (sink_crop->width / binh) & ~1;
+	sel->r.height = (sink_crop->height / binv) & ~1;
 }
 
 /*
@@ -2569,8 +2553,7 @@ static void ccs_set_compose_binner(struct v4l2_subdev *subdev,
 static void ccs_set_compose_scaler(struct v4l2_subdev *subdev,
 				   struct v4l2_subdev_state *sd_state,
 				   struct v4l2_subdev_selection *sel,
-				   struct v4l2_rect **crops,
-				   struct v4l2_rect *comp)
+				   const struct v4l2_rect *sink_crop)
 {
 	struct i2c_client *client = v4l2_get_subdevdata(subdev);
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
@@ -2582,17 +2565,12 @@ static void ccs_set_compose_scaler(struct v4l2_subdev *subdev,
 	unsigned int i;
 	int best = INT_MIN;
 
-	sel->r.width = min_t(unsigned int, sel->r.width,
-			     crops[CCS_PAD_SINK]->width);
-	sel->r.height = min_t(unsigned int, sel->r.height,
-			      crops[CCS_PAD_SINK]->height);
-
-	a = crops[CCS_PAD_SINK]->width
-		* CCS_LIM(sensor, SCALER_N_MIN) / sel->r.width;
-	b = crops[CCS_PAD_SINK]->height
-		* CCS_LIM(sensor, SCALER_N_MIN) / sel->r.height;
-	max_m = crops[CCS_PAD_SINK]->width
-		* CCS_LIM(sensor, SCALER_N_MIN)
+	sel->r.width = min_t(unsigned int, sel->r.width, sink_crop->width);
+	sel->r.height = min_t(unsigned int, sel->r.height, sink_crop->height);
+
+	a = sink_crop->width * CCS_LIM(sensor, SCALER_N_MIN) / sel->r.width;
+	b = sink_crop->height * CCS_LIM(sensor, SCALER_N_MIN) / sel->r.height;
+	max_m = sink_crop->width * CCS_LIM(sensor, SCALER_N_MIN)
 		/ CCS_LIM(sensor, MIN_X_OUTPUT_SIZE);
 
 	a = clamp(a, CCS_LIM(sensor, SCALER_M_MIN),
@@ -2623,14 +2601,12 @@ static void ccs_set_compose_scaler(struct v4l2_subdev *subdev,
 	}
 
 	for (i = 0; i < ntry; i++) {
-		int this = scaling_goodness(
-			subdev,
-			crops[CCS_PAD_SINK]->width
-			/ try[i] * CCS_LIM(sensor, SCALER_N_MIN),
-			sel->r.width,
-			crops[CCS_PAD_SINK]->height,
-			sel->r.height,
-			sel->flags);
+		int this = scaling_goodness(subdev,
+					    sink_crop->width
+					    / try[i]
+					    * CCS_LIM(sensor, SCALER_N_MIN),
+					    sel->r.width, sink_crop->height,
+					    sel->r.height, sel->flags);
 
 		dev_dbg(&client->dev, "trying factor %u (%u)\n", try[i], i);
 
@@ -2645,12 +2621,10 @@ static void ccs_set_compose_scaler(struct v4l2_subdev *subdev,
 			continue;
 
 		this = scaling_goodness(
-			subdev, crops[CCS_PAD_SINK]->width
-			/ try[i]
+			subdev, sink_crop->width / try[i]
 			* CCS_LIM(sensor, SCALER_N_MIN),
 			sel->r.width,
-			crops[CCS_PAD_SINK]->height
-			/ try[i]
+			sink_crop->height / try[i]
 			* CCS_LIM(sensor, SCALER_N_MIN),
 			sel->r.height,
 			sel->flags);
@@ -2662,18 +2636,13 @@ static void ccs_set_compose_scaler(struct v4l2_subdev *subdev,
 		}
 	}
 
-	sel->r.width =
-		(crops[CCS_PAD_SINK]->width
-		 / scale_m
-		 * CCS_LIM(sensor, SCALER_N_MIN)) & ~1;
+	sel->r.width = (sink_crop->width / scale_m
+			* CCS_LIM(sensor, SCALER_N_MIN)) & ~1;
 	if (mode == SMIAPP_SCALING_MODE_BOTH)
-		sel->r.height =
-			(crops[CCS_PAD_SINK]->height
-			 / scale_m
-			 * CCS_LIM(sensor, SCALER_N_MIN))
-			& ~1;
+		sel->r.height = (sink_crop->height / scale_m
+				 * CCS_LIM(sensor, SCALER_N_MIN)) & ~1;
 	else
-		sel->r.height = crops[CCS_PAD_SINK]->height;
+		sel->r.height = sink_crop->height;
 
 	if (sel->which == V4L2_SUBDEV_FORMAT_ACTIVE) {
 		sensor->scale_m = scale_m;
@@ -2687,17 +2656,21 @@ static int ccs_set_compose(struct v4l2_subdev *subdev,
 {
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
 	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
-	struct v4l2_rect *comp, *crops[CCS_PADS];
+	const struct v4l2_rect *sink_crop;
+	struct v4l2_rect *comp;
 
-	ccs_get_crop_compose(subdev, sd_state, crops, &comp);
+	sink_crop = v4l2_subdev_state_get_crop(sd_state, CCS_PAD_SINK,
+					       CCS_STREAM_PIXEL);
+	comp = v4l2_subdev_state_get_compose(sd_state, ssd->sink_pad,
+					     CCS_STREAM_PIXEL);
 
 	sel->r.top = 0;
 	sel->r.left = 0;
 
 	if (ssd == sensor->binner)
-		ccs_set_compose_binner(subdev, sd_state, sel, crops, comp);
+		ccs_set_compose_binner(subdev, sd_state, sel, sink_crop);
 	else
-		ccs_set_compose_scaler(subdev, sd_state, sel, crops, comp);
+		ccs_set_compose_scaler(subdev, sd_state, sel, sink_crop);
 
 	*comp = sel->r;
 	ccs_propagate(subdev, sd_state, sel->which, V4L2_SEL_TGT_COMPOSE);
@@ -2758,9 +2731,13 @@ static int ccs_set_crop(struct v4l2_subdev *subdev,
 {
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
 	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
-	struct v4l2_rect src_size = { 0 }, *crops[CCS_PADS], *comp;
+	struct v4l2_rect src_size = { 0 }, *crop;
+	const struct v4l2_rect *comp;
 
-	ccs_get_crop_compose(subdev, sd_state, crops, &comp);
+	crop = v4l2_subdev_state_get_crop(sd_state, sel->pad,
+					  CCS_STREAM_PIXEL);
+	comp = v4l2_subdev_state_get_compose(sd_state, ssd->sink_pad,
+					     CCS_STREAM_PIXEL);
 
 	if (sel->pad == ssd->sink_pad) {
 		struct v4l2_mbus_framefmt *mfmt =
@@ -2784,7 +2761,7 @@ static int ccs_set_crop(struct v4l2_subdev *subdev,
 	sel->r.left = min_t(int, sel->r.left, src_size.width - sel->r.width);
 	sel->r.top = min_t(int, sel->r.top, src_size.height - sel->r.height);
 
-	*crops[sel->pad] = sel->r;
+	*crop = sel->r;
 
 	if (ssd != sensor->pixel_array && sel->pad == CCS_PAD_SINK)
 		ccs_propagate(subdev, sd_state, sel->which, V4L2_SEL_TGT_CROP);
@@ -2809,14 +2786,17 @@ static int ccs_get_selection(struct v4l2_subdev *subdev,
 {
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
 	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
-	struct v4l2_rect *comp, *crops[CCS_PADS];
+	const struct v4l2_rect *crop, *comp;
 	int ret;
 
 	ret = ccs_sel_supported(subdev, sel);
 	if (ret)
 		return ret;
 
-	ccs_get_crop_compose(subdev, sd_state, crops, &comp);
+	crop = v4l2_subdev_state_get_crop(sd_state, sel->pad,
+					  CCS_STREAM_PIXEL);
+	comp = v4l2_subdev_state_get_compose(sd_state, ssd->sink_pad,
+					     CCS_STREAM_PIXEL);
 
 	switch (sel->target) {
 	case V4L2_SEL_TGT_CROP_BOUNDS:
@@ -2838,7 +2818,7 @@ static int ccs_get_selection(struct v4l2_subdev *subdev,
 		break;
 	case V4L2_SEL_TGT_CROP:
 	case V4L2_SEL_TGT_COMPOSE_BOUNDS:
-		sel->r = *crops[sel->pad];
+		sel->r = *crop;
 		break;
 	case V4L2_SEL_TGT_COMPOSE:
 		sel->r = *comp;
-- 
2.39.2

[PATCH v9 32/46] media: ccs: Rely on sub-device state locking

[Sakari Ailus]

Rely on sub-device state locking to serialise access to driver's data
structures. The driver-provided mutex is used as the state lock for all
driver sub-devices.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ccs/ccs-core.c | 192 +++++++++++++------------------
 drivers/media/i2c/ccs/ccs.h      |   1 -
 2 files changed, 81 insertions(+), 112 deletions(-)

diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
index 299b5ac6538c..f82f3ec37c7c 100644
--- a/drivers/media/i2c/ccs/ccs-core.c
+++ b/drivers/media/i2c/ccs/ccs-core.c
@@ -541,12 +541,13 @@ static int ccs_pll_update(struct ccs_sensor *sensor)
  *
  */
 
-static void __ccs_update_exposure_limits(struct ccs_sensor *sensor)
+static void __ccs_update_exposure_limits(struct ccs_sensor *sensor,
+					 struct v4l2_rect *pa_src)
 {
 	struct v4l2_ctrl *ctrl = sensor->exposure;
 	int max;
 
-	max = sensor->pa_src.height + sensor->vblank->val -
+	max = pa_src->height + sensor->vblank->val -
 		CCS_LIM(sensor, COARSE_INTEGRATION_TIME_MAX_MARGIN);
 
 	__v4l2_ctrl_modify_range(ctrl, ctrl->minimum, max, ctrl->step, max);
@@ -649,12 +650,20 @@ static int ccs_set_ctrl(struct v4l2_ctrl *ctrl)
 		container_of(ctrl->handler, struct ccs_subdev, ctrl_handler)
 			->sensor;
 	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
+	struct v4l2_subdev_state *state;
+	struct v4l2_rect *pa_src;
 	int pm_status;
 	u32 orient = 0;
 	unsigned int i;
 	int exposure;
 	int rval;
 
+	if (ctrl->id == V4L2_CID_VBLANK || ctrl->id == V4L2_CID_HBLANK) {
+		state = v4l2_subdev_get_locked_active_state(&sensor->pixel_array->sd);
+		pa_src = v4l2_subdev_state_get_crop(state, CCS_PA_PAD_SRC,
+						    CCS_STREAM_PIXEL);
+	}
+
 	switch (ctrl->id) {
 	case V4L2_CID_HFLIP:
 	case V4L2_CID_VFLIP:
@@ -673,7 +682,7 @@ static int ccs_set_ctrl(struct v4l2_ctrl *ctrl)
 	case V4L2_CID_VBLANK:
 		exposure = sensor->exposure->val;
 
-		__ccs_update_exposure_limits(sensor);
+		__ccs_update_exposure_limits(sensor, pa_src);
 
 		if (exposure > sensor->exposure->maximum) {
 			sensor->exposure->val =	sensor->exposure->maximum;
@@ -765,12 +774,12 @@ static int ccs_set_ctrl(struct v4l2_ctrl *ctrl)
 		break;
 	case V4L2_CID_VBLANK:
 		rval = ccs_write(sensor, FRAME_LENGTH_LINES,
-				 sensor->pa_src.height + ctrl->val);
+				 pa_src->height + ctrl->val);
 
 		break;
 	case V4L2_CID_HBLANK:
 		rval = ccs_write(sensor, LINE_LENGTH_PCK,
-				 sensor->pa_src.width + ctrl->val);
+				 pa_src->width + ctrl->val);
 
 		break;
 	case V4L2_CID_TEST_PATTERN:
@@ -1225,7 +1234,8 @@ static int ccs_get_mbus_formats(struct ccs_sensor *sensor)
 	return 0;
 }
 
-static void ccs_update_blanking(struct ccs_sensor *sensor)
+static void ccs_update_blanking(struct ccs_sensor *sensor,
+				struct v4l2_rect *pa_src)
 {
 	struct v4l2_ctrl *vblank = sensor->vblank;
 	struct v4l2_ctrl *hblank = sensor->hblank;
@@ -1248,21 +1258,26 @@ static void ccs_update_blanking(struct ccs_sensor *sensor)
 
 	min = max_t(int,
 		    CCS_LIM(sensor, MIN_FRAME_BLANKING_LINES),
-		    min_fll - sensor->pa_src.height);
-	max = max_fll -	sensor->pa_src.height;
+		    min_fll - pa_src->height);
+	max = max_fll -	pa_src->height;
 
 	__v4l2_ctrl_modify_range(vblank, min, max, vblank->step, min);
 
-	min = max_t(int, min_llp - sensor->pa_src.width, min_lbp);
-	max = max_llp - sensor->pa_src.width;
+	min = max_t(int, min_llp - pa_src->width, min_lbp);
+	max = max_llp - pa_src->width;
 
 	__v4l2_ctrl_modify_range(hblank, min, max, hblank->step, min);
 
-	__ccs_update_exposure_limits(sensor);
+	__ccs_update_exposure_limits(sensor, pa_src);
 }
 
 static int ccs_pll_blanking_update(struct ccs_sensor *sensor)
 {
+	struct v4l2_subdev_state *state =
+		v4l2_subdev_get_locked_active_state(&sensor->pixel_array->sd);
+	struct v4l2_rect *pa_src =
+		v4l2_subdev_state_get_crop(state, CCS_PA_PAD_SRC,
+					   CCS_STREAM_PIXEL);
 	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
 	int rval;
 
@@ -1271,15 +1286,15 @@ static int ccs_pll_blanking_update(struct ccs_sensor *sensor)
 		return rval;
 
 	/* Output from pixel array, including blanking */
-	ccs_update_blanking(sensor);
+	ccs_update_blanking(sensor, pa_src);
 
 	dev_dbg(&client->dev, "vblank\t\t%d\n", sensor->vblank->val);
 	dev_dbg(&client->dev, "hblank\t\t%d\n", sensor->hblank->val);
 
 	dev_dbg(&client->dev, "real timeperframe\t100/%d\n",
 		sensor->pll.pixel_rate_pixel_array /
-		((sensor->pa_src.width + sensor->hblank->val) *
-		 (sensor->pa_src.height + sensor->vblank->val) / 100));
+		((pa_src->width + sensor->hblank->val) *
+		 (pa_src->height + sensor->vblank->val) / 100));
 
 	return 0;
 }
@@ -1788,6 +1803,16 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
 			      u64 streams_mask)
 {
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
+	struct v4l2_subdev_state *pa_state =
+		v4l2_subdev_get_locked_active_state(&sensor->pixel_array->sd);
+	struct v4l2_subdev_state *src_state =
+		v4l2_subdev_get_locked_active_state(&sensor->src->sd);
+	struct v4l2_rect *pa_src =
+		v4l2_subdev_state_get_crop(pa_state, CCS_PA_PAD_SRC,
+					   CCS_STREAM_PIXEL);
+	struct v4l2_rect *src_src =
+		v4l2_subdev_state_get_crop(src_state, CCS_PAD_SRC,
+					   CCS_STREAM_PIXEL);
 	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
 	unsigned int binning_mode;
 	int rval;
@@ -1835,22 +1860,20 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
 		goto err_pm_put;
 
 	/* Analog crop start coordinates */
-	rval = ccs_write(sensor, X_ADDR_START, sensor->pa_src.left);
+	rval = ccs_write(sensor, X_ADDR_START, pa_src->left);
 	if (rval < 0)
 		goto err_pm_put;
 
-	rval = ccs_write(sensor, Y_ADDR_START, sensor->pa_src.top);
+	rval = ccs_write(sensor, Y_ADDR_START, pa_src->top);
 	if (rval < 0)
 		goto err_pm_put;
 
 	/* Analog crop end coordinates */
-	rval = ccs_write(sensor, X_ADDR_END,
-			 sensor->pa_src.left + sensor->pa_src.width - 1);
+	rval = ccs_write(sensor, X_ADDR_END, pa_src->left + pa_src->width - 1);
 	if (rval < 0)
 		goto err_pm_put;
 
-	rval = ccs_write(sensor, Y_ADDR_END,
-			 sensor->pa_src.top + sensor->pa_src.height - 1);
+	rval = ccs_write(sensor, Y_ADDR_END, pa_src->top + pa_src->height - 1);
 	if (rval < 0)
 		goto err_pm_put;
 
@@ -1862,23 +1885,30 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
 	/* Digital crop */
 	if (CCS_LIM(sensor, DIGITAL_CROP_CAPABILITY)
 	    == CCS_DIGITAL_CROP_CAPABILITY_INPUT_CROP) {
+		struct v4l2_subdev_state *scaler_state =
+			v4l2_subdev_get_locked_active_state(&sensor->scaler->sd);
+		struct v4l2_rect *scaler_sink =
+			v4l2_subdev_state_get_crop(scaler_state,
+						   sensor->scaler->sink_pad,
+						   CCS_STREAM_PIXEL);
+
 		rval = ccs_write(sensor, DIGITAL_CROP_X_OFFSET,
-				 sensor->scaler_sink.left);
+				 scaler_sink->left);
 		if (rval < 0)
 			goto err_pm_put;
 
 		rval = ccs_write(sensor, DIGITAL_CROP_Y_OFFSET,
-				 sensor->scaler_sink.top);
+				 scaler_sink->top);
 		if (rval < 0)
 			goto err_pm_put;
 
 		rval = ccs_write(sensor, DIGITAL_CROP_IMAGE_WIDTH,
-				 sensor->scaler_sink.width);
+				 scaler_sink->width);
 		if (rval < 0)
 			goto err_pm_put;
 
 		rval = ccs_write(sensor, DIGITAL_CROP_IMAGE_HEIGHT,
-				 sensor->scaler_sink.height);
+				 scaler_sink->height);
 		if (rval < 0)
 			goto err_pm_put;
 	}
@@ -1896,10 +1926,10 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
 	}
 
 	/* Output size from sensor */
-	rval = ccs_write(sensor, X_OUTPUT_SIZE, sensor->src_src.width);
+	rval = ccs_write(sensor, X_OUTPUT_SIZE, src_src->width);
 	if (rval < 0)
 		goto err_pm_put;
-	rval = ccs_write(sensor, Y_OUTPUT_SIZE, sensor->src_src.height);
+	rval = ccs_write(sensor, Y_OUTPUT_SIZE, src_src->height);
 	if (rval < 0)
 		goto err_pm_put;
 
@@ -2088,9 +2118,6 @@ static int ccs_enum_mbus_code(struct v4l2_subdev *subdev,
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
 	unsigned int i;
 	int idx = -1;
-	int rval = -EINVAL;
-
-	mutex_lock(&sensor->mutex);
 
 	dev_err(&client->dev, "subdev %s, pad %u, index %u\n",
 		subdev->name, code->pad, code->index);
@@ -2098,12 +2125,11 @@ static int ccs_enum_mbus_code(struct v4l2_subdev *subdev,
 	if (subdev == &sensor->src->sd) {
 		if (code->pad == CCS_PAD_META) {
 			if (code->index)
-				goto out;
+				return -EINVAL;
 
 			code->code = MEDIA_BUS_FMT_CCS_EMBEDDED;
 
-			rval = 0;
-			goto out;
+			return 0;
 		}
 		if (code->stream == CCS_STREAM_META) {
 			struct v4l2_mbus_framefmt *pix_fmt =
@@ -2150,21 +2176,21 @@ static int ccs_enum_mbus_code(struct v4l2_subdev *subdev,
 			}
 
 			if (WARN_ON(i > ARRAY_SIZE(codes)) || code->index >= i)
-				goto out;
+				return -EINVAL;
 
 			code->code = codes[code->index];
-			rval = 0;
-			goto out;
+
+			return 0;
 		}
 	}
 
 	if (subdev != &sensor->src->sd || code->pad != CCS_PAD_SRC) {
 		if (code->index)
-			goto out;
+			return -EINVAL;
 
 		code->code = sensor->internal_csi_format->code;
-		rval = 0;
-		goto out;
+
+		return 0;
 	}
 
 	for (i = 0; i < ARRAY_SIZE(ccs_csi_data_formats); i++) {
@@ -2175,18 +2201,14 @@ static int ccs_enum_mbus_code(struct v4l2_subdev *subdev,
 			code->code = ccs_csi_data_formats[i].code;
 			dev_err(&client->dev, "found index %u, i %u, code %x\n",
 				code->index, i, code->code);
-			rval = 0;
-			break;
+			return 0;
 		}
 	}
 
-out:
-	mutex_unlock(&sensor->mutex);
-
-	return rval;
+	return -EINVAL;
 }
 
-static u32 __ccs_get_mbus_code(struct v4l2_subdev *subdev, unsigned int pad)
+static u32 ccs_get_mbus_code(struct v4l2_subdev *subdev, unsigned int pad)
 {
 	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
 
@@ -2196,33 +2218,19 @@ static u32 __ccs_get_mbus_code(struct v4l2_subdev *subdev, unsigned int pad)
 		return sensor->internal_csi_format->code;
 }
 
-static int __ccs_get_format(struct v4l2_subdev *subdev,
-			    struct v4l2_subdev_state *sd_state,
-			    struct v4l2_subdev_format *fmt)
+static int ccs_get_format(struct v4l2_subdev *subdev,
+			  struct v4l2_subdev_state *sd_state,
+			  struct v4l2_subdev_format *fmt)
 {
 	fmt->format = *v4l2_subdev_state_get_format(sd_state, fmt->pad,
 						    fmt->stream);
 
 	if (fmt->pad != CCS_PAD_META && fmt->stream != CCS_STREAM_META)
-		fmt->format.code = __ccs_get_mbus_code(subdev, fmt->pad);
+		fmt->format.code = ccs_get_mbus_code(subdev, fmt->pad);
 
 	return 0;
 }
 
-static int ccs_get_format(struct v4l2_subdev *subdev,
-			  struct v4l2_subdev_state *sd_state,
-			  struct v4l2_subdev_format *fmt)
-{
-	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
-	int rval;
-
-	mutex_lock(&sensor->mutex);
-	rval = __ccs_get_format(subdev, sd_state, fmt);
-	mutex_unlock(&sensor->mutex);
-
-	return rval;
-}
-
 /* Changes require propagation only on sink pad. */
 static void ccs_propagate(struct v4l2_subdev *subdev,
 			  struct v4l2_subdev_state *sd_state, int which,
@@ -2246,7 +2254,6 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
 				sensor->scale_m = CCS_LIM(sensor, SCALER_N_MIN);
 				sensor->scaling_mode =
 					CCS_SCALING_MODE_NO_SCALING;
-				sensor->scaler_sink = *comp;
 			} else if (ssd == sensor->binner) {
 				sensor->binning_horizontal = 1;
 				sensor->binning_vertical = 1;
@@ -2261,8 +2268,6 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
 						   CCS_STREAM_PIXEL);
 		fmt->width = comp->width;
 		fmt->height = comp->height;
-		if (which == V4L2_SUBDEV_FORMAT_ACTIVE && ssd == sensor->src)
-			sensor->src_src = *crop;
 		break;
 	default:
 		WARN_ON_ONCE(1);
@@ -2281,7 +2286,7 @@ static int ccs_set_format_source(struct v4l2_subdev *subdev,
 	unsigned int i;
 	int rval;
 
-	rval = __ccs_get_format(subdev, sd_state, fmt);
+	rval = ccs_get_format(subdev, sd_state, fmt);
 	if (rval)
 		return rval;
 
@@ -2410,13 +2415,9 @@ static int ccs_set_format(struct v4l2_subdev *subdev,
 	if (subdev == &sensor->src->sd && fmt->pad == CCS_PAD_META)
 		return ccs_get_format(subdev, sd_state, fmt);
 
-	mutex_lock(&sensor->mutex);
-
 	if (subdev == &sensor->src->sd && fmt->stream == CCS_STREAM_META) {
 		ccs_set_format_meta(subdev, sd_state, &fmt->format);
 
-		mutex_unlock(&sensor->mutex);
-
 		return 0;
 	}
 
@@ -2427,13 +2428,12 @@ static int ccs_set_format(struct v4l2_subdev *subdev,
 		if (sensor->embedded_start != sensor->embedded_end)
 			ccs_set_format_meta(subdev, sd_state, NULL);
 
-		mutex_unlock(&sensor->mutex);
-
 		return rval;
 	}
 
 	/* Sink pad. Width and height are changeable here. */
-	fmt->format.code = __ccs_get_mbus_code(subdev, fmt->pad);
+	fmt->format.code = ccs_get_mbus_code(subdev, fmt->pad);
+
 	fmt->format.width &= ~1;
 	fmt->format.height &= ~1;
 	fmt->format.field = V4L2_FIELD_NONE;
@@ -2456,8 +2456,6 @@ static int ccs_set_format(struct v4l2_subdev *subdev,
 	crop->height = fmt->format.height;
 	ccs_propagate(subdev, sd_state, fmt->which, V4L2_SEL_TGT_CROP);
 
-	mutex_unlock(&sensor->mutex);
-
 	return 0;
 }
 
@@ -2765,9 +2763,6 @@ static int ccs_set_crop(struct v4l2_subdev *subdev,
 
 	if (ssd != sensor->pixel_array && sel->pad == CCS_PAD_SINK)
 		ccs_propagate(subdev, sd_state, sel->which, V4L2_SEL_TGT_CROP);
-	else if (sel->which == V4L2_SUBDEV_FORMAT_ACTIVE &&
-		 ssd == sensor->pixel_array)
-		sensor->pa_src = sel->r;
 
 	return 0;
 }
@@ -2839,8 +2834,6 @@ static int ccs_set_selection(struct v4l2_subdev *subdev,
 	if (ret)
 		return ret;
 
-	mutex_lock(&sensor->mutex);
-
 	sel->r.left = max(0, sel->r.left & ~1);
 	sel->r.top = max(0, sel->r.top & ~1);
 	sel->r.width = CCS_ALIGN_DIM(sel->r.width, sel->flags);
@@ -2862,7 +2855,6 @@ static int ccs_set_selection(struct v4l2_subdev *subdev,
 		ret = -EINVAL;
 	}
 
-	mutex_unlock(&sensor->mutex);
 	return ret;
 }
 
@@ -3238,6 +3230,7 @@ static int ccs_init_subdev(struct ccs_sensor *sensor,
 
 	ssd->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
 	ssd->sd.entity.function = function;
+	ssd->sd.state_lock = &sensor->mutex;
 	ssd->sensor = sensor;
 
 	ssd->npads = num_pads;
@@ -3277,8 +3270,8 @@ static int ccs_init_subdev(struct ccs_sensor *sensor,
 	return 0;
 }
 
-static int __ccs_init_state(struct v4l2_subdev *sd,
-			    struct v4l2_subdev_state *sd_state)
+static int ccs_init_state(struct v4l2_subdev *sd,
+			  struct v4l2_subdev_state *sd_state)
 {
 	struct ccs_subdev *ssd = to_ccs_subdev(sd);
 	struct ccs_sensor *sensor = ssd->sensor;
@@ -3297,12 +3290,8 @@ static int __ccs_init_state(struct v4l2_subdev *sd,
 	fmt->code = sensor->internal_csi_format->code;
 	fmt->field = V4L2_FIELD_NONE;
 
-	if (ssd == sensor->pixel_array) {
-		if (is_active)
-			sensor->pa_src = *crop;
-
+	if (ssd == sensor->pixel_array)
 		return 0;
-	}
 
 	fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC,
 					   CCS_STREAM_PIXEL);
@@ -3315,20 +3304,6 @@ static int __ccs_init_state(struct v4l2_subdev *sd,
 	return 0;
 }
 
-static int ccs_init_state(struct v4l2_subdev *sd,
-			  struct v4l2_subdev_state *sd_state)
-{
-	struct ccs_subdev *ssd = to_ccs_subdev(sd);
-	struct ccs_sensor *sensor = ssd->sensor;
-	int rval;
-
-	mutex_lock(&sensor->mutex);
-	rval = __ccs_init_state(sd, sd_state);
-	mutex_unlock(&sensor->mutex);
-
-	return rval;
-}
-
 static int ccs_src_init_state(struct v4l2_subdev *sd,
 			      struct v4l2_subdev_state *sd_state)
 {
@@ -3353,25 +3328,20 @@ static int ccs_src_init_state(struct v4l2_subdev *sd,
 	struct ccs_sensor *sensor = ssd->sensor;
 	int rval;
 
-	mutex_lock(&sensor->mutex);
-
 	if (sensor->embedded_start != sensor->embedded_end)
 		routing.num_routes++;
 
 	rval = v4l2_subdev_set_routing(sd, sd_state, &routing);
 	if (rval)
-		goto out;
+		return 0;
 
-	rval = __ccs_init_state(sd, sd_state);
+	rval = ccs_init_state(sd, sd_state);
 	if (rval)
-		goto out;
+		return 0;
 
 	if (sensor->embedded_start != sensor->embedded_end)
 		ccs_set_format_meta(sd, sd_state, NULL);
 
-out:
-	mutex_unlock(&sensor->mutex);
-
 	return rval;
 }
 
diff --git a/drivers/media/i2c/ccs/ccs.h b/drivers/media/i2c/ccs/ccs.h
index 477b2fb99aa0..dcca3f88ea67 100644
--- a/drivers/media/i2c/ccs/ccs.h
+++ b/drivers/media/i2c/ccs/ccs.h
@@ -230,7 +230,6 @@ struct ccs_sensor {
 	u32 mbus_frame_fmts;
 	const struct ccs_csi_data_format *csi_format;
 	const struct ccs_csi_data_format *internal_csi_format;
-	struct v4l2_rect pa_src, scaler_sink, src_src;
 	u32 default_mbus_frame_fmts;
 	int default_pixel_order;
 	struct ccs_data_container sdata, mdata;
-- 
2.39.2

[PATCH v9 33/46] media: ccs: Compute binning configuration from sub-device state

[Sakari Ailus]

Calculate binning configuration from sub-device state so the state related
configuration can be removed from the driver's device context struct.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ccs/ccs-core.c | 56 +++++++++++++++++++-------------
 drivers/media/i2c/ccs/ccs.h      |  3 --
 2 files changed, 33 insertions(+), 26 deletions(-)

diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
index f82f3ec37c7c..08e719d611fb 100644
--- a/drivers/media/i2c/ccs/ccs-core.c
+++ b/drivers/media/i2c/ccs/ccs-core.c
@@ -511,13 +511,36 @@ static int ccs_pll_try(struct ccs_sensor *sensor, struct ccs_pll *pll)
 	return ccs_pll_calculate(&client->dev, &lim, pll);
 }
 
+static void
+ccs_get_binning(struct ccs_sensor *sensor, u8 *binning_mode, u8 *binh, u8 *binv)
+{
+	struct v4l2_subdev_state *state =
+		v4l2_subdev_get_locked_active_state(&sensor->binner->sd);
+	const struct v4l2_rect *sink_crop =
+		v4l2_subdev_state_get_crop(state, CCS_PAD_SINK,
+					   CCS_STREAM_PIXEL);
+	const struct v4l2_rect *sink_comp =
+		v4l2_subdev_state_get_compose(state, CCS_PAD_SINK,
+					      CCS_STREAM_PIXEL);
+
+	if (binning_mode)
+		*binning_mode =	sink_crop->width == sink_comp->width &&
+				sink_crop->height == sink_comp->height ? 0 : 1;
+
+	*binh = sink_crop->width / sink_comp->width;
+	*binv = sink_crop->height / sink_comp->height;
+}
+
 static int ccs_pll_update(struct ccs_sensor *sensor)
 {
 	struct ccs_pll *pll = &sensor->pll;
+	u8 binh, binv;
 	int rval;
 
-	pll->binning_horizontal = sensor->binning_horizontal;
-	pll->binning_vertical = sensor->binning_vertical;
+	ccs_get_binning(sensor, NULL, &binh, &binv);
+
+	pll->binning_horizontal = binh;
+	pll->binning_vertical = binv;
 	pll->link_freq =
 		sensor->link_freq->qmenu_int[sensor->link_freq->val];
 	pll->scale_m = sensor->scale_m;
@@ -1241,8 +1264,11 @@ static void ccs_update_blanking(struct ccs_sensor *sensor,
 	struct v4l2_ctrl *hblank = sensor->hblank;
 	u16 min_fll, max_fll, min_llp, max_llp, min_lbp;
 	int min, max;
+	u8 binh, binv;
+
+	ccs_get_binning(sensor, NULL, &binh, &binv);
 
-	if (sensor->binning_vertical > 1 || sensor->binning_horizontal > 1) {
+	if (binv > 1 || binh > 1) {
 		min_fll = CCS_LIM(sensor, MIN_FRAME_LENGTH_LINES_BIN);
 		max_fll = CCS_LIM(sensor, MAX_FRAME_LENGTH_LINES_BIN);
 		min_llp = CCS_LIM(sensor, MIN_LINE_LENGTH_PCK_BIN);
@@ -1814,7 +1840,7 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
 		v4l2_subdev_state_get_crop(src_state, CCS_PAD_SRC,
 					   CCS_STREAM_PIXEL);
 	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
-	unsigned int binning_mode;
+	u8 binning_mode, binh, binv;
 	int rval;
 
 	if (pad != CCS_PAD_SRC)
@@ -1836,19 +1862,12 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
 		goto err_pm_put;
 
 	/* Binning configuration */
-	if (sensor->binning_horizontal == 1 &&
-	    sensor->binning_vertical == 1) {
-		binning_mode = 0;
-	} else {
-		u8 binning_type =
-			(sensor->binning_horizontal << 4)
-			| sensor->binning_vertical;
+	ccs_get_binning(sensor,	&binning_mode, &binh, &binv);
 
-		rval = ccs_write(sensor, BINNING_TYPE, binning_type);
+	if (binning_mode) {
+		rval = ccs_write(sensor, BINNING_TYPE, (binh << 4) | binv);
 		if (rval < 0)
 			goto err_pm_put;
-
-		binning_mode = 1;
 	}
 	rval = ccs_write(sensor, BINNING_MODE, binning_mode);
 	if (rval < 0)
@@ -2254,9 +2273,6 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
 				sensor->scale_m = CCS_LIM(sensor, SCALER_N_MIN);
 				sensor->scaling_mode =
 					CCS_SCALING_MODE_NO_SCALING;
-			} else if (ssd == sensor->binner) {
-				sensor->binning_horizontal = 1;
-				sensor->binning_vertical = 1;
 			}
 		}
 		fallthrough;
@@ -2530,10 +2546,6 @@ static void ccs_set_compose_binner(struct v4l2_subdev *subdev,
 			best = this;
 		}
 	}
-	if (sel->which == V4L2_SUBDEV_FORMAT_ACTIVE) {
-		sensor->binning_vertical = binv;
-		sensor->binning_horizontal = binh;
-	}
 
 	sel->r.width = (sink_crop->width / binh) & ~1;
 	sel->r.height = (sink_crop->height / binv) & ~1;
@@ -3715,8 +3727,6 @@ static int ccs_probe(struct i2c_client *client)
 				sensor->binning_subtypes[i].vertical);
 		}
 	}
-	sensor->binning_horizontal = 1;
-	sensor->binning_vertical = 1;
 
 	if (device_create_file(&client->dev, &dev_attr_ident) != 0) {
 		dev_err(&client->dev, "sysfs ident entry creation failed\n");
diff --git a/drivers/media/i2c/ccs/ccs.h b/drivers/media/i2c/ccs/ccs.h
index dcca3f88ea67..aadbd4302607 100644
--- a/drivers/media/i2c/ccs/ccs.h
+++ b/drivers/media/i2c/ccs/ccs.h
@@ -237,9 +237,6 @@ struct ccs_sensor {
 	u32 embedded_mbus_code;
 	u8 emb_data_ctrl;
 
-	u8 binning_horizontal;
-	u8 binning_vertical;
-
 	u8 scale_m;
 	u8 scaling_mode;
 
-- 
2.39.2

[PATCH v9 34/46] media: ccs: Compute scaling configuration from sub-device state

[Sakari Ailus]

Compute scaling configuration from sub-device state instead of storing it
to the driver's device context struct.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ccs/ccs-core.c | 59 ++++++++++++++++++++++----------
 drivers/media/i2c/ccs/ccs.h      |  3 --
 2 files changed, 40 insertions(+), 22 deletions(-)

diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
index 08e719d611fb..541faa7d84a6 100644
--- a/drivers/media/i2c/ccs/ccs-core.c
+++ b/drivers/media/i2c/ccs/ccs-core.c
@@ -531,19 +531,51 @@ ccs_get_binning(struct ccs_sensor *sensor, u8 *binning_mode, u8 *binh, u8 *binv)
 	*binv = sink_crop->height / sink_comp->height;
 }
 
+static void ccs_get_scaling(struct ccs_sensor *sensor, u8 *scaling_mode,
+			    u8 *scale_m)
+{
+	struct v4l2_subdev_state *state =
+		v4l2_subdev_get_locked_active_state(&sensor->scaler->sd);
+	const struct v4l2_rect *sink_crop =
+		v4l2_subdev_state_get_crop(state, CCS_PAD_SINK,
+					   CCS_STREAM_PIXEL);
+	const struct v4l2_rect *sink_comp =
+		v4l2_subdev_state_get_compose(state, CCS_PAD_SINK,
+					      CCS_STREAM_PIXEL);
+
+	*scale_m = sink_crop->width * CCS_LIM(sensor, SCALER_N_MIN) /
+		sink_comp->width;
+
+	if (!scaling_mode)
+		return;
+
+	if (sink_crop->width == sink_comp->width)
+		*scaling_mode = CCS_SCALING_MODE_NO_SCALING;
+	else if (sink_crop->height == sink_comp->height)
+		*scaling_mode = CCS_SCALING_MODE_HORIZONTAL;
+	else
+		*scaling_mode = SMIAPP_SCALING_MODE_BOTH;
+}
+
 static int ccs_pll_update(struct ccs_sensor *sensor)
 {
 	struct ccs_pll *pll = &sensor->pll;
 	u8 binh, binv;
+	u8 scale_m;
 	int rval;
 
 	ccs_get_binning(sensor, NULL, &binh, &binv);
 
+	if (sensor->scaler)
+		ccs_get_scaling(sensor, NULL, &scale_m);
+	else
+		scale_m = CCS_LIM(sensor, SCALER_N_MIN);
+
 	pll->binning_horizontal = binh;
 	pll->binning_vertical = binv;
 	pll->link_freq =
 		sensor->link_freq->qmenu_int[sensor->link_freq->val];
-	pll->scale_m = sensor->scale_m;
+	pll->scale_m = scale_m;
 	pll->bits_per_pixel = sensor->csi_format->compressed;
 
 	rval = ccs_pll_try(sensor, pll);
@@ -1186,7 +1218,7 @@ static int ccs_get_mbus_formats(struct ccs_sensor *sensor)
 	/* Figure out which BPP values can be used with which formats. */
 	pll->binning_horizontal = 1;
 	pll->binning_vertical = 1;
-	pll->scale_m = sensor->scale_m;
+	pll->scale_m = CCS_LIM(sensor, SCALER_N_MIN);
 
 	for (i = 0; i < ARRAY_SIZE(ccs_csi_data_formats); i++) {
 		sensor->compressed_min_bpp =
@@ -1935,11 +1967,15 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
 	/* Scaling */
 	if (CCS_LIM(sensor, SCALING_CAPABILITY)
 	    != CCS_SCALING_CAPABILITY_NONE) {
-		rval = ccs_write(sensor, SCALING_MODE, sensor->scaling_mode);
+		u8 scaling_mode, scale_m;
+
+		ccs_get_scaling(sensor, &scaling_mode, &scale_m);
+
+		rval = ccs_write(sensor, SCALING_MODE, scaling_mode);
 		if (rval < 0)
 			goto err_pm_put;
 
-		rval = ccs_write(sensor, SCALE_M, sensor->scale_m);
+		rval = ccs_write(sensor, SCALE_M, scale_m);
 		if (rval < 0)
 			goto err_pm_put;
 	}
@@ -2255,7 +2291,6 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
 			  struct v4l2_subdev_state *sd_state, int which,
 			  int target)
 {
-	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
 	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
 	struct v4l2_rect *comp, *crop;
 	struct v4l2_mbus_framefmt *fmt;
@@ -2268,13 +2303,6 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
 						  CCS_STREAM_PIXEL);
 		comp->width = crop->width;
 		comp->height = crop->height;
-		if (which == V4L2_SUBDEV_FORMAT_ACTIVE) {
-			if (ssd == sensor->scaler) {
-				sensor->scale_m = CCS_LIM(sensor, SCALER_N_MIN);
-				sensor->scaling_mode =
-					CCS_SCALING_MODE_NO_SCALING;
-			}
-		}
 		fallthrough;
 	case V4L2_SEL_TGT_COMPOSE:
 		crop = v4l2_subdev_state_get_crop(sd_state, CCS_PAD_SRC,
@@ -2653,11 +2681,6 @@ static void ccs_set_compose_scaler(struct v4l2_subdev *subdev,
 				 * CCS_LIM(sensor, SCALER_N_MIN)) & ~1;
 	else
 		sel->r.height = sink_crop->height;
-
-	if (sel->which == V4L2_SUBDEV_FORMAT_ACTIVE) {
-		sensor->scale_m = scale_m;
-		sensor->scaling_mode = mode;
-	}
 }
 /* We're only called on source pads. This function sets scaling. */
 static int ccs_set_compose(struct v4l2_subdev *subdev,
@@ -3763,8 +3786,6 @@ static int ccs_probe(struct i2c_client *client)
 	sensor->pixel_array = &sensor->ssds[sensor->ssds_used];
 	sensor->ssds_used++;
 
-	sensor->scale_m = CCS_LIM(sensor, SCALER_N_MIN);
-
 	/* prepare PLL configuration input values */
 	sensor->pll.bus_type = CCS_PLL_BUS_TYPE_CSI2_DPHY;
 	sensor->pll.csi2.lanes = sensor->hwcfg.lanes;
diff --git a/drivers/media/i2c/ccs/ccs.h b/drivers/media/i2c/ccs/ccs.h
index aadbd4302607..1c30fa85bed6 100644
--- a/drivers/media/i2c/ccs/ccs.h
+++ b/drivers/media/i2c/ccs/ccs.h
@@ -237,9 +237,6 @@ struct ccs_sensor {
 	u32 embedded_mbus_code;
 	u8 emb_data_ctrl;
 
-	u8 scale_m;
-	u8 scaling_mode;
-
 	u8 frame_skip;
 	u16 embedded_start; /* embedded data start line */
 	u16 embedded_end;
-- 
2.39.2

[PATCH v9 35/46] media: ccs: Remove which parameter from ccs_propagate

[Sakari Ailus]

ccs_propagate() no longer stores information in the driver's context
struct. The which parameter can thus be removed.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 drivers/media/i2c/ccs/ccs-core.c | 12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)

diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
index 541faa7d84a6..0f68e64a4042 100644
--- a/drivers/media/i2c/ccs/ccs-core.c
+++ b/drivers/media/i2c/ccs/ccs-core.c
@@ -2288,8 +2288,7 @@ static int ccs_get_format(struct v4l2_subdev *subdev,
 
 /* Changes require propagation only on sink pad. */
 static void ccs_propagate(struct v4l2_subdev *subdev,
-			  struct v4l2_subdev_state *sd_state, int which,
-			  int target)
+			  struct v4l2_subdev_state *sd_state, int target)
 {
 	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
 	struct v4l2_rect *comp, *crop;
@@ -2498,7 +2497,7 @@ static int ccs_set_format(struct v4l2_subdev *subdev,
 	crop->top = 0;
 	crop->width = fmt->format.width;
 	crop->height = fmt->format.height;
-	ccs_propagate(subdev, sd_state, fmt->which, V4L2_SEL_TGT_CROP);
+	ccs_propagate(subdev, sd_state, V4L2_SEL_TGT_CROP);
 
 	return 0;
 }
@@ -2706,7 +2705,7 @@ static int ccs_set_compose(struct v4l2_subdev *subdev,
 		ccs_set_compose_scaler(subdev, sd_state, sel, sink_crop);
 
 	*comp = sel->r;
-	ccs_propagate(subdev, sd_state, sel->which, V4L2_SEL_TGT_COMPOSE);
+	ccs_propagate(subdev, sd_state, V4L2_SEL_TGT_COMPOSE);
 
 	if (sel->which == V4L2_SUBDEV_FORMAT_ACTIVE)
 		return ccs_pll_blanking_update(sensor);
@@ -2797,7 +2796,7 @@ static int ccs_set_crop(struct v4l2_subdev *subdev,
 	*crop = sel->r;
 
 	if (ssd != sensor->pixel_array && sel->pad == CCS_PAD_SINK)
-		ccs_propagate(subdev, sd_state, sel->which, V4L2_SEL_TGT_CROP);
+		ccs_propagate(subdev, sd_state, V4L2_SEL_TGT_CROP);
 
 	return 0;
 }
@@ -3316,7 +3315,6 @@ static int ccs_init_state(struct v4l2_subdev *sd,
 		v4l2_subdev_state_get_format(sd_state, pad, CCS_STREAM_PIXEL);
 	struct v4l2_rect *crop =
 		v4l2_subdev_state_get_crop(sd_state, pad, CCS_STREAM_PIXEL);
-	bool is_active = !sd->active_state || sd->active_state == sd_state;
 
 	ccs_get_native_size(ssd, crop);
 
@@ -3334,7 +3332,7 @@ static int ccs_init_state(struct v4l2_subdev *sd,
 		sensor->csi_format->code : sensor->internal_csi_format->code;
 	fmt->field = V4L2_FIELD_NONE;
 
-	ccs_propagate(sd, sd_state, is_active, V4L2_SEL_TGT_CROP);
+	ccs_propagate(sd, sd_state, V4L2_SEL_TGT_CROP);
 
 	return 0;
 }
-- 
2.39.2

[PATCH v9 36/46] media: uapi: Add media bus code for ov2740 embedded data

[Sakari Ailus]

Add a media bus code for ov2740 camera sensor embedded data and document
it.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Julien Massot <julien.massot@collabora.com>
---
 .../media/v4l/subdev-formats.rst              | 70 +++++++++++++++++++
 include/uapi/linux/media-bus-format.h         |  3 +-
 2 files changed, 72 insertions(+), 1 deletion(-)

diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
index a04756092238..c99b58cb8c7b 100644
--- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
+++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
@@ -8596,3 +8596,73 @@ This mbus code are only used for "2-byte simplified tagged data format" (code
 embedded data format codes.
 
 Also see :ref:`CCS driver documentation <media-ccs-routes>`.
+
+Omnivision OV2740 Embedded Data Format
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The Omnivision OV2740 camera sensor produces the following embedded data format,
+indicated by mbus code MEDIA_BUS_FMT_OV2740_EMBEDDED. The format conforms to
+:ref:`CCS embedded data format <MEDIA-BUS-FMT-CCS-EMBEDDED>` up to level 1.
+
+.. flat-table:: Omnivision OV2740 Embedded Data Format. Octets at indices marked
+                reserved or unused have been omitted from the table. The values
+                are big endian byte order.
+    :header-rows: 1
+
+    * - Offset
+      - Size in bits (active bits if not the same as size)
+      - Content description
+    * - 4
+      - 16 (10--0)
+      - Analogue gain
+    * - 6
+      - 16
+      - Coarse integration time
+    * - 10
+      - 8
+      - Dpc correction threshold
+    * - 15
+      - 16
+      - Output image width
+    * - 17
+      - 16
+      - Output image height
+    * - 23
+      - 8
+      - MIPI header revision number (2)
+    * - 31
+      - 8
+      - Vertical (bit 1) and horizontal flip (bit 0)
+    * - 32
+      - 8
+      - Frame duration A
+    * - 33
+      - 8
+      - Frame duration B
+    * - 34
+      - 8
+      - Context count (2)
+    * - 35
+      - 8
+      - Context select
+    * - 54
+      - 8
+      - Data pedestal bits 9--2
+    * - 63
+      - 8
+      - Frame average bits 9--2
+    * - 64
+      - 16
+      - Digital gain red
+    * - 66
+      - 16
+      - Digital gain greenr
+    * - 68
+      - 16
+      - Digital gain blue
+    * - 70
+      - 16
+      - Digital gain greenb
+    * - 89
+      - 8
+      - Frame counter (starts at 1, wraps to 0 after 255)
diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
index 03f7e9ab517b..13e68c2ccb61 100644
--- a/include/uapi/linux/media-bus-format.h
+++ b/include/uapi/linux/media-bus-format.h
@@ -183,7 +183,8 @@
 #define MEDIA_BUS_FMT_META_20			0x8006
 #define MEDIA_BUS_FMT_META_24			0x8007
 
-/* Specific metadata formats. Next is 0x9002. */
+/* Specific metadata formats. Next is 0x9003. */
 #define MEDIA_BUS_FMT_CCS_EMBEDDED		0x9001
+#define MEDIA_BUS_FMT_OV2740_EMBEDDED		0x9002
 
 #endif /* __LINUX_MEDIA_BUS_FORMAT_H */
-- 
2.39.2

[PATCH v9 37/46] media: ov2740: Fix LINK_FREQ and PIXEL_RATE control value reporting

[Sakari Ailus]

The driver dug the supported link frequency up from the V4L2 fwnode
endpoint and used it internally, but failed to report this in the
LINK_FREQ and PIXEL_RATE controls. Fix this.

Fixes: 0677a2d9b735 ("media: ov2740: Add support for 180 MHz link frequency")
Cc: stable@vger.kernel.org # for v6.8 and later
Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ov2740.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/drivers/media/i2c/ov2740.c b/drivers/media/i2c/ov2740.c
index 552935ccb4a9..57906df7be4e 100644
--- a/drivers/media/i2c/ov2740.c
+++ b/drivers/media/i2c/ov2740.c
@@ -768,14 +768,15 @@ static int ov2740_init_controls(struct ov2740 *ov2740)
 	cur_mode = ov2740->cur_mode;
 	size = ARRAY_SIZE(link_freq_menu_items);
 
-	ov2740->link_freq = v4l2_ctrl_new_int_menu(ctrl_hdlr, &ov2740_ctrl_ops,
-						   V4L2_CID_LINK_FREQ,
-						   size - 1, 0,
-						   link_freq_menu_items);
+	ov2740->link_freq =
+		v4l2_ctrl_new_int_menu(ctrl_hdlr, &ov2740_ctrl_ops,
+				       V4L2_CID_LINK_FREQ, size - 1,
+				       ov2740->supported_modes->link_freq_index,
+				       link_freq_menu_items);
 	if (ov2740->link_freq)
 		ov2740->link_freq->flags |= V4L2_CTRL_FLAG_READ_ONLY;
 
-	pixel_rate = to_pixel_rate(OV2740_LINK_FREQ_360MHZ_INDEX);
+	pixel_rate = to_pixel_rate(ov2740->supported_modes->link_freq_index);
 	ov2740->pixel_rate = v4l2_ctrl_new_std(ctrl_hdlr, &ov2740_ctrl_ops,
 					       V4L2_CID_PIXEL_RATE, 0,
 					       pixel_rate, 1, pixel_rate);
-- 
2.39.2

[PATCH v9 38/46] media: ov2740: Remove shorthand variables

[Sakari Ailus]

Remove two variables in ov2740_init_control() that are used as a shorthand
for where the information is really located. Make the code more readable
by removing them.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ov2740.c | 18 +++++++-----------
 1 file changed, 7 insertions(+), 11 deletions(-)

diff --git a/drivers/media/i2c/ov2740.c b/drivers/media/i2c/ov2740.c
index 57906df7be4e..196a111516b0 100644
--- a/drivers/media/i2c/ov2740.c
+++ b/drivers/media/i2c/ov2740.c
@@ -754,10 +754,8 @@ static const struct v4l2_ctrl_ops ov2740_ctrl_ops = {
 static int ov2740_init_controls(struct ov2740 *ov2740)
 {
 	struct v4l2_ctrl_handler *ctrl_hdlr;
-	const struct ov2740_mode *cur_mode;
 	s64 exposure_max, h_blank, pixel_rate;
 	u32 vblank_min, vblank_max, vblank_default;
-	int size;
 	int ret;
 
 	ctrl_hdlr = &ov2740->ctrl_handler;
@@ -765,12 +763,10 @@ static int ov2740_init_controls(struct ov2740 *ov2740)
 	if (ret)
 		return ret;
 
-	cur_mode = ov2740->cur_mode;
-	size = ARRAY_SIZE(link_freq_menu_items);
-
 	ov2740->link_freq =
 		v4l2_ctrl_new_int_menu(ctrl_hdlr, &ov2740_ctrl_ops,
-				       V4L2_CID_LINK_FREQ, size - 1,
+				       V4L2_CID_LINK_FREQ,
+				       ARRAY_SIZE(link_freq_menu_items) - 1,
 				       ov2740->supported_modes->link_freq_index,
 				       link_freq_menu_items);
 	if (ov2740->link_freq)
@@ -781,14 +777,14 @@ static int ov2740_init_controls(struct ov2740 *ov2740)
 					       V4L2_CID_PIXEL_RATE, 0,
 					       pixel_rate, 1, pixel_rate);
 
-	vblank_min = cur_mode->vts_min - cur_mode->height;
-	vblank_max = cur_mode->vts_max - cur_mode->height;
-	vblank_default = cur_mode->vts_def - cur_mode->height;
+	vblank_min = ov2740->cur_mode->vts_min - ov2740->cur_mode->height;
+	vblank_max = ov2740->cur_mode->vts_max - ov2740->cur_mode->height;
+	vblank_default = ov2740->cur_mode->vts_def - ov2740->cur_mode->height;
 	ov2740->vblank = v4l2_ctrl_new_std(ctrl_hdlr, &ov2740_ctrl_ops,
 					   V4L2_CID_VBLANK, vblank_min,
 					   vblank_max, 1, vblank_default);
 
-	h_blank = cur_mode->hts - cur_mode->width;
+	h_blank = ov2740->cur_mode->hts - ov2740->cur_mode->width;
 	ov2740->hblank = v4l2_ctrl_new_std(ctrl_hdlr, &ov2740_ctrl_ops,
 					   V4L2_CID_HBLANK, h_blank, h_blank, 1,
 					   h_blank);
@@ -801,7 +797,7 @@ static int ov2740_init_controls(struct ov2740 *ov2740)
 	v4l2_ctrl_new_std(ctrl_hdlr, &ov2740_ctrl_ops, V4L2_CID_DIGITAL_GAIN,
 			  OV2740_DGTL_GAIN_MIN, OV2740_DGTL_GAIN_MAX,
 			  OV2740_DGTL_GAIN_STEP, OV2740_DGTL_GAIN_DEFAULT);
-	exposure_max = cur_mode->vts_def - OV2740_EXPOSURE_MAX_MARGIN;
+	exposure_max = ov2740->cur_mode->vts_def - OV2740_EXPOSURE_MAX_MARGIN;
 	ov2740->exposure = v4l2_ctrl_new_std(ctrl_hdlr, &ov2740_ctrl_ops,
 					     V4L2_CID_EXPOSURE,
 					     OV2740_EXPOSURE_MIN, exposure_max,
-- 
2.39.2

[PATCH v9 39/46] media: ov2740: Switch to {enable,disable}_streams

[Sakari Ailus]

Switch from s_stream to enable_streams and disable_streams callbacks.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Julien Massot <julien.massot@collabora.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 drivers/media/i2c/ov2740.c | 72 +++++++++++++++++---------------------
 1 file changed, 32 insertions(+), 40 deletions(-)

diff --git a/drivers/media/i2c/ov2740.c b/drivers/media/i2c/ov2740.c
index 196a111516b0..a12dc58fa0c0 100644
--- a/drivers/media/i2c/ov2740.c
+++ b/drivers/media/i2c/ov2740.c
@@ -916,16 +916,23 @@ static int ov2740_load_otp_data(struct nvm_data *nvm)
 	return ret;
 }
 
-static int ov2740_start_streaming(struct ov2740 *ov2740)
+static int ov2740_enable_streams(struct v4l2_subdev *sd,
+				 struct v4l2_subdev_state *state, u32 pad,
+				 u64 streams_mask)
 {
-	struct i2c_client *client = v4l2_get_subdevdata(&ov2740->sd);
+	struct i2c_client *client = v4l2_get_subdevdata(sd);
+	struct ov2740 *ov2740 = to_ov2740(sd);
 	const struct ov2740_reg_list *reg_list;
 	int link_freq_index;
 	int ret;
 
+	ret = pm_runtime_resume_and_get(&client->dev);
+	if (ret < 0)
+		return ret;
+
 	ret = ov2740_identify_module(ov2740);
 	if (ret)
-		return ret;
+		goto out_pm_put;
 
 	if (ov2740->nvm)
 		ov2740_load_otp_data(ov2740->nvm);
@@ -934,7 +941,7 @@ static int ov2740_start_streaming(struct ov2740 *ov2740)
 	ret = ov2740_write_reg(ov2740, 0x0103, 1, 0x01);
 	if (ret) {
 		dev_err(&client->dev, "failed to reset\n");
-		return ret;
+		goto out_pm_put;
 	}
 
 	usleep_range(10000, 15000);
@@ -944,64 +951,47 @@ static int ov2740_start_streaming(struct ov2740 *ov2740)
 	ret = ov2740_write_reg_list(ov2740, reg_list);
 	if (ret) {
 		dev_err(&client->dev, "failed to set plls\n");
-		return ret;
+		goto out_pm_put;
 	}
 
 	reg_list = &ov2740->cur_mode->reg_list;
 	ret = ov2740_write_reg_list(ov2740, reg_list);
 	if (ret) {
 		dev_err(&client->dev, "failed to set mode\n");
-		return ret;
+		goto out_pm_put;
 	}
 
 	ret = __v4l2_ctrl_handler_setup(ov2740->sd.ctrl_handler);
 	if (ret)
-		return ret;
+		goto out_pm_put;
 
 	ret = ov2740_write_reg(ov2740, OV2740_REG_MODE_SELECT, 1,
 			       OV2740_MODE_STREAMING);
-	if (ret)
+	if (ret) {
 		dev_err(&client->dev, "failed to start streaming\n");
+		goto out_pm_put;
+	}
 
-	return ret;
-}
+	return 0;
 
-static void ov2740_stop_streaming(struct ov2740 *ov2740)
-{
-	struct i2c_client *client = v4l2_get_subdevdata(&ov2740->sd);
+out_pm_put:
+	pm_runtime_put(&client->dev);
 
-	if (ov2740_write_reg(ov2740, OV2740_REG_MODE_SELECT, 1,
-			     OV2740_MODE_STANDBY))
-		dev_err(&client->dev, "failed to stop streaming\n");
+	return ret;
 }
 
-static int ov2740_set_stream(struct v4l2_subdev *sd, int enable)
+static int ov2740_disable_streams(struct v4l2_subdev *sd,
+				  struct v4l2_subdev_state *state, u32 pad,
+				  u64 streams_mask)
 {
-	struct ov2740 *ov2740 = to_ov2740(sd);
 	struct i2c_client *client = v4l2_get_subdevdata(sd);
-	struct v4l2_subdev_state *sd_state;
-	int ret = 0;
-
-	sd_state = v4l2_subdev_lock_and_get_active_state(&ov2740->sd);
-
-	if (enable) {
-		ret = pm_runtime_resume_and_get(&client->dev);
-		if (ret < 0)
-			goto out_unlock;
+	struct ov2740 *ov2740 = to_ov2740(sd);
+	int ret;
 
-		ret = ov2740_start_streaming(ov2740);
-		if (ret) {
-			enable = 0;
-			ov2740_stop_streaming(ov2740);
-			pm_runtime_put(&client->dev);
-		}
-	} else {
-		ov2740_stop_streaming(ov2740);
-		pm_runtime_put(&client->dev);
-	}
+	ret = ov2740_write_reg(ov2740, OV2740_REG_MODE_SELECT, 1,
+			       OV2740_MODE_STANDBY);
 
-out_unlock:
-	v4l2_subdev_unlock_state(sd_state);
+	pm_runtime_put(&client->dev);
 
 	return ret;
 }
@@ -1086,7 +1076,7 @@ static int ov2740_init_state(struct v4l2_subdev *sd,
 }
 
 static const struct v4l2_subdev_video_ops ov2740_video_ops = {
-	.s_stream = ov2740_set_stream,
+	.s_stream = v4l2_subdev_s_stream_helper,
 };
 
 static const struct v4l2_subdev_pad_ops ov2740_pad_ops = {
@@ -1094,6 +1084,8 @@ static const struct v4l2_subdev_pad_ops ov2740_pad_ops = {
 	.set_fmt = ov2740_set_format,
 	.enum_mbus_code = ov2740_enum_mbus_code,
 	.enum_frame_size = ov2740_enum_frame_size,
+	.enable_streams = ov2740_enable_streams,
+	.disable_streams = ov2740_disable_streams,
 };
 
 static const struct v4l2_subdev_ops ov2740_subdev_ops = {
-- 
2.39.2

[PATCH v9 40/46] media: ov2740: Track streaming state

[Sakari Ailus]

With enable_streams and disable_streams, the driver for a device where
streams are not independently started and stopped needs to maintain state
information on streams that have been requested to be started. Do that
now.

In the future, a helper function in the framework is a desirable way to do
this instead.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Julien Massot <julien.massot@collabora.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 drivers/media/i2c/ov2740.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/drivers/media/i2c/ov2740.c b/drivers/media/i2c/ov2740.c
index a12dc58fa0c0..c29e9b8cde15 100644
--- a/drivers/media/i2c/ov2740.c
+++ b/drivers/media/i2c/ov2740.c
@@ -539,6 +539,9 @@ struct ov2740 {
 
 	/* True if the device has been identified */
 	bool identified;
+
+	/* Track streaming state */
+	u8 streaming;
 };
 
 static inline struct ov2740 *to_ov2740(struct v4l2_subdev *subdev)
@@ -926,6 +929,11 @@ static int ov2740_enable_streams(struct v4l2_subdev *sd,
 	int link_freq_index;
 	int ret;
 
+	if (ov2740->streaming) {
+		ov2740->streaming |= streams_mask;
+		return 0;
+	}
+
 	ret = pm_runtime_resume_and_get(&client->dev);
 	if (ret < 0)
 		return ret;
@@ -972,6 +980,8 @@ static int ov2740_enable_streams(struct v4l2_subdev *sd,
 		goto out_pm_put;
 	}
 
+	ov2740->streaming |= streams_mask;
+
 	return 0;
 
 out_pm_put:
@@ -988,6 +998,10 @@ static int ov2740_disable_streams(struct v4l2_subdev *sd,
 	struct ov2740 *ov2740 = to_ov2740(sd);
 	int ret;
 
+	ov2740->streaming &= ~streams_mask;
+	if (ov2740->streaming)
+		return 0;
+
 	ret = ov2740_write_reg(ov2740, OV2740_REG_MODE_SELECT, 1,
 			       OV2740_MODE_STANDBY);
 
-- 
2.39.2

[PATCH v9 41/46] media: ov2740: Add support for embedded data

[Sakari Ailus]

Add support for embedded data. This introduces two internal pads for pixel
and embedded data streams. As the driver supports a single mode only,
there's no need for backward compatibility in mode selection.

The embedded data is configured to be placed before the image data whereas
after the image data is the default.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ov2740.c | 156 +++++++++++++++++++++++++++++++++----
 1 file changed, 141 insertions(+), 15 deletions(-)

diff --git a/drivers/media/i2c/ov2740.c b/drivers/media/i2c/ov2740.c
index c29e9b8cde15..dc0931308053 100644
--- a/drivers/media/i2c/ov2740.c
+++ b/drivers/media/i2c/ov2740.c
@@ -11,6 +11,7 @@
 #include <linux/pm_runtime.h>
 #include <linux/nvmem-provider.h>
 #include <linux/regmap.h>
+#include <media/mipi-csi2.h>
 #include <media/v4l2-ctrls.h>
 #include <media/v4l2-device.h>
 #include <media/v4l2-fwnode.h>
@@ -71,11 +72,31 @@
 #define OV2740_REG_ISP_CTRL00		0x5000
 /* ISP CTRL01 */
 #define OV2740_REG_ISP_CTRL01		0x5001
+
+/* Embedded data line location control */
+#define OV2740_REG_EMBEDDED_FLAG	0x5a08
+#define OV2740_EMBEDDED_FLAG_FOOTER	BIT(2) /* otherwise it's in header */
+#define OV2740_EMBEDDED_FLAG_MYSTERY	BIT(1)
 /* Customer Addresses: 0x7010 - 0x710F */
 #define CUSTOMER_USE_OTP_SIZE		0x100
 /* OTP registers from sensor */
 #define OV2740_REG_OTP_CUSTOMER		0x7010
 
+enum {
+	OV2740_PAD_SOURCE,
+	OV2740_PAD_PIXEL,
+	OV2740_PAD_META,
+	OV2740_NUM_PADS,
+};
+
+enum {
+	OV2740_STREAM_PIXEL,
+	OV2740_STREAM_META,
+};
+
+#define OV2740_META_WIDTH		100U
+#define OV2740_META_HEIGHT		1U
+
 struct nvm_data {
 	struct nvmem_device *nvmem;
 	struct regmap *regmap;
@@ -149,6 +170,7 @@ static const struct ov2740_reg mode_1932x1092_regs_360mhz[] = {
 	{0x3000, 0x00},
 	{0x3018, 0x32},
 	{0x3031, 0x0a},
+	{0x3036, 0x12},
 	{0x3080, 0x08},
 	{0x3083, 0xB4},
 	{0x3103, 0x00},
@@ -253,7 +275,7 @@ static const struct ov2740_reg mode_1932x1092_regs_360mhz[] = {
 	{0x4017, 0x10},
 	{0x4044, 0x02},
 	{0x4304, 0x08},
-	{0x4307, 0x30},
+	{0x4307, 0x31},
 	{0x4320, 0x80},
 	{0x4322, 0x00},
 	{0x4323, 0x00},
@@ -302,6 +324,7 @@ static const struct ov2740_reg mode_1932x1092_regs_180mhz[] = {
 	{0x3000, 0x00},
 	{0x3018, 0x32},	/* 0x32 for 2 lanes, 0x12 for 1 lane */
 	{0x3031, 0x0a},
+	{0x3036, 0x12},
 	{0x3080, 0x08},
 	{0x3083, 0xB4},
 	{0x3103, 0x00},
@@ -406,7 +429,7 @@ static const struct ov2740_reg mode_1932x1092_regs_180mhz[] = {
 	{0x4017, 0x10},
 	{0x4044, 0x02},
 	{0x4304, 0x08},
-	{0x4307, 0x30},
+	{0x4307, 0x31},
 	{0x4320, 0x80},
 	{0x4322, 0x00},
 	{0x4323, 0x00},
@@ -513,7 +536,7 @@ static const struct ov2740_mode supported_modes_180mhz[] = {
 
 struct ov2740 {
 	struct v4l2_subdev sd;
-	struct media_pad pad;
+	struct media_pad pads[OV2740_NUM_PADS];
 	struct v4l2_ctrl_handler ctrl_handler;
 
 	/* V4L2 Controls */
@@ -973,6 +996,11 @@ static int ov2740_enable_streams(struct v4l2_subdev *sd,
 	if (ret)
 		goto out_pm_put;
 
+	ret = ov2740_write_reg(ov2740, OV2740_REG_EMBEDDED_FLAG, 1,
+			       OV2740_EMBEDDED_FLAG_MYSTERY);
+	if (ret)
+		goto out_pm_put;
+
 	ret = ov2740_write_reg(ov2740, OV2740_REG_MODE_SELECT, 1,
 			       OV2740_MODE_STREAMING);
 	if (ret) {
@@ -1010,23 +1038,49 @@ static int ov2740_disable_streams(struct v4l2_subdev *sd,
 	return ret;
 }
 
-static int ov2740_set_format(struct v4l2_subdev *sd,
-			     struct v4l2_subdev_state *sd_state,
-			     struct v4l2_subdev_format *fmt)
+static int __ov2740_set_format(struct v4l2_subdev *sd,
+			       struct v4l2_subdev_state *sd_state,
+			       struct v4l2_mbus_framefmt *format,
+			       enum v4l2_subdev_format_whence which,
+			       unsigned int pad, unsigned int stream)
 {
+	struct v4l2_mbus_framefmt *src_pix_fmt, *src_meta_fmt, *pix_fmt,
+		*meta_fmt;
 	struct ov2740 *ov2740 = to_ov2740(sd);
 	const struct ov2740_mode *mode;
 	s32 vblank_def, h_blank;
 
+	/*
+	 * Allow setting format on internal pixel pad as well as the source
+	 * pad's pixel stream (for compatibility).
+	 */
+	if ((pad == OV2740_PAD_SOURCE && stream == OV2740_STREAM_META) ||
+	    pad == OV2740_PAD_META) {
+		*format = *v4l2_subdev_state_get_format(sd_state, pad, stream);
+		return 0;
+	}
+
+	pix_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_PIXEL, 0);
+	meta_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_META, 0);
+	src_pix_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_SOURCE,
+						   OV2740_STREAM_PIXEL);
+	src_meta_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_SOURCE,
+						    OV2740_STREAM_META);
+
 	mode = v4l2_find_nearest_size(ov2740->supported_modes,
 				      ov2740->supported_modes_count,
 				      width, height,
-				      fmt->format.width, fmt->format.height);
+				      format->width, format->height);
+	ov2740_update_pad_format(mode, pix_fmt);
+	*format = *src_pix_fmt = *pix_fmt;
 
-	ov2740_update_pad_format(mode, &fmt->format);
-	*v4l2_subdev_state_get_format(sd_state, fmt->pad) = fmt->format;
+	meta_fmt->code = MEDIA_BUS_FMT_OV2740_EMBEDDED;
+	meta_fmt->width = OV2740_META_WIDTH;
+	meta_fmt->height = OV2740_META_HEIGHT;
+	*src_meta_fmt = *meta_fmt;
+	src_meta_fmt->code = MEDIA_BUS_FMT_META_10;
 
-	if (fmt->which == V4L2_SUBDEV_FORMAT_TRY)
+	if (which == V4L2_SUBDEV_FORMAT_TRY)
 		return 0;
 
 	ov2740->cur_mode = mode;
@@ -1046,6 +1100,14 @@ static int ov2740_set_format(struct v4l2_subdev *sd,
 	return 0;
 }
 
+static int ov2740_set_format(struct v4l2_subdev *sd,
+			     struct v4l2_subdev_state *sd_state,
+			     struct v4l2_subdev_format *fmt)
+{
+	return __ov2740_set_format(sd, sd_state, &fmt->format, fmt->which,
+				   fmt->pad, fmt->stream);
+}
+
 static int ov2740_enum_mbus_code(struct v4l2_subdev *sd,
 				 struct v4l2_subdev_state *sd_state,
 				 struct v4l2_subdev_mbus_code_enum *code)
@@ -1082,10 +1144,68 @@ static int ov2740_enum_frame_size(struct v4l2_subdev *sd,
 static int ov2740_init_state(struct v4l2_subdev *sd,
 			     struct v4l2_subdev_state *sd_state)
 {
+	struct v4l2_subdev_route routes[] = {
+		{
+			.sink_pad = OV2740_PAD_PIXEL,
+			.source_pad = OV2740_PAD_SOURCE,
+			.source_stream = OV2740_STREAM_PIXEL,
+			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
+		}, {
+			.sink_pad = OV2740_PAD_META,
+			.source_pad = OV2740_PAD_SOURCE,
+			.source_stream = OV2740_STREAM_META,
+			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
+		},
+	};
+	struct v4l2_subdev_krouting routing = {
+		.routes = routes,
+		.num_routes = ARRAY_SIZE(routes),
+	};
+	struct v4l2_subdev_state *active_state;
+	struct v4l2_mbus_framefmt format = { 0 };
 	struct ov2740 *ov2740 = to_ov2740(sd);
+	int ret;
+
+	ret = v4l2_subdev_set_routing(sd, sd_state, &routing);
+	if (ret)
+		return ret;
+
+	active_state = v4l2_subdev_get_locked_active_state(sd);
+
+	ov2740_update_pad_format(&ov2740->supported_modes[0], &format);
+
+	return __ov2740_set_format(sd, sd_state, &format,
+				   active_state == sd_state ?
+				   V4L2_SUBDEV_FORMAT_ACTIVE :
+				   V4L2_SUBDEV_FORMAT_TRY, OV2740_PAD_PIXEL, 0);
+}
+
+static int ov2740_get_frame_desc(struct v4l2_subdev *sd, unsigned int pad,
+				 struct v4l2_mbus_frame_desc *desc)
+{
+	struct v4l2_mbus_frame_desc_entry *entry = desc->entry;
+	struct v4l2_subdev_state *sd_state;
+	struct v4l2_mbus_framefmt *fmt;
+
+	desc->type = V4L2_MBUS_FRAME_DESC_TYPE_CSI2;
+
+	sd_state = v4l2_subdev_lock_and_get_active_state(sd);
+	fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_SOURCE,
+					   OV2740_STREAM_PIXEL);
+	entry->pixelcode = fmt->code;
+	v4l2_subdev_unlock_state(sd_state);
+
+	entry->stream = OV2740_STREAM_PIXEL;
+	entry->bus.csi2.dt = MIPI_CSI2_DT_RAW10;
+	entry++;
+	desc->num_entries++;
+
+	entry->pixelcode = MEDIA_BUS_FMT_META_10;
+	entry->stream = OV2740_STREAM_META;
+	entry->bus.csi2.dt = MIPI_CSI2_DT_EMBEDDED_8B;
+	entry++;
+	desc->num_entries++;
 
-	ov2740_update_pad_format(&ov2740->supported_modes[0],
-				 v4l2_subdev_state_get_format(sd_state, 0));
 	return 0;
 }
 
@@ -1100,6 +1220,7 @@ static const struct v4l2_subdev_pad_ops ov2740_pad_ops = {
 	.enum_frame_size = ov2740_enum_frame_size,
 	.enable_streams = ov2740_enable_streams,
 	.disable_streams = ov2740_disable_streams,
+	.get_frame_desc = ov2740_get_frame_desc,
 };
 
 static const struct v4l2_subdev_ops ov2740_subdev_ops = {
@@ -1366,11 +1487,16 @@ static int ov2740_probe(struct i2c_client *client)
 	}
 
 	ov2740->sd.state_lock = ov2740->ctrl_handler.lock;
-	ov2740->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
+	ov2740->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_STREAMS;
 	ov2740->sd.entity.ops = &ov2740_subdev_entity_ops;
 	ov2740->sd.entity.function = MEDIA_ENT_F_CAM_SENSOR;
-	ov2740->pad.flags = MEDIA_PAD_FL_SOURCE;
-	ret = media_entity_pads_init(&ov2740->sd.entity, 1, &ov2740->pad);
+	ov2740->pads[OV2740_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
+	ov2740->pads[OV2740_PAD_PIXEL].flags =
+		MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL;
+	ov2740->pads[OV2740_PAD_META].flags =
+		MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL;
+	ret = media_entity_pads_init(&ov2740->sd.entity,
+				     ARRAY_SIZE(ov2740->pads), ov2740->pads);
 	if (ret) {
 		dev_err_probe(dev, ret, "failed to init entity pads\n");
 		goto probe_error_v4l2_ctrl_handler_free;
-- 
2.39.2

[PATCH v9 42/46] media: ov2740: Add generic sensor fwnode properties as controls

[Sakari Ailus]

Add generic sensor property information as controĺs by using
v4l2_fwnode_device_parse() and v4l2_ctrl_new_fwnode_properties().

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ov2740.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/drivers/media/i2c/ov2740.c b/drivers/media/i2c/ov2740.c
index dc0931308053..e37d824291fe 100644
--- a/drivers/media/i2c/ov2740.c
+++ b/drivers/media/i2c/ov2740.c
@@ -779,6 +779,8 @@ static const struct v4l2_ctrl_ops ov2740_ctrl_ops = {
 
 static int ov2740_init_controls(struct ov2740 *ov2740)
 {
+	struct i2c_client *client = v4l2_get_subdevdata(&ov2740->sd);
+	struct v4l2_fwnode_device_properties props;
 	struct v4l2_ctrl_handler *ctrl_hdlr;
 	s64 exposure_max, h_blank, pixel_rate;
 	u32 vblank_min, vblank_max, vblank_default;
@@ -789,6 +791,10 @@ static int ov2740_init_controls(struct ov2740 *ov2740)
 	if (ret)
 		return ret;
 
+	if (!v4l2_fwnode_device_parse(&client->dev, &props))
+		v4l2_ctrl_new_fwnode_properties(ctrl_hdlr, &ov2740_ctrl_ops,
+						&props);
+
 	ov2740->link_freq =
 		v4l2_ctrl_new_int_menu(ctrl_hdlr, &ov2740_ctrl_ops,
 				       V4L2_CID_LINK_FREQ,
-- 
2.39.2

[PATCH v9 43/46] media: ov2740: Add support for G_SELECTION IOCTL

[Sakari Ailus]

Add support for the G_SELECTION IOCTL in the ov2740 driver.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/i2c/ov2740.c | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)

diff --git a/drivers/media/i2c/ov2740.c b/drivers/media/i2c/ov2740.c
index e37d824291fe..6e355e986b88 100644
--- a/drivers/media/i2c/ov2740.c
+++ b/drivers/media/i2c/ov2740.c
@@ -23,6 +23,11 @@
 #define OV2740_DATA_LANES		2
 #define OV2740_RGB_DEPTH		10
 
+#define OV2740_DUMMY_LINES_PRE		8U
+#define OV2740_DUMMY_LINES_POST		8U
+#define OV2740_ACTIVE_WIDTH		1932U
+#define OV2740_ACTIVE_HEIGHT		1092U
+
 #define OV2740_REG_CHIP_ID		0x300a
 #define OV2740_CHIP_ID			0x2740
 
@@ -1114,6 +1119,31 @@ static int ov2740_set_format(struct v4l2_subdev *sd,
 				   fmt->pad, fmt->stream);
 }
 
+static int ov2740_get_selection(struct v4l2_subdev *sd,
+				struct v4l2_subdev_state *state,
+				struct v4l2_subdev_selection *sel)
+{
+	switch (sel->target) {
+	case V4L2_SEL_TGT_NATIVE_SIZE:
+	case V4L2_SEL_TGT_CROP_BOUNDS:
+		sel->r.top = 0;
+		sel->r.left = 0;
+		sel->r.width = OV2740_ACTIVE_WIDTH;
+		sel->r.height = OV2740_DUMMY_LINES_PRE + OV2740_ACTIVE_HEIGHT +
+			OV2740_DUMMY_LINES_POST;
+		return 0;
+	case V4L2_SEL_TGT_CROP:
+	case V4L2_SEL_TGT_CROP_DEFAULT:
+		sel->r.top = OV2740_DUMMY_LINES_PRE;
+		sel->r.left = 0;
+		sel->r.width = OV2740_ACTIVE_WIDTH;
+		sel->r.height = OV2740_ACTIVE_HEIGHT;
+		return 0;
+	}
+
+	return -EINVAL;
+}
+
 static int ov2740_enum_mbus_code(struct v4l2_subdev *sd,
 				 struct v4l2_subdev_state *sd_state,
 				 struct v4l2_subdev_mbus_code_enum *code)
@@ -1222,6 +1252,7 @@ static const struct v4l2_subdev_video_ops ov2740_video_ops = {
 static const struct v4l2_subdev_pad_ops ov2740_pad_ops = {
 	.get_fmt = v4l2_subdev_get_fmt,
 	.set_fmt = ov2740_set_format,
+	.get_selection = ov2740_get_selection,
 	.enum_mbus_code = ov2740_enum_mbus_code,
 	.enum_frame_size = ov2740_enum_frame_size,
 	.enable_streams = ov2740_enable_streams,
-- 
2.39.2

[PATCH v9 44/46] media: v4l: Add V4L2_SUBDEV_ROUTE_FL_IMMUTABLE sub-device routing flag

[Sakari Ailus]

Add a flag to denote immutable routes, V4L2_SUBDEV_ROUTE_FL_IMMUTABLE.
Such routes cannot be changed and they're always active.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 Documentation/userspace-api/media/v4l/dev-subdev.rst         | 4 +++-
 .../userspace-api/media/v4l/vidioc-subdev-g-routing.rst      | 5 +++++
 include/uapi/linux/v4l2-subdev.h                             | 5 +++++
 3 files changed, 13 insertions(+), 1 deletion(-)

diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index de8dfd4f11a5..5c63c8c24108 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -572,7 +572,9 @@ internal pad always has a single stream only (0).
 Routes from an internal sink pad to an external source pad are typically not
 modifiable but they can be activated and deactivated using the
 :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE <v4l2-subdev-routing-flags>` flag, depending
-on driver capabilities.
+on driver capabilities. The :ref:`V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
+<v4l2-subdev-routing-flags>` flag indicates that the
+``V4L2_SUBDEV_ROUTE_FLAG_ACTIVE`` of the route may not be unset.
 
 Interaction between routes, streams, formats and selections
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
index 88df7bf80b00..c0f6d49cac6d 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
@@ -150,6 +150,11 @@ Also ``VIDIOC_SUBDEV_S_ROUTING`` may return more route than the user provided in
     * - V4L2_SUBDEV_ROUTE_FL_ACTIVE
       - 0x0001
       - The route is enabled. Set by applications.
+    * - V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
+      - 0x0002
+      - The route is immutable. Set by the driver. Indicates that the
+	``V4L2_SUBDEV_ROUTE_FL_ACTIVE`` flag of an immutable route may not be
+	unset.
 
 Return Value
 ============
diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
index 6a39128d0606..c6f1228d43b1 100644
--- a/include/uapi/linux/v4l2-subdev.h
+++ b/include/uapi/linux/v4l2-subdev.h
@@ -204,6 +204,11 @@ struct v4l2_subdev_capability {
  * on a video node.
  */
 #define V4L2_SUBDEV_ROUTE_FL_ACTIVE		(1U << 0)
+/*
+ * Is the route immutable. The ACTIVE flag of an immutable route may not be
+ * changed.
+ */
+#define V4L2_SUBDEV_ROUTE_FL_IMMUTABLE		(1U << 1)
 
 /**
  * struct v4l2_subdev_route - A route inside a subdev
-- 
2.39.2

[PATCH v9 45/46] media: ccs: Add IMMUTABLE route flag

[Sakari Ailus]

Add immutable route flag to the routing table. The embedded data stream is
always there.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Julien Massot <julien.massot@collabora.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 drivers/media/i2c/ccs/ccs-core.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
index 0f68e64a4042..785e5f2dc107 100644
--- a/drivers/media/i2c/ccs/ccs-core.c
+++ b/drivers/media/i2c/ccs/ccs-core.c
@@ -3345,12 +3345,14 @@ static int ccs_src_init_state(struct v4l2_subdev *sd,
 			.sink_pad = CCS_PAD_SINK,
 			.source_pad = CCS_PAD_SRC,
 			.source_stream = CCS_STREAM_PIXEL,
-			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
+			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE |
+				 V4L2_SUBDEV_ROUTE_FL_IMMUTABLE,
 		}, {
 			.sink_pad = CCS_PAD_META,
 			.source_pad = CCS_PAD_SRC,
 			.source_stream = CCS_STREAM_META,
-			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
+			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE |
+				 V4L2_SUBDEV_ROUTE_FL_IMMUTABLE,
 		}
 	};
 	struct v4l2_subdev_krouting routing = {
-- 
2.39.2

[PATCH v9 46/46] media: ov2740: Add IMMUTABLE route flag

[Sakari Ailus]

Add immutable route flag to the routing table. The driver does not support
disabling the embedded data whereas the sensor itself does.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Julien Massot <julien.massot@collabora.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 drivers/media/i2c/ov2740.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/drivers/media/i2c/ov2740.c b/drivers/media/i2c/ov2740.c
index 6e355e986b88..d6b4533a834a 100644
--- a/drivers/media/i2c/ov2740.c
+++ b/drivers/media/i2c/ov2740.c
@@ -1185,12 +1185,14 @@ static int ov2740_init_state(struct v4l2_subdev *sd,
 			.sink_pad = OV2740_PAD_PIXEL,
 			.source_pad = OV2740_PAD_SOURCE,
 			.source_stream = OV2740_STREAM_PIXEL,
-			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
+			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE |
+				 V4L2_SUBDEV_ROUTE_FL_IMMUTABLE,
 		}, {
 			.sink_pad = OV2740_PAD_META,
 			.source_pad = OV2740_PAD_SOURCE,
 			.source_stream = OV2740_STREAM_META,
-			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
+			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE |
+				 V4L2_SUBDEV_ROUTE_FL_IMMUTABLE,
 		},
 	};
 	struct v4l2_subdev_krouting routing = {
-- 
2.39.2

Re: [PATCH v9 03/46] media: Documentation: Add "stream" into glossary

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Tue, Apr 16, 2024 at 10:32:36PM +0300, Sakari Ailus wrote:
> Add term "stream" to the glossary of the Media subsystem documentation.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> Reviewed-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> ---
>  Documentation/userspace-api/media/glossary.rst | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/Documentation/userspace-api/media/glossary.rst b/Documentation/userspace-api/media/glossary.rst
> index 96a360edbf3b..ef0ab601b5bf 100644
> --- a/Documentation/userspace-api/media/glossary.rst
> +++ b/Documentation/userspace-api/media/glossary.rst
> @@ -173,6 +173,12 @@ Glossary
>  	An integrated circuit that integrates all components of a computer
>  	or other electronic systems.
>  
> +_media-glossary-stream:
> +    Stream
> +	A distinct flow of data (image data or metadata) from an initial source
> +	to a final sink. The initial source may be e.g. an image sensor and the
> +	final sink e.g. a memory buffer.
> +
>      V4L2 API
>  	**V4L2 userspace API**
>  
> -- 
> 2.39.2
> 

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 05/46] media: uapi: Document which mbus format fields are valid for metadata

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Tue, Apr 16, 2024 at 10:32:38PM +0300, Sakari Ailus wrote:
> Now that metadata mbus formats have been added, it is necessary to define
> which fields in struct v4l2_mbus_format are applicable to them (not many).
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> Reviewed-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  .../userspace-api/media/v4l/subdev-formats.rst | 15 ++++++++-------
>  include/uapi/linux/v4l2-mediabus.h             | 18 ++++++++++++------
>  2 files changed, 20 insertions(+), 13 deletions(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> index d9a5ee954cdd..0547f2733ee3 100644
> --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
> +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> @@ -33,7 +33,7 @@ Media Bus Formats
>      * - __u32
>        - ``field``
>        - Field order, from enum :c:type:`v4l2_field`. See
> -	:ref:`field-order` for details.
> +	:ref:`field-order` for details. Zero on metadata mbus codes.

I would write "Zero for" instead of "Zero on".

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

>      * - __u32
>        - ``colorspace``
>        - Image colorspace, from enum :c:type:`v4l2_colorspace`.
> @@ -45,7 +45,7 @@ Media Bus Formats
>  	conversion is supported by setting the flag
>  	V4L2_SUBDEV_MBUS_CODE_CSC_COLORSPACE in the corresponding struct
>  	:c:type:`v4l2_subdev_mbus_code_enum` during enumeration.
> -	See :ref:`v4l2-subdev-mbus-code-flags`.
> +	See :ref:`v4l2-subdev-mbus-code-flags`. Zero on metadata mbus codes.
>      * - union {
>        - (anonymous)
>      * - __u16
> @@ -61,7 +61,7 @@ Media Bus Formats
>  	that ycbcr_enc conversion is supported by setting the flag
>  	V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC in the corresponding struct
>  	:c:type:`v4l2_subdev_mbus_code_enum` during enumeration.
> -	See :ref:`v4l2-subdev-mbus-code-flags`.
> +	See :ref:`v4l2-subdev-mbus-code-flags`. Zero on metadata mbus codes.
>      * - __u16
>        - ``hsv_enc``
>        - HSV encoding, from enum :c:type:`v4l2_hsv_encoding`.
> @@ -75,7 +75,7 @@ Media Bus Formats
>  	that hsv_enc conversion is supported by setting the flag
>  	V4L2_SUBDEV_MBUS_CODE_CSC_HSV_ENC in the corresponding struct
>  	:c:type:`v4l2_subdev_mbus_code_enum` during enumeration.
> -	See :ref:`v4l2-subdev-mbus-code-flags`
> +	See :ref:`v4l2-subdev-mbus-code-flags`. Zero on metadata mbus codes.
>      * - }
>        -
>      * - __u16
> @@ -90,8 +90,8 @@ Media Bus Formats
>  	The driver indicates that quantization conversion is supported by
>  	setting the flag V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION in the
>  	corresponding struct :c:type:`v4l2_subdev_mbus_code_enum`
> -	during enumeration. See :ref:`v4l2-subdev-mbus-code-flags`.
> -
> +	during enumeration. See :ref:`v4l2-subdev-mbus-code-flags`. Zero on
> +	metadata mbus codes.
>      * - __u16
>        - ``xfer_func``
>        - Transfer function, from enum :c:type:`v4l2_xfer_func`.
> @@ -104,7 +104,8 @@ Media Bus Formats
>  	The driver indicates that the transfer function conversion is supported by
>  	setting the flag V4L2_SUBDEV_MBUS_CODE_CSC_XFER_FUNC in the
>  	corresponding struct :c:type:`v4l2_subdev_mbus_code_enum`
> -	during enumeration. See :ref:`v4l2-subdev-mbus-code-flags`.
> +	during enumeration. See :ref:`v4l2-subdev-mbus-code-flags`. Zero on
> +	metadata mbus codes.
>      * - __u16
>        - ``flags``
>        - flags See:  :ref:v4l2-mbus-framefmt-flags
> diff --git a/include/uapi/linux/v4l2-mediabus.h b/include/uapi/linux/v4l2-mediabus.h
> index 6b07b73473b5..de1d6161bf62 100644
> --- a/include/uapi/linux/v4l2-mediabus.h
> +++ b/include/uapi/linux/v4l2-mediabus.h
> @@ -19,12 +19,18 @@
>   * @width:	image width
>   * @height:	image height
>   * @code:	data format code (from enum v4l2_mbus_pixelcode)
> - * @field:	used interlacing type (from enum v4l2_field)
> - * @colorspace:	colorspace of the data (from enum v4l2_colorspace)
> - * @ycbcr_enc:	YCbCr encoding of the data (from enum v4l2_ycbcr_encoding)
> - * @hsv_enc:	HSV encoding of the data (from enum v4l2_hsv_encoding)
> - * @quantization: quantization of the data (from enum v4l2_quantization)
> - * @xfer_func:  transfer function of the data (from enum v4l2_xfer_func)
> + * @field:	used interlacing type (from enum v4l2_field), zero on metadata
> + *		mbus codes
> + * @colorspace:	colorspace of the data (from enum v4l2_colorspace), zero on
> + *		metadata mbus codes
> + * @ycbcr_enc:	YCbCr encoding of the data (from enum v4l2_ycbcr_encoding), zero
> + *		on metadata mbus codes
> + * @hsv_enc:	HSV encoding of the data (from enum v4l2_hsv_encoding), zero on
> + *		metadata mbus codes
> + * @quantization: quantization of the data (from enum v4l2_quantization), zero
> + *		on metadata mbus codes
> + * @xfer_func:  transfer function of the data (from enum v4l2_xfer_func), zero
> + *		on metadata mbus codes
>   * @flags:	flags (V4L2_MBUS_FRAMEFMT_*)
>   * @reserved:  reserved bytes that can be later used
>   */

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 06/46] media: uapi: v4l: Add generic 8-bit metadata format definitions

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Tue, Apr 16, 2024 at 10:32:39PM +0300, Sakari Ailus wrote:
> Generic 8-bit metadata formats define the in-memory data layout but not
> the format of the data itself. The reasoning for having such formats is to
> allow CSI-2 receiver drivers to receive and DMA drivers to write the data
> to memory without knowing a large number of device-specific formats.
> 
> These formats may be used only in conjunction with a Media controller

"Media Controller" or "media controller"

> pipeline where the internal pad of the source sub-device defines the
> specific format of the data (using an mbus code).
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../userspace-api/media/v4l/dev-subdev.rst    |   2 +
>  .../userspace-api/media/v4l/meta-formats.rst  |   3 +-
>  .../media/v4l/metafmt-generic.rst             | 328 ++++++++++++++++++
>  drivers/media/v4l2-core/v4l2-ioctl.c          |   7 +
>  include/uapi/linux/videodev2.h                |   8 +
>  5 files changed, 347 insertions(+), 1 deletion(-)
>  create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
> 
> diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> index 43988516acdd..f375b820ab68 100644
> --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> @@ -506,6 +506,8 @@ source pads.
>  
>      subdev-formats
>  
> +.. _subdev-routing:
> +
>  Streams, multiplexed media pads and internal routing
>  ----------------------------------------------------
>  
> diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
> index 0bb61fc5bc00..c23aac823d2c 100644
> --- a/Documentation/userspace-api/media/v4l/meta-formats.rst
> +++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
> @@ -13,9 +13,10 @@ These formats are used for the :ref:`metadata` interface only.
>      :maxdepth: 1
>  
>      metafmt-d4xx
> +    metafmt-generic
>      metafmt-intel-ipu3
>      metafmt-rkisp1
>      metafmt-uvc
> +    metafmt-vivid
>      metafmt-vsp1-hgo
>      metafmt-vsp1-hgt
> -    metafmt-vivid
> diff --git a/Documentation/userspace-api/media/v4l/metafmt-generic.rst b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> new file mode 100644
> index 000000000000..cba34be64dfe
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> @@ -0,0 +1,328 @@
> +.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
> +
> +********************************************************************************************************************************************************************************************************************************************************************************
> +V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O')
> +********************************************************************************************************************************************************************************************************************************************************************************
> +
> +
> +Generic line-based metadata formats
> +
> +
> +Description
> +===========
> +
> +These generic line-based metadata formats define the memory layout of the data
> +without defining the format or meaning of the metadata itself. These formats may
> +only be used with a Media Controller pipeline where the more specific format is
> +reported by an :ref:`internal sink pad <MEDIA-PAD-FL-INTERNAL>` of the source
> +sub-device. See also :ref:`source routes <subdev-routing>`.
> +
> +.. _v4l2-meta-fmt-generic-8:
> +
> +V4L2_META_FMT_GENERIC_8
> +-----------------------
> +
> +The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format.
> +
> +This format is also used on CSI-2 for both 8 bits per ``Data Unit
> +<media-glossary-data-unit>`` as well as for 16 bits per Data Unit when two bytes
> +of metadata are packed into one 16-bit Data Unit.
> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_8.**
> +Each cell is one byte. "M" denotes a byte of metadata.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|
> +
> +.. flat-table:: Sample 4x2 Metadata Frame
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - M\ :sub:`10`
> +      - M\ :sub:`20`
> +      - M\ :sub:`30`
> +    * - start + 4:
> +      - M\ :sub:`01`
> +      - M\ :sub:`11`
> +      - M\ :sub:`21`
> +      - M\ :sub:`31`
> +
> +.. _v4l2-meta-fmt-generic-csi2-10:
> +
> +V4L2_META_FMT_GENERIC_CSI2_10
> +-----------------------------
> +
> +V4L2_META_FMT_GENERIC_CSI2_10 contains 8-bit generic metadata packed in 10-bit
> +Data Units, with one padding byte after every four bytes of metadata. This
> +format is typically used by CSI-2 receivers with a source that transmits
> +MEDIA_BUS_FMT_META_10 and the CSI-2 receiver writes the received data to memory
> +as-is.
> +
> +The packing of the data follows the MIPI CSI-2 specification and the padding of
> +the data is defined in the MIPI CCS specification.
> +
> +This format is also used in conjunction with 20 bits per ``Data Unit

s/in conjunction with/for/

Same below.

> +<media-glossary-data-unit>`` formats that pack two bytes of metadata into one
> +Data Unit.

How about adding a mention of MEDIA_BUS_FMT_META_20 here ? The CCS
specification calls this "RAW20 (with more optimal RAW10 packing)",
using the same vocabulary could help. Same for
V4L2_META_FMT_GENERIC_CSI2_12, and actually V4L2_META_FMT_GENERIC_8 too.

> +
> +This format is little endian.
> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
> +Each cell is one byte. "M" denotes a byte of metadata and "x" a byte of padding.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
> +
> +.. flat-table:: Sample 4x2 Metadata Frame
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - M\ :sub:`10`
> +      - M\ :sub:`20`
> +      - M\ :sub:`30`
> +      - x
> +    * - start + 5:
> +      - M\ :sub:`01`
> +      - M\ :sub:`11`
> +      - M\ :sub:`21`
> +      - M\ :sub:`31`
> +      - x
> +
> +.. _v4l2-meta-fmt-generic-csi2-12:
> +
> +V4L2_META_FMT_GENERIC_CSI2_12
> +-----------------------------
> +
> +V4L2_META_FMT_GENERIC_CSI2_12 contains 8-bit generic metadata packed in 12-bit
> +Data Units, with two padding bytes after every four bytes of metadata. This

Isn't it one padding byte every two bytes of metadata ?

> +format is typically used by CSI-2 receivers with a source that transmits
> +MEDIA_BUS_FMT_META_12 and the CSI-2 receiver writes the received data to memory
> +as-is.
> +
> +The packing of the data follows the MIPI CSI-2 specification and the padding of
> +the data is defined in the MIPI CCS specification.
> +
> +This format is also used in conjunction with 24 bits per ``Data Unit
> +<media-glossary-data-unit>`` formats that pack two bytes of metadata into one
> +Data Unit.
> +
> +This format is little endian.
> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_12.**
> +Each cell is one byte. "M" denotes a byte of metadata and "x" a byte of padding.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> +
> +.. flat-table:: Sample 4x2 Metadata Frame
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - M\ :sub:`10`
> +      - x
> +      - M\ :sub:`20`
> +      - M\ :sub:`30`
> +      - x
> +    * - start + 6:
> +      - M\ :sub:`01`
> +      - M\ :sub:`11`
> +      - x
> +      - M\ :sub:`21`
> +      - M\ :sub:`31`
> +      - x
> +
> +.. _v4l2-meta-fmt-generic-csi2-14:
> +
> +V4L2_META_FMT_GENERIC_CSI2_14
> +-----------------------------
> +
> +V4L2_META_FMT_GENERIC_CSI2_14 contains 8-bit generic metadata packed in 14-bit
> +Data Units, with three padding bytes after every four bytes of metadata. This
> +format is typically used by CSI-2 receivers with a source that transmits
> +MEDIA_BUS_FMT_META_14 and the CSI-2 receiver writes the received data to memory
> +as-is.
> +
> +The packing of the data follows the MIPI CSI-2 specification and the padding of
> +the data is defined in the MIPI CCS specification.
> +
> +This format is little endian.
> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_14.**
> +Each cell is one byte. "M" denotes a byte of metadata and "x" a byte of padding.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{.8cm}|
> +
> +.. flat-table:: Sample 4x2 Metadata Frame
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - M\ :sub:`10`
> +      - M\ :sub:`20`
> +      - M\ :sub:`30`
> +      - x
> +      - x
> +      - x
> +    * - start + 7:
> +      - M\ :sub:`01`
> +      - M\ :sub:`11`
> +      - M\ :sub:`21`
> +      - M\ :sub:`31`
> +      - x
> +      - x
> +      - x
> +
> +.. _v4l2-meta-fmt-generic-csi2-16:
> +
> +V4L2_META_FMT_GENERIC_CSI2_16
> +-----------------------------
> +
> +V4L2_META_FMT_GENERIC_CSI2_16 contains 8-bit generic metadata packed in 16-bit
> +Data Units, with one padding byte after every byte of metadata. This format is
> +typically used by CSI-2 receivers with a source that transmits
> +MEDIA_BUS_FMT_META_16 and the CSI-2 receiver writes the received data to memory
> +as-is.
> +
> +The packing of the data follows the MIPI CSI-2 specification and the padding of
> +the data is defined in the MIPI CCS specification.

Here you could also mention that more optimal packing schemes for
MEDIA_BUS_FMT_META_16 use V4L2_META_FMT_GENERIC_8. Same below.

> +
> +This format is little endian.
> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_16.**
> +Each cell is one byte. "M" denotes a byte of metadata and "x" a byte of padding.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|
> +
> +.. flat-table:: Sample 4x2 Metadata Frame
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - x
> +      - M\ :sub:`10`
> +      - x
> +      - M\ :sub:`20`
> +      - x
> +      - M\ :sub:`30`
> +      - x
> +    * - start + 8:
> +      - M\ :sub:`01`
> +      - x
> +      - M\ :sub:`11`
> +      - x
> +      - M\ :sub:`21`
> +      - x
> +      - M\ :sub:`31`
> +      - x
> +
> +.. _v4l2-meta-fmt-generic-csi2-20:
> +
> +V4L2_META_FMT_GENERIC_CSI2_20
> +-----------------------------
> +
> +V4L2_META_FMT_GENERIC_CSI2_20 contains 8-bit generic metadata packed in 20-bit
> +Data Units, with alternating one or two padding bytes after every byte of
> +metadata. This format is typically used by CSI-2 receivers with a source that
> +transmits MEDIA_BUS_FMT_META_20 and the CSI-2 receiver writes the received data
> +to memory as-is.
> +
> +The packing of the data follows the MIPI CSI-2 specification and the padding of
> +the data is defined in the MIPI CCS specification.
> +
> +This format is little endian.
> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_20.**
> +Each cell is one byte. "M" denotes a byte of metadata and "x" a byte of padding.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> +
> +.. flat-table:: Sample 4x2 Metadata Frame
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8 8 8 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - x
> +      - M\ :sub:`10`
> +      - x
> +      - x
> +      - M\ :sub:`20`
> +      - x
> +      - M\ :sub:`30`
> +      - x
> +      - x
> +    * - start + 10:
> +      - M\ :sub:`01`
> +      - x
> +      - M\ :sub:`11`
> +      - x
> +      - x
> +      - M\ :sub:`21`
> +      - x
> +      - M\ :sub:`31`
> +      - x
> +      - x
> +
> +.. _v4l2-meta-fmt-generic-csi2-24:
> +
> +V4L2_META_FMT_GENERIC_CSI2_24
> +-----------------------------
> +
> +V4L2_META_FMT_GENERIC_CSI2_24 contains 8-bit generic metadata packed in 24-bit
> +Data Units, with two padding bytes after every byte of metadata. This format is
> +typically used by CSI-2 receivers with a source that transmits
> +MEDIA_BUS_FMT_META_24 and the CSI-2 receiver writes the received data to memory
> +as-is.
> +
> +The packing of the data follows the MIPI CSI-2 specification and the padding of
> +the data is defined in the MIPI CCS specification.
> +
> +This format is little endian.
> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_24.**
> +Each cell is one byte. "M" denotes a byte of metadata and "x" a byte of padding.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> +
> +.. flat-table:: Sample 4x2 Metadata Frame
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8 8 8 8 8 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - x
> +      - x
> +      - M\ :sub:`10`
> +      - x
> +      - x
> +      - M\ :sub:`20`
> +      - x
> +      - x
> +      - M\ :sub:`30`
> +      - x
> +      - x
> +    * - start + 12:
> +      - M\ :sub:`01`
> +      - x
> +      - x
> +      - M\ :sub:`11`
> +      - x
> +      - x
> +      - M\ :sub:`21`
> +      - x
> +      - x
> +      - M\ :sub:`31`
> +      - x
> +      - x
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index 5aeff5519407..ae2dca7f2817 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -1460,6 +1460,13 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
>  	case V4L2_PIX_FMT_Y210:		descr = "10-bit YUYV Packed"; break;
>  	case V4L2_PIX_FMT_Y212:		descr = "12-bit YUYV Packed"; break;
>  	case V4L2_PIX_FMT_Y216:		descr = "16-bit YUYV Packed"; break;
> +	case V4L2_META_FMT_GENERIC_8:	descr = "8-bit Generic Metadata"; break;
> +	case V4L2_META_FMT_GENERIC_CSI2_10:	descr = "8-bit Generic Meta, 10b CSI-2"; break;
> +	case V4L2_META_FMT_GENERIC_CSI2_12:	descr = "8-bit Generic Meta, 12b CSI-2"; break;
> +	case V4L2_META_FMT_GENERIC_CSI2_14:	descr = "8-bit Generic Meta, 14b CSI-2"; break;
> +	case V4L2_META_FMT_GENERIC_CSI2_16:	descr = "8-bit Generic Meta, 16b CSI-2"; break;
> +	case V4L2_META_FMT_GENERIC_CSI2_20:	descr = "8-bit Generic Meta, 20b CSI-2"; break;
> +	case V4L2_META_FMT_GENERIC_CSI2_24:	descr = "8-bit Generic Meta, 24b CSI-2"; break;
>  
>  	default:
>  		/* Compressed formats */
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index 2663213b76a4..c7cf20b5da67 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -839,6 +839,14 @@ struct v4l2_pix_format {
>  #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
>  #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
>  
> +#define V4L2_META_FMT_GENERIC_8		v4l2_fourcc('M', 'E', 'T', '8') /* Generic 8-bit metadata */
> +#define V4L2_META_FMT_GENERIC_CSI2_10	v4l2_fourcc('M', 'C', '1', 'A') /* 10-bit CSI-2 packed 8-bit metadata */
> +#define V4L2_META_FMT_GENERIC_CSI2_12	v4l2_fourcc('M', 'C', '1', 'C') /* 12-bit CSI-2 packed 8-bit metadata */
> +#define V4L2_META_FMT_GENERIC_CSI2_14	v4l2_fourcc('M', 'C', '1', 'E') /* 14-bit CSI-2 packed 8-bit metadata */
> +#define V4L2_META_FMT_GENERIC_CSI2_16	v4l2_fourcc('M', 'C', '1', 'G') /* 16-bit CSI-2 packed 8-bit metadata */
> +#define V4L2_META_FMT_GENERIC_CSI2_20	v4l2_fourcc('M', 'C', '1', 'K') /* 20-bit CSI-2 packed 8-bit metadata */
> +#define V4L2_META_FMT_GENERIC_CSI2_24	v4l2_fourcc('M', 'C', '1', 'O') /* 24-bit CSI-2 packed 8-bit metadata */
> +
>  /* priv field value to indicates that subsequent fields are valid. */
>  #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
>  

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 07/46] media: v4l: Support line-based metadata capture

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Tue, Apr 16, 2024 at 10:32:40PM +0300, Sakari Ailus wrote:
> Many camera sensors, among other devices, transmit embedded data and image
> data for each CSI-2 frame. This embedded data typically contains register
> configuration of the sensor that has been used to capture the image data
> of the same frame.
> 
> The embedded data is received by the CSI-2 receiver and has the same
> properties as the image data, including that it is line based: it has
> width, height and bytesperline (stride).
> 
> Add these fields to struct v4l2_meta_format and document them.
> 
> Also add V4L2_FMT_FLAG_META_LINE_BASED to tell a given format is
> line-based i.e. these fields of struct v4l2_meta_format are valid for it.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../userspace-api/media/v4l/dev-meta.rst      | 21 +++++++++++++++++++
>  .../media/v4l/vidioc-enum-fmt.rst             |  7 +++++++
>  .../media/videodev2.h.rst.exceptions          |  1 +
>  drivers/media/v4l2-core/v4l2-ioctl.c          |  5 +++--
>  include/uapi/linux/videodev2.h                | 10 +++++++++
>  5 files changed, 42 insertions(+), 2 deletions(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/dev-meta.rst b/Documentation/userspace-api/media/v4l/dev-meta.rst
> index 0e7e1ee1471a..5eee9ab60395 100644
> --- a/Documentation/userspace-api/media/v4l/dev-meta.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-meta.rst
> @@ -47,6 +47,12 @@ member of the ``fmt`` union as needed per the desired operation. Both drivers
>  and applications must set the remainder of the :c:type:`v4l2_format` structure
>  to 0.
>  
> +Devices that capture metadata by line have the struct v4l2_fmtdesc
> +``V4L2_FMT_FLAG_META_LINE_BASED`` flag set for :c:func:`VIDIOC_ENUM_FMT`. Such
> +devices can typically also :ref:`capture image data <capture>`. This primarily
> +involves devices that receive the data from a different devices such as a camera
> +sensor.
> +
>  .. c:type:: v4l2_meta_format
>  
>  .. tabularcolumns:: |p{1.4cm}|p{2.4cm}|p{13.5cm}|
> @@ -65,3 +71,18 @@ to 0.
>        - ``buffersize``
>        - Maximum buffer size in bytes required for data. The value is set by the
>          driver.
> +    * - __u32
> +      - ``width``
> +      - Width of a line of metadata in Data Units. Valid when
> +	:c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set,
> +	otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
> +    * - __u32
> +      - ``height``
> +      - Number of rows of metadata. Valid when :c:type`v4l2_fmtdesc` flag
> +	``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
> +	:c:func:`VIDIOC_ENUM_FMT`.
> +    * - __u32
> +      - ``bytesperline``
> +      - Offset in bytes between the beginning of two consecutive lines. Valid
> +	when :c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is
> +	set, otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> index 000c154b0f98..a439be1b15d1 100644
> --- a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> +++ b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> @@ -227,6 +227,13 @@ the ``mbus_code`` field is handled differently:
>  	The application can ask to configure the quantization of the capture
>  	device when calling the :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl with
>  	:ref:`V4L2_PIX_FMT_FLAG_SET_CSC <v4l2-pix-fmt-flag-set-csc>` set.
> +    * - ``V4L2_FMT_FLAG_META_LINE_BASED``
> +      - 0x0200
> +      - The metadata format is line-based. In this case the ``width``,
> +	``height`` and ``bytesperline`` fields of :c:type:`v4l2_meta_format` are
> +	valid. The buffer consists of ``height`` lines, each having ``width``
> +	Data Units of data and offset (in bytes) between the beginning of each

s/and offset/, and the offset/

> +	two consecutive lines is ``bytesperline``.
>  
>  Return Value
>  ============
> diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b/Documentation/userspace-api/media/videodev2.h.rst.exceptions
> index 3e58aac4ef0b..bdc628e8c1d6 100644
> --- a/Documentation/userspace-api/media/videodev2.h.rst.exceptions
> +++ b/Documentation/userspace-api/media/videodev2.h.rst.exceptions
> @@ -215,6 +215,7 @@ replace define V4L2_FMT_FLAG_CSC_XFER_FUNC fmtdesc-flags
>  replace define V4L2_FMT_FLAG_CSC_YCBCR_ENC fmtdesc-flags
>  replace define V4L2_FMT_FLAG_CSC_HSV_ENC fmtdesc-flags
>  replace define V4L2_FMT_FLAG_CSC_QUANTIZATION fmtdesc-flags
> +replace define V4L2_FMT_FLAG_META_LINE_BASED fmtdesc-flags
>  
>  # V4L2 timecode types
>  replace define V4L2_TC_TYPE_24FPS timecode-type
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index ae2dca7f2817..2cfc9106857a 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -343,8 +343,9 @@ static void v4l_print_format(const void *arg, bool write_only)
>  	case V4L2_BUF_TYPE_META_OUTPUT:
>  		meta = &p->fmt.meta;
>  		pixelformat = meta->dataformat;
> -		pr_cont(", dataformat=%p4cc, buffersize=%u\n",
> -			&pixelformat, meta->buffersize);
> +		pr_cont(", dataformat=%p4cc, buffersize=%u, width=%u, height=%u, bytesperline=%u\n",
> +			&pixelformat, meta->buffersize, meta->width,
> +			meta->height, meta->bytesperline);
>  		break;
>  	}
>  }
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index c7cf20b5da67..37112dfebd0c 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -877,6 +877,7 @@ struct v4l2_fmtdesc {
>  #define V4L2_FMT_FLAG_CSC_YCBCR_ENC		0x0080
>  #define V4L2_FMT_FLAG_CSC_HSV_ENC		V4L2_FMT_FLAG_CSC_YCBCR_ENC
>  #define V4L2_FMT_FLAG_CSC_QUANTIZATION		0x0100
> +#define V4L2_FMT_FLAG_META_LINE_BASED		0x0200

Could the V4L2_FMT_FLAG_META_LINE_BASED flag be set by the V4L2 core for
line-based metadata formats, instead of requiring drivers to set it ? It
would ensure consistency.

>  
>  	/* Frame Size and frame rate enumeration */
>  /*
> @@ -2424,10 +2425,19 @@ struct v4l2_sdr_format {
>   * struct v4l2_meta_format - metadata format definition
>   * @dataformat:		little endian four character code (fourcc)
>   * @buffersize:		maximum size in bytes required for data
> + * @width:		number of data units of data per line (valid for line
> + *			based formats only, see format documentation)
> + * @height:		number of lines of data per buffer (valid for line based
> + *			formats only)
> + * @bytesperline:	offset between the beginnings of two adjacent lines in
> + *			bytes (valid for line based formats only)
>   */
>  struct v4l2_meta_format {
>  	__u32				dataformat;
>  	__u32				buffersize;
> +	__u32				width;
> +	__u32				height;
> +	__u32				bytesperline;
>  } __attribute__ ((packed));
>  
>  /**

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 11/46] media: Documentation: Document S_ROUTING behaviour

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Tue, Apr 16, 2024 at 10:32:44PM +0300, Sakari Ailus wrote:
> Document S_ROUTING behaviour for different devices.
> 
> Generally in devices that produce streams the streams are static and some
> can be enabled and disabled, whereas in devices that just transport them
> or write them to memory, more configurability is allowed. Document this.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> Reviewed-by: Julien Massot <julien.massot@collabora.com>
> ---
>  .../userspace-api/media/v4l/dev-subdev.rst    | 24 +++++++++++++++++++
>  1 file changed, 24 insertions(+)
> 
> diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> index d30dcb9e2537..de8dfd4f11a5 100644
> --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> @@ -593,6 +593,30 @@ Any configurations of a stream within a pad, such as format or selections,
>  are independent of similar configurations on other streams. This is
>  subject to change in the future.
>  
> +Device types and routing setup
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +Different kinds of sub-devices have differing behaviour for route activation,
> +depending on the hardware. In all cases, however, only routes that have the
> +``V4L2_SUBDEV_STREAM_FL_ACTIVE`` flag set are active.
> +
> +Devices generating the streams may allow enabling and disabling some of the
> +routes or the configuration is fixed. If the routes can be disabled, not

"... some of the routes, or have a fixed routing configuration."

> +declaring the routes (or declaring them without
> +``VIDIOC_SUBDEV_STREAM_FL_ACTIVE`` flag set) in ``VIDIOC_SUBDEV_S_ROUTING`` will
> +disable the routes while the sub-device driver retains the streams and their
> +format and selection configuration.

I still find the "retains their format and selection configuration"
quite unclear :-S

> The ``VIDIOC_SUBDEV_S_ROUTING`` will still

s/will still/ioctl will still/

> +return such routes back to the user in the routes array, with the
> +``V4L2_SUBDEV_STREAM_FL_ACTIVE`` flag unset.
> +
> +Devices transporting the streams almost always have more configurability with
> +respect to routing. Typically any route between the sub-device's sink and source
> +pads is possible, and multiple routes (usually up to certain limited number) may
> +be active simultaneously. For such devices, no routes are created by the driver
> +and user-created routes are fully replaced when ``VIDIOC_SUBDEV_S_ROUTING`` is
> +called on the sub-device. Such newly created routes have the device's default
> +configuration for format and selection rectangles.
> +
>  Configuring streams
>  ^^^^^^^^^^^^^^^^^^^
>  

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 10/46] media: Documentation: v4l: Document internal sink pads

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Tue, Apr 16, 2024 at 10:32:43PM +0300, Sakari Ailus wrote:
> Document internal sink pads, pads that have both SINK and INTERNAL flags
> set. Use the IMX219 camera sensor as an example.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> Reviewed-by Julien Massot <julien.massot@collabora.com>
> ---
>  .../userspace-api/media/v4l/dev-subdev.rst    | 145 ++++++++++++++++++
>  1 file changed, 145 insertions(+)
> 
> diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> index b76e02e54512..d30dcb9e2537 100644
> --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> @@ -553,6 +553,27 @@ A stream at a specific point in the media pipeline is identified by the
>  sub-device and a (pad, stream) pair. For sub-devices that do not support
>  multiplexed streams the 'stream' field is always 0.
>  
> +.. _v4l2-subdev-internal-source-pads:
> +
> +Internal sink pads and routing
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +Cases where a single sub-device source pad is traversed by multiple streams, one
> +or more of which originate from within the sub-device itself, are special as
> +there is no external sink pad for such routes. In those cases, the sources of
> +the internally generated streams are represented by internal sink pads, which
> +are sink pads that have the :ref:`MEDIA_PAD_FL_INTERNAL <MEDIA-PAD-FL-INTERNAL>`
> +pad flag set.
> +
> +Internal pads have all the properties of an external pad, including formats and
> +selections. The format in this case is the source format of the stream. An
> +internal pad always has a single stream only (0).
> +
> +Routes from an internal sink pad to an external source pad are typically not
> +modifiable but they can be activated and deactivated using the
> +:ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE <v4l2-subdev-routing-flags>` flag, depending
> +on driver capabilities.
> +
>  Interaction between routes, streams, formats and selections
>  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>  
> @@ -668,3 +689,127 @@ To configure this pipeline, the userspace must take the following steps:
>     the configurations along the stream towards the receiver, using
>     :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each
>     stream endpoint in each sub-device.
> +
> +Internal pads setup example
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +A simple example of a multiplexed stream setup might be as follows:
> +
> +- An IMX219 camera sensor source sub-device, with one source pad (0), one
> +  internal sink pad (1) as the source of the image data and an internal sink
> +  pad (2) as the source of the embedded data. There are two routes, one from the
> +  internal sink pad 1 to the source 0 (image data) and another from the internal

s/source 0/source pad 0/

> +  sink pad 2 to the source pad 0 (embedded data). Both streams are always
> +  active, i.e. there is no need to separately enable the embedded data
> +  stream. The sensor uses the CSI-2 interface.
> +
> +- A CSI-2 receiver in the SoC. The receiver has a single sink pad (pad 0),
> +  connected to the sensor, and two source pads (pads 1 and 2), to the DMA
> +  engine. The receiver demultiplexes the incoming streams to the source pads.

s/engine/engines/

maybe better, "to two DMA engines".

> +
> +- DMA engines in the SoC, one for each stream. Each DMA engine is connected to a
> +  single source pad of the receiver.
> +
> +The sensor and the receiver are modelled as V4L2 sub-devices, exposed to
> +userspace via /dev/v4l-subdevX device nodes. The DMA engines are modelled as
> +V4L2 devices, exposed to userspace via /dev/videoX nodes.
> +
> +To configure this pipeline, the userspace must take the following steps:
> +
> +1) Set up media links between entities: enable the links from the sensor to the
> +   receiver and from the receiver to the DMA engines. This step does not differ
> +   from normal non-multiplexed media controller setup.
> +
> +2) Configure routing
> +
> +.. flat-table:: Camera sensor. There are no configurable routes.
> +    :header-rows: 1
> +
> +    * - Sink Pad/Stream
> +      - Source Pad/Stream
> +      - Routing Flags
> +      - Comments
> +    * - 1/0
> +      - 0/0
> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +      - Pixel data stream from the sink pad

"from the internal image sink pad"

> +    * - 2/0
> +      - 0/1
> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +      - Metadata stream from the internal sink pad

"internal embedded data sink pad"

> +
> +.. flat-table:: Receiver routing table. Typically both routes need to be
> +		explicitly set.
> +    :header-rows:  1
> +
> +    * - Sink Pad/Stream
> +      - Source Pad/Stream
> +      - Routing Flags
> +      - Comments
> +    * - 0/0
> +      - 1/0
> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +      - Pixel data stream from camera sensor
> +    * - 0/1
> +      - 2/0
> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +      - Metadata stream from camera sensor
> +
> +The options available in sensor's routing configuration are dictated by hardware

s/in sensor/in the sensor/

> +capabilities: typically camera sensors always produce an image data stream while
> +it may be possible to enable and disable the embedded data stream.

Should this go after the first table ?

> +
> +3) Configure formats and selections
> +
> +   This example assumes that the formats are propagated from sink pad to the
> +   source pad as-is. The tables contain fields of both struct v4l2_subdev_format
> +   and struct v4l2_mbus_framefmt.
> +
> +.. flat-table:: Formats set on the sub-devices. Bold values are set, others are
> +                static or propagated. The order is aligned with configured
> +                routes.
> +    :header-rows: 1
> +    :fill-cells:
> +
> +    * - Sub-device
> +      - Pad/Stream
> +      - Width
> +      - Height
> +      - Code
> +    * - :rspan:`3` IMX219
> +      - 1/0
> +      - 3296
> +      - 2480
> +      - MEDIA_BUS_FMT_SRGGB10
> +    * - 0/0
> +      - **3296**
> +      - **2480**
> +      - **MEDIA_BUS_FMT_SRGGB10**
> +    * - 2/0
> +      - 3296
> +      - 2
> +      - MEDIA_BUS_FMT_IMX219_EMBEDDED

We need a patch in this series to define this format.

> +    * - 1/1

I think this should be 0/1.

> +      - 3296
> +      - 2
> +      - MEDIA_BUS_FMT_META_10
> +    * - :rspan:`3` CSI-2 receiver
> +      - 0/0
> +      - **3296**
> +      - **2480**
> +      - **MEDIA_BUS_FMT_SRGGB10**
> +    * - 1/0
> +      - 3296
> +      - 2480
> +      - MEDIA_BUS_FMT_SRGGB10
> +    * - 0/1
> +      - **3296**
> +      - **2**
> +      - **MEDIA_BUS_FMT_META_10**
> +    * - 2/0
> +      - 3296
> +      - 2
> +      - MEDIA_BUS_FMT_META_10
> +
> +The embedded data format does not need to be configured as the format is

Do you mean the "format on the sensor's pads" ? It's a bit confusing if
you don't specify it.

> +dictated by the pixel data format in this case.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 15/46] media: v4l: subdev: Add len_routes field to struct v4l2_subdev_routing

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Tue, Apr 16, 2024 at 10:32:48PM +0300, Sakari Ailus wrote:
> The len_routes field is used to tell the size of the routes array in
> struct v4l2_subdev_routing. This way the number of routes returned from
> S_ROUTING IOCTL may be larger than the number of routes provided, in case
> there are more routes returned by the driver.
> 
> Note that this uAPI is still disabled in the code, so this change can
> safely be done. Anyone who manually patched the code to enable this uAPI
> must update their code.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../media/v4l/vidioc-subdev-g-routing.rst     | 50 +++++++++++++------
>  drivers/media/v4l2-core/v4l2-ioctl.c          |  4 +-
>  drivers/media/v4l2-core/v4l2-subdev.c         | 12 ++---
>  include/media/v4l2-subdev.h                   |  2 +
>  include/uapi/linux/v4l2-subdev.h              |  9 ++--
>  5 files changed, 52 insertions(+), 25 deletions(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> index 26b5004bfe6d..27eb4c82a0e1 100644
> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> @@ -43,23 +43,42 @@ The routing configuration determines the flows of data inside an entity.
>  Drivers report their current routing tables using the
>  ``VIDIOC_SUBDEV_G_ROUTING`` ioctl and application may enable or disable routes
>  with the ``VIDIOC_SUBDEV_S_ROUTING`` ioctl, by adding or removing routes and
> -setting or clearing flags of the  ``flags`` field of a
> -struct :c:type:`v4l2_subdev_route`.
> +setting or clearing flags of the ``flags`` field of a struct
> +:c:type:`v4l2_subdev_route`.
>  
> -All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called. This
> -means that the userspace must reconfigure all streams after calling the ioctl
> -with e.g. ``VIDIOC_SUBDEV_S_FMT``.
> +All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called.
> +This means that the userspace must reconfigure all stream formats and selections
> +after calling the ioctl with e.g. ``VIDIOC_SUBDEV_S_FMT``.
>  
>  Only subdevices which have both sink and source pads can support routing.
>  
> -When inspecting routes through ``VIDIOC_SUBDEV_G_ROUTING`` and the application
> -provided ``num_routes`` is not big enough to contain all the available routes
> -the subdevice exposes, drivers return the ENOSPC error code and adjust the
> -value of the ``num_routes`` field. Application should then reserve enough memory
> -for all the route entries and call ``VIDIOC_SUBDEV_G_ROUTING`` again.
> -
> -On a successful ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the
> -``num_routes`` field to reflect the actual number of routes returned.
> +The ``len_routes`` field indicates the number of routes that can fit in the
> +``routes`` array allocated by userspace. It is set by applications for both
> +ioctls to indicate how many routes the kernel can return, and is never modified
> +by the kernel.
> +
> +The ``num_routes`` field, when returned from the kernel on both IOCTLs,
> +indicates the number of routes in the subdevice routing table and when calling
> +``VIDIOC_SUBDEV_S_ROUTING``, it is set by userspace to the number of routes that
> +the application stored in the ``routes`` array. The value returned by the kernel
> +may be smaller or larger than the value of ``num_routes`` set by the application
> +for ``VIDIOC_SUBDEV_S_ROUTING``, as drivers may adjust the requested routing
> +table.

I still think the proposal I made when reviewing the previous version is
clearer :-)

----
The ``num_routes`` field indicates the number of routes in the subdevice routing
table. For ``VIDIOC_SUBDEV_S_ROUTING``, it is set by userspace to the number of
routes that the application stored in the ``routes`` array. For both ioctls, it
is returned by the kernel and indicates how many routes are stored in the
subdevice routing table. This may be smaller or larger than the value of
``num_routes`` set by the application for ``VIDIOC_SUBDEV_S_ROUTING``, as
drivers may adjust the requested routing table.
----

You replied that

> For S_ROUTING this is the number of routes in the IOCTL argument. The
> routing table may contain more (static routes).

and that's right, but, even when set by userspace for S_ROUTING, the
num_routes fields is the number of routes that userspace tries to set in
the routing table. I think starting with a first sentence that describes
what the field contains, and then explaining how it's used for the
different ioctls by userspace and kernel space, is clearer.

> +
> +The kernel can return a ``num_routes`` value larger than ``len_routes`` from
> +both ioctls. This indicates thare are more routes in the routing table than fits
> +the ``routes`` array. In this case, the ``routes`` array is filled by the kernel
> +with the first ``len_routes`` entries of the subdevice routing table. This is
> +not considered to be an error, and the ioctl call succeeds. If the applications
> +wants to retrieve the missing routes, it can issue a new
> +``VIDIOC_SUBDEV_G_ROUTING`` call with a large enough ``routes`` array.
> +
> +indicate there are more routes than fits to the ``routes`` array. In this
> +case first ``len_routes`` were returned back to the userspace in the
> +``routes`` array. This is not considered as an error.

I think these 3 lines are a leftover.

> +
> +Also ``VIDIOC_SUBDEV_S_ROUTING`` may return more route than the user provided in

s/Also //
s/route/routes/

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> +``num_routes`` field due to e.g. hardware properties.
>  
>  .. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
>  
> @@ -74,6 +93,9 @@ On a successful ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the
>        - ``which``
>        - Routing table to be accessed, from enum
>          :ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
> +    * - __u32
> +      - ``len_routes``
> +      - The length of the array (as in memory reserved for the array)
>      * - struct :c:type:`v4l2_subdev_route`
>        - ``routes[]``
>        - Array of struct :c:type:`v4l2_subdev_route` entries
> @@ -81,7 +103,7 @@ On a successful ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the
>        - ``num_routes``
>        - Number of entries of the routes array
>      * - __u32
> -      - ``reserved``\ [5]
> +      - ``reserved``\ [11]
>        - Reserved for future extensions. Applications and drivers must set
>  	the array to zero.
>  
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index 1863ecae9ec9..f30f61c008c7 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -3185,13 +3185,13 @@ static int check_array_args(unsigned int cmd, void *parg, size_t *array_size,
>  	case VIDIOC_SUBDEV_S_ROUTING: {
>  		struct v4l2_subdev_routing *routing = parg;
>  
> -		if (routing->num_routes > 256)
> +		if (routing->len_routes > 256)
>  			return -E2BIG;
>  
>  		*user_ptr = u64_to_user_ptr(routing->routes);
>  		*kernel_ptr = (void **)&routing->routes;
>  		*array_size = sizeof(struct v4l2_subdev_route)
> -			    * routing->num_routes;
> +			    * routing->len_routes;
>  		ret = 1;
>  		break;
>  	}
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 2ba11e5343f0..904378007a79 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -927,6 +927,9 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>  		if (routing->which != V4L2_SUBDEV_FORMAT_TRY && ro_subdev)
>  			return -EPERM;
>  
> +		if (routing->num_routes > routing->len_routes)
> +			return -EINVAL;
> +
>  		memset(routing->reserved, 0, sizeof(routing->reserved));
>  
>  		for (i = 0; i < routing->num_routes; ++i) {
> @@ -953,6 +956,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>  		}
>  
>  		krouting.num_routes = routing->num_routes;
> +		krouting.len_routes = routing->len_routes;
>  		krouting.routes = routes;
>  
>  		return v4l2_subdev_call(sd, pad, set_routing, state,
> @@ -973,14 +977,10 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>  
>  		krouting = &state->routing;
>  
> -		if (routing->num_routes < krouting->num_routes) {
> -			routing->num_routes = krouting->num_routes;
> -			return -ENOSPC;
> -		}
> -
>  		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
>  		       krouting->routes,
> -		       krouting->num_routes * sizeof(*krouting->routes));
> +		       min(krouting->num_routes, routing->len_routes) *
> +		       sizeof(*krouting->routes));
>  		routing->num_routes = krouting->num_routes;
>  
>  		return 0;
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 9cce48365975..1df6b963a1c9 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -728,12 +728,14 @@ struct v4l2_subdev_stream_configs {
>  /**
>   * struct v4l2_subdev_krouting - subdev routing table
>   *
> + * @len_routes: length of routes array, in routes
>   * @num_routes: number of routes
>   * @routes: &struct v4l2_subdev_route
>   *
>   * This structure contains the routing table for a subdev.
>   */
>  struct v4l2_subdev_krouting {
> +	unsigned int len_routes;
>  	unsigned int num_routes;
>  	struct v4l2_subdev_route *routes;
>  };
> diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
> index 81a24bd38003..6a39128d0606 100644
> --- a/include/uapi/linux/v4l2-subdev.h
> +++ b/include/uapi/linux/v4l2-subdev.h
> @@ -228,15 +228,18 @@ struct v4l2_subdev_route {
>   * struct v4l2_subdev_routing - Subdev routing information
>   *
>   * @which: configuration type (from enum v4l2_subdev_format_whence)
> - * @num_routes: the total number of routes in the routes array
> + * @len_routes: the length of the routes array, in routes
>   * @routes: pointer to the routes array
> + * @num_routes: the total number of routes, possibly more than fits in the
> + *		routes array
>   * @reserved: drivers and applications must zero this array
>   */
>  struct v4l2_subdev_routing {
>  	__u32 which;
> -	__u32 num_routes;
> +	__u32 len_routes;
>  	__u64 routes;
> -	__u32 reserved[6];
> +	__u32 num_routes;
> +	__u32 reserved[11];
>  };
>  
>  /*

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 16/46] media: v4l: subdev: Return routes set using S_ROUTING

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Tue, Apr 16, 2024 at 10:32:49PM +0300, Sakari Ailus wrote:
> Return the routes set using S_ROUTING back to the user. Also reflect this
> in documentation.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../media/v4l/vidioc-subdev-g-routing.rst            |  7 ++-----
>  drivers/media/v4l2-core/v4l2-subdev.c                | 12 +++++++++++-
>  2 files changed, 13 insertions(+), 6 deletions(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> index 27eb4c82a0e1..88df7bf80b00 100644
> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> @@ -44,7 +44,8 @@ Drivers report their current routing tables using the
>  ``VIDIOC_SUBDEV_G_ROUTING`` ioctl and application may enable or disable routes
>  with the ``VIDIOC_SUBDEV_S_ROUTING`` ioctl, by adding or removing routes and
>  setting or clearing flags of the ``flags`` field of a struct
> -:c:type:`v4l2_subdev_route`.
> +:c:type:`v4l2_subdev_route`. Similarly to ``VIDIOC_SUBDEV_G_ROUTING``, also
> +``VIDIOC_SUBDEV_S_ROUTING`` returns the routes back to the user.

:c:type:`v4l2_subdev_route`. Similarly to ``VIDIOC_SUBDEV_G_ROUTING``,
``VIDIOC_SUBDEV_S_ROUTING`` also returns the routes back to the user.

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

>  
>  All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called.
>  This means that the userspace must reconfigure all stream formats and selections
> @@ -157,10 +158,6 @@ On success 0 is returned, on error -1 and the ``errno`` variable is set
>  appropriately. The generic error codes are described at the
>  :ref:`Generic Error Codes <gen-errors>` chapter.
>  
> -ENOSPC
> -   The application provided ``num_routes`` is not big enough to contain
> -   all the available routes the subdevice exposes.
> -
>  EINVAL
>     The sink or source pad identifiers reference a non-existing pad or reference
>     pads of different types (ie. the sink_pad identifiers refers to a source
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 904378007a79..36b2f78cd551 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -959,8 +959,18 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>  		krouting.len_routes = routing->len_routes;
>  		krouting.routes = routes;
>  
> -		return v4l2_subdev_call(sd, pad, set_routing, state,
> +		rval = v4l2_subdev_call(sd, pad, set_routing, state,
>  					routing->which, &krouting);
> +		if (rval < 0)
> +			return rval;
> +
> +		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
> +		       state->routing.routes,
> +		       min(state->routing.num_routes, routing->len_routes) *
> +		       sizeof(*state->routing.routes));
> +		routing->num_routes = state->routing.num_routes;
> +
> +		return 0;
>  	}
>  
>  	case VIDIOC_SUBDEV_G_ROUTING: {

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 19/46] media: ccs: Move ccs_pm_get_init function up

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Tue, Apr 16, 2024 at 10:32:52PM +0300, Sakari Ailus wrote:
> Prepare for using ccs_pm_get_init from locations earlier than its the
> current place.
> 
> Also add a missing newline while at it.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> ---
>  drivers/media/i2c/ccs/ccs-core.c | 73 ++++++++++++++++----------------
>  1 file changed, 37 insertions(+), 36 deletions(-)
> 
> diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
> index 7e5474e38732..d7bc9418eabb 100644
> --- a/drivers/media/i2c/ccs/ccs-core.c
> +++ b/drivers/media/i2c/ccs/ccs-core.c
> @@ -1718,6 +1718,43 @@ static int ccs_power_off(struct device *dev)
>   * Video stream management
>   */
>  
> +static int ccs_pm_get_init(struct ccs_sensor *sensor)
> +{
> +	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
> +	int rval;
> +
> +	/*
> +	 * It can't use pm_runtime_resume_and_get() here, as the driver
> +	 * relies at the returned value to detect if the device was already
> +	 * active or not.
> +	 */
> +	rval = pm_runtime_get_sync(&client->dev);
> +	if (rval < 0)
> +		goto error;
> +
> +	/* Device was already active, so don't set controls */
> +	if (rval == 1 && !sensor->handler_setup_needed)
> +		return 0;
> +
> +	sensor->handler_setup_needed = false;
> +
> +	/* Restore V4L2 controls to the previously suspended device */
> +	rval = v4l2_ctrl_handler_setup(&sensor->pixel_array->ctrl_handler);
> +	if (rval)
> +		goto error;
> +
> +	rval = v4l2_ctrl_handler_setup(&sensor->src->ctrl_handler);
> +	if (rval)
> +		goto error;
> +
> +	/* Keep PM runtime usage_count incremented on success */
> +	return 0;
> +
> +error:
> +	pm_runtime_put(&client->dev);
> +	return rval;
> +}
> +
>  static int ccs_start_streaming(struct ccs_sensor *sensor)
>  {
>  	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
> @@ -1872,42 +1909,6 @@ static int ccs_stop_streaming(struct ccs_sensor *sensor)
>   * V4L2 subdev video operations
>   */
>  
> -static int ccs_pm_get_init(struct ccs_sensor *sensor)
> -{
> -	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
> -	int rval;
> -
> -	/*
> -	 * It can't use pm_runtime_resume_and_get() here, as the driver
> -	 * relies at the returned value to detect if the device was already
> -	 * active or not.
> -	 */
> -	rval = pm_runtime_get_sync(&client->dev);
> -	if (rval < 0)
> -		goto error;
> -
> -	/* Device was already active, so don't set controls */
> -	if (rval == 1 && !sensor->handler_setup_needed)
> -		return 0;
> -
> -	sensor->handler_setup_needed = false;
> -
> -	/* Restore V4L2 controls to the previously suspended device */
> -	rval = v4l2_ctrl_handler_setup(&sensor->pixel_array->ctrl_handler);
> -	if (rval)
> -		goto error;
> -
> -	rval = v4l2_ctrl_handler_setup(&sensor->src->ctrl_handler);
> -	if (rval)
> -		goto error;
> -
> -	/* Keep PM runtime usage_count incremented on success */
> -	return 0;
> -error:
> -	pm_runtime_put(&client->dev);
> -	return rval;
> -}
> -
>  static int ccs_set_stream(struct v4l2_subdev *subdev, int enable)
>  {
>  	struct ccs_sensor *sensor = to_ccs_sensor(subdev);

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 21/46] media: ccs: Use {enable,disable}_streams operations

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Tue, Apr 16, 2024 at 10:32:54PM +0300, Sakari Ailus wrote:
> Switch from s_stream() video op to enable_streams() and disable_streams()
> pad operations. They are preferred and required for streams support.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> ---
>  drivers/media/i2c/ccs/ccs-core.c | 83 ++++++++++++++------------------
>  1 file changed, 36 insertions(+), 47 deletions(-)
> 
> diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
> index d14e90f8568a..aaa6bf8495e1 100644
> --- a/drivers/media/i2c/ccs/ccs-core.c
> +++ b/drivers/media/i2c/ccs/ccs-core.c
> @@ -1715,7 +1715,7 @@ static int ccs_power_off(struct device *dev)
>  }
>  
>  /* -----------------------------------------------------------------------------
> - * Video stream management
> + * V4L2 subdev video operations
>   */
>  
>  static int ccs_pm_get_init(struct ccs_sensor *sensor)
> @@ -1739,11 +1739,11 @@ static int ccs_pm_get_init(struct ccs_sensor *sensor)
>  	sensor->handler_setup_needed = false;
>  
>  	/* Restore V4L2 controls to the previously suspended device */
> -	rval = v4l2_ctrl_handler_setup(&sensor->pixel_array->ctrl_handler);
> +	rval = __v4l2_ctrl_handler_setup(&sensor->pixel_array->ctrl_handler);
>  	if (rval)
>  		goto error;
>  
> -	rval = v4l2_ctrl_handler_setup(&sensor->src->ctrl_handler);
> +	rval = __v4l2_ctrl_handler_setup(&sensor->src->ctrl_handler);
>  	if (rval)
>  		goto error;
>  
> @@ -1755,13 +1755,21 @@ static int ccs_pm_get_init(struct ccs_sensor *sensor)
>  	return rval;
>  }
>  
> -static int ccs_start_streaming(struct ccs_sensor *sensor)
> +static int ccs_enable_streams(struct v4l2_subdev *subdev,
> +			      struct v4l2_subdev_state *state, u32 pad,
> +			      u64 streams_mask)
>  {
> +	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
>  	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
>  	unsigned int binning_mode;
>  	int rval;
>  
> -	mutex_lock(&sensor->mutex);
> +	if (pad != CCS_PAD_SRC)
> +		return -EINVAL;
> +
> +	rval = ccs_pm_get_init(sensor);
> +	if (rval)
> +		return rval;
>  
>  	rval = ccs_write(sensor, CSI_DATA_FORMAT,
>  			 (sensor->csi_format->width << 8) |
> @@ -1880,64 +1888,41 @@ static int ccs_start_streaming(struct ccs_sensor *sensor)
>  
>  	rval = ccs_write(sensor, MODE_SELECT, CCS_MODE_SELECT_STREAMING);
>  
> -err_pm_put:
> -	mutex_unlock(&sensor->mutex);
> -
> -	return rval;
> -}
> -
> -static int ccs_stop_streaming(struct ccs_sensor *sensor)
> -{
> -	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
> -	int rval;
> +	sensor->streaming = true;
>  
> -	mutex_lock(&sensor->mutex);
> -	rval = ccs_write(sensor, MODE_SELECT, CCS_MODE_SELECT_SOFTWARE_STANDBY);
> -	if (rval)
> -		goto out;
> +	return 0;
>  
> -	rval = ccs_call_quirk(sensor, post_streamoff);
> -	if (rval)
> -		dev_err(&client->dev, "post_streamoff quirks failed\n");
> +err_pm_put:
> +	pm_runtime_mark_last_busy(&client->dev);
> +	pm_runtime_put_autosuspend(&client->dev);
>  
> -out:
> -	mutex_unlock(&sensor->mutex);
>  	return rval;
>  }
>  
> -/* -----------------------------------------------------------------------------
> - * V4L2 subdev video operations
> - */
> -
> -static int ccs_set_stream(struct v4l2_subdev *subdev, int enable)
> +static int ccs_disable_streams(struct v4l2_subdev *subdev,
> +			       struct v4l2_subdev_state *state, u32 pad,
> +			       u64 streams_mask)
>  {
>  	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
>  	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
>  	int rval;
>  
> -	if (!enable) {
> -		ccs_stop_streaming(sensor);
> -		sensor->streaming = false;
> -		pm_runtime_mark_last_busy(&client->dev);
> -		pm_runtime_put_autosuspend(&client->dev);
> -
> -		return 0;
> -	}
> +	if (pad != CCS_PAD_SRC)
> +		return -EINVAL;
>  
> -	rval = ccs_pm_get_init(sensor);
> +	rval = ccs_write(sensor, MODE_SELECT, CCS_MODE_SELECT_SOFTWARE_STANDBY);
>  	if (rval)
>  		return rval;
>  
> -	sensor->streaming = true;
> +	rval = ccs_call_quirk(sensor, post_streamoff);
> +	if (rval)
> +		dev_err(&client->dev, "post_streamoff quirks failed\n");
>  
> -	rval = ccs_start_streaming(sensor);
> -	if (rval < 0) {
> -		sensor->streaming = false;
> -		pm_runtime_mark_last_busy(&client->dev);
> -		pm_runtime_put_autosuspend(&client->dev);
> -	}
> +	sensor->streaming = false;
> +	pm_runtime_mark_last_busy(&client->dev);
> +	pm_runtime_put_autosuspend(&client->dev);
>  
> -	return rval;
> +	return 0;
>  }
>  
>  static int ccs_pre_streamon(struct v4l2_subdev *subdev, u32 flags)
> @@ -1963,7 +1948,9 @@ static int ccs_pre_streamon(struct v4l2_subdev *subdev, u32 flags)
>  		}
>  	}
>  
> +	mutex_lock(&sensor->mutex);
>  	rval = ccs_pm_get_init(sensor);
> +	mutex_unlock(&sensor->mutex);
>  	if (rval)
>  		return rval;
>  
> @@ -3047,7 +3034,7 @@ static int ccs_init_state(struct v4l2_subdev *sd,
>  }
>  
>  static const struct v4l2_subdev_video_ops ccs_video_ops = {
> -	.s_stream = ccs_set_stream,
> +	.s_stream = v4l2_subdev_s_stream_helper,
>  	.pre_streamon = ccs_pre_streamon,
>  	.post_streamoff = ccs_post_streamoff,
>  };
> @@ -3058,6 +3045,8 @@ static const struct v4l2_subdev_pad_ops ccs_pad_ops = {
>  	.set_fmt = ccs_set_format,
>  	.get_selection = ccs_get_selection,
>  	.set_selection = ccs_set_selection,
> +	.enable_streams = ccs_enable_streams,
> +	.disable_streams = ccs_disable_streams,
>  };
>  
>  static const struct v4l2_subdev_sensor_ops ccs_sensor_ops = {

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 24/46] media: ccs: Support frame descriptors

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Tue, Apr 16, 2024 at 10:32:57PM +0300, Sakari Ailus wrote:
> Provide information on the frame layout using frame descriptors.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> Reviewed-by: Julien Massot <julien.massot@collabora.com>

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> ---
>  drivers/media/i2c/ccs/ccs-core.c | 57 ++++++++++++++++++++++++++++++++
>  drivers/media/i2c/ccs/ccs.h      |  4 +++
>  2 files changed, 61 insertions(+)
> 
> diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
> index a711233f6fbf..3ca2415fca3b 100644
> --- a/drivers/media/i2c/ccs/ccs-core.c
> +++ b/drivers/media/i2c/ccs/ccs-core.c
> @@ -25,6 +25,7 @@
>  #include <linux/slab.h>
>  #include <linux/smiapp.h>
>  #include <linux/v4l2-mediabus.h>
> +#include <media/mipi-csi2.h>
>  #include <media/v4l2-cci.h>
>  #include <media/v4l2-device.h>
>  #include <media/v4l2-fwnode.h>
> @@ -245,6 +246,33 @@ static int ccs_read_all_limits(struct ccs_sensor *sensor)
>  	return ret;
>  }
>  
> +static u8 ccs_mipi_csi2_data_type(unsigned int bpp)
> +{
> +	switch (bpp) {
> +	case 6:
> +		return MIPI_CSI2_DT_RAW6;
> +	case 7:
> +		return MIPI_CSI2_DT_RAW7;
> +	case 8:
> +		return MIPI_CSI2_DT_RAW8;
> +	case 10:
> +		return MIPI_CSI2_DT_RAW10;
> +	case 12:
> +		return MIPI_CSI2_DT_RAW12;
> +	case 14:
> +		return MIPI_CSI2_DT_RAW14;
> +	case 16:
> +		return MIPI_CSI2_DT_RAW16;
> +	case 20:
> +		return MIPI_CSI2_DT_RAW20;
> +	case 24:
> +		return MIPI_CSI2_DT_RAW24;
> +	default:
> +		WARN_ON(1);
> +		return 0;
> +	}
> +}
> +
>  static int ccs_read_frame_fmt(struct ccs_sensor *sensor)
>  {
>  	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
> @@ -2633,6 +2661,34 @@ static int ccs_set_selection(struct v4l2_subdev *subdev,
>  	return ret;
>  }
>  
> +static int ccs_get_frame_desc(struct v4l2_subdev *subdev, unsigned int pad,
> +				 struct v4l2_mbus_frame_desc *desc)
> +{
> +	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
> +	struct v4l2_mbus_frame_desc_entry *entry = desc->entry;
> +
> +	switch (sensor->hwcfg.csi_signalling_mode) {
> +	case CCS_CSI_SIGNALING_MODE_CSI_2_DPHY:
> +	case CCS_CSI_SIGNALING_MODE_CSI_2_CPHY:
> +		desc->type = V4L2_MBUS_FRAME_DESC_TYPE_CSI2;
> +		break;
> +	default:
> +		/* FIXME: CCP2 support */
> +		return -EINVAL;
> +	}
> +
> +	entry->pixelcode = sensor->csi_format->code;
> +	entry->stream = CCS_STREAM_PIXEL;
> +	entry->bus.csi2.dt =
> +		sensor->csi_format->width == sensor->csi_format->compressed ?
> +		ccs_mipi_csi2_data_type(sensor->csi_format->width) :
> +		CCS_DEFAULT_COMPRESSED_DT;
> +	entry++;
> +	desc->num_entries++;
> +
> +	return 0;
> +}
> +
>  static int ccs_get_skip_frames(struct v4l2_subdev *subdev, u32 *frames)
>  {
>  	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
> @@ -3055,6 +3111,7 @@ static const struct v4l2_subdev_pad_ops ccs_pad_ops = {
>  	.set_selection = ccs_set_selection,
>  	.enable_streams = ccs_enable_streams,
>  	.disable_streams = ccs_disable_streams,
> +	.get_frame_desc = ccs_get_frame_desc,
>  };
>  
>  static const struct v4l2_subdev_sensor_ops ccs_sensor_ops = {
> diff --git a/drivers/media/i2c/ccs/ccs.h b/drivers/media/i2c/ccs/ccs.h
> index 4725e6eca8d0..90b442a3d53e 100644
> --- a/drivers/media/i2c/ccs/ccs.h
> +++ b/drivers/media/i2c/ccs/ccs.h
> @@ -46,6 +46,8 @@
>  
>  #define CCS_COLOUR_COMPONENTS		4
>  
> +#define CCS_DEFAULT_COMPRESSED_DT	MIPI_CSI2_DT_USER_DEFINED(0)
> +
>  #define SMIAPP_NAME			"smiapp"
>  #define CCS_NAME			"ccs"
>  
> @@ -175,6 +177,8 @@ struct ccs_csi_data_format {
>  #define CCS_PAD_SRC			1
>  #define CCS_PADS			2
>  
> +#define CCS_STREAM_PIXEL		0
> +
>  struct ccs_binning_subtype {
>  	u8 horizontal:4;
>  	u8 vertical:4;

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 20/46] media: ccs: Rename out label of ccs_start_streaming

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Tue, Apr 16, 2024 at 10:32:53PM +0300, Sakari Ailus wrote:
> In preparation for upcoming changes in the function, rename the out label
> as err_pm_put. The purpose of the label is changed to match its name in
> the next patch.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> ---
>  drivers/media/i2c/ccs/ccs-core.c | 38 ++++++++++++++++----------------
>  1 file changed, 19 insertions(+), 19 deletions(-)
> 
> diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
> index d7bc9418eabb..d14e90f8568a 100644
> --- a/drivers/media/i2c/ccs/ccs-core.c
> +++ b/drivers/media/i2c/ccs/ccs-core.c
> @@ -1767,7 +1767,7 @@ static int ccs_start_streaming(struct ccs_sensor *sensor)
>  			 (sensor->csi_format->width << 8) |
>  			 sensor->csi_format->compressed);
>  	if (rval)
> -		goto out;
> +		goto err_pm_put;
>  
>  	/* Binning configuration */
>  	if (sensor->binning_horizontal == 1 &&
> @@ -1780,38 +1780,38 @@ static int ccs_start_streaming(struct ccs_sensor *sensor)
>  
>  		rval = ccs_write(sensor, BINNING_TYPE, binning_type);
>  		if (rval < 0)
> -			goto out;
> +			goto err_pm_put;
>  
>  		binning_mode = 1;
>  	}
>  	rval = ccs_write(sensor, BINNING_MODE, binning_mode);
>  	if (rval < 0)
> -		goto out;
> +		goto err_pm_put;
>  
>  	/* Set up PLL */
>  	rval = ccs_pll_configure(sensor);
>  	if (rval)
> -		goto out;
> +		goto err_pm_put;
>  
>  	/* Analog crop start coordinates */
>  	rval = ccs_write(sensor, X_ADDR_START, sensor->pa_src.left);
>  	if (rval < 0)
> -		goto out;
> +		goto err_pm_put;
>  
>  	rval = ccs_write(sensor, Y_ADDR_START, sensor->pa_src.top);
>  	if (rval < 0)
> -		goto out;
> +		goto err_pm_put;
>  
>  	/* Analog crop end coordinates */
>  	rval = ccs_write(sensor, X_ADDR_END,
>  			 sensor->pa_src.left + sensor->pa_src.width - 1);
>  	if (rval < 0)
> -		goto out;
> +		goto err_pm_put;
>  
>  	rval = ccs_write(sensor, Y_ADDR_END,
>  			 sensor->pa_src.top + sensor->pa_src.height - 1);
>  	if (rval < 0)
> -		goto out;
> +		goto err_pm_put;
>  
>  	/*
>  	 * Output from pixel array, including blanking, is set using
> @@ -1824,22 +1824,22 @@ static int ccs_start_streaming(struct ccs_sensor *sensor)
>  		rval = ccs_write(sensor, DIGITAL_CROP_X_OFFSET,
>  				 sensor->scaler_sink.left);
>  		if (rval < 0)
> -			goto out;
> +			goto err_pm_put;
>  
>  		rval = ccs_write(sensor, DIGITAL_CROP_Y_OFFSET,
>  				 sensor->scaler_sink.top);
>  		if (rval < 0)
> -			goto out;
> +			goto err_pm_put;
>  
>  		rval = ccs_write(sensor, DIGITAL_CROP_IMAGE_WIDTH,
>  				 sensor->scaler_sink.width);
>  		if (rval < 0)
> -			goto out;
> +			goto err_pm_put;
>  
>  		rval = ccs_write(sensor, DIGITAL_CROP_IMAGE_HEIGHT,
>  				 sensor->scaler_sink.height);
>  		if (rval < 0)
> -			goto out;
> +			goto err_pm_put;
>  	}
>  
>  	/* Scaling */
> @@ -1847,20 +1847,20 @@ static int ccs_start_streaming(struct ccs_sensor *sensor)
>  	    != CCS_SCALING_CAPABILITY_NONE) {
>  		rval = ccs_write(sensor, SCALING_MODE, sensor->scaling_mode);
>  		if (rval < 0)
> -			goto out;
> +			goto err_pm_put;
>  
>  		rval = ccs_write(sensor, SCALE_M, sensor->scale_m);
>  		if (rval < 0)
> -			goto out;
> +			goto err_pm_put;
>  	}
>  
>  	/* Output size from sensor */
>  	rval = ccs_write(sensor, X_OUTPUT_SIZE, sensor->src_src.width);
>  	if (rval < 0)
> -		goto out;
> +		goto err_pm_put;
>  	rval = ccs_write(sensor, Y_OUTPUT_SIZE, sensor->src_src.height);
>  	if (rval < 0)
> -		goto out;
> +		goto err_pm_put;
>  
>  	if (CCS_LIM(sensor, FLASH_MODE_CAPABILITY) &
>  	    (CCS_FLASH_MODE_CAPABILITY_SINGLE_STROBE |
> @@ -1869,18 +1869,18 @@ static int ccs_start_streaming(struct ccs_sensor *sensor)
>  	    sensor->hwcfg.strobe_setup->trigger != 0) {
>  		rval = ccs_setup_flash_strobe(sensor);
>  		if (rval)
> -			goto out;
> +			goto err_pm_put;
>  	}
>  
>  	rval = ccs_call_quirk(sensor, pre_streamon);
>  	if (rval) {
>  		dev_err(&client->dev, "pre_streamon quirks failed\n");
> -		goto out;
> +		goto err_pm_put;
>  	}
>  
>  	rval = ccs_write(sensor, MODE_SELECT, CCS_MODE_SELECT_STREAMING);
>  
> -out:
> +err_pm_put:
>  	mutex_unlock(&sensor->mutex);
>  
>  	return rval;

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 27/46] media: uapi: ccs: Add media bus code for MIPI CCS embedded data

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Tue, Apr 16, 2024 at 10:33:00PM +0300, Sakari Ailus wrote:
> Add new MIPI CCS embedded data media bus code
> (MEDIA_BUS_FMT_CCS_EMBEDDED).
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> Reviewed-by: Julien Massot <julien.massot@collabora.com>

There's still an open question on how to document embedded data formats
for sensors compatible with level 2 only. The discussion is ongoing and
may affect this patch, but the changes can be implemented later, so

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> ---
>  .../media/v4l/subdev-formats.rst              | 32 +++++++++++++++++++
>  include/uapi/linux/media-bus-format.h         |  3 ++
>  2 files changed, 35 insertions(+)
> 
> diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> index 0547f2733ee3..fa181ce8f48c 100644
> --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
> +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> @@ -8316,6 +8316,10 @@ content is more or less device specific but the data is transmitted and received
>  by multiple devices that do not process the data in any way, simply writing
>  it to system memory for processing in software at the end of the pipeline.
>  
> +The exact format of the data generated by the device is reported on the internal
> +source pad of the originating sub-device, using one of the more specific
> +metadata formats such as MEDIA_BUS_FMT_CCS_EMBEDDED.
> +
>  "b" in an array cell signifies a byte of data, followed by the number of the bit
>  and finally the bit number in subscript. "x" indicates a padding bit.
>  
> @@ -8562,3 +8566,31 @@ and finally the bit number in subscript. "x" indicates a padding bit.
>        - x
>        - x
>        - x
> +
> +.. _MEDIA-BUS-FMT-CCS-EMBEDDED:
> +
> +MIPI CCS Embedded Data Formats
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +`MIPI CCS <https://www.mipi.org/specifications/camera-command-set>`_ defines a
> +metadata format for sensor embedded data, which is used to store the register
> +configuration used for capturing a given frame. The format is defined in the CCS
> +specification. The media bus code for this format is
> +``MEDIA_BUS_FMT_CCS_EMBEDDED``.
> +
> +The CCS embedded data format definition includes three levels:
> +
> +1. Padding within CSI-2 bus :ref:`Data Unit <media-glossary-data-unit>` as
> +   documented in the MIPI CCS specification.
> +
> +2. The tagged data format as documented in the MIPI CCS specification.
> +
> +3. Register addresses and register documentation as documented in the MIPI CCS
> +   specification.
> +
> +The format definition shall be used only by devices that fulfill all three
> +levels above.
> +
> +This mbus code are only used for "2-byte simplified tagged data format" (code
> +0xa) but their use may be extended further in the future, to cover other CCS
> +embedded data format codes.
> diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
> index d4c1d991014b..03f7e9ab517b 100644
> --- a/include/uapi/linux/media-bus-format.h
> +++ b/include/uapi/linux/media-bus-format.h
> @@ -183,4 +183,7 @@
>  #define MEDIA_BUS_FMT_META_20			0x8006
>  #define MEDIA_BUS_FMT_META_24			0x8007
>  
> +/* Specific metadata formats. Next is 0x9002. */
> +#define MEDIA_BUS_FMT_CCS_EMBEDDED		0x9001
> +
>  #endif /* __LINUX_MEDIA_BUS_FORMAT_H */

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 28/46] media: Documentation: Document non-CCS use of CCS embedded data format

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Tue, Apr 16, 2024 at 10:33:01PM +0300, Sakari Ailus wrote:
> The CCS embedded data format has multiple aspects (packing, encoding and
> the rest, including register addresses and semantics). Explicitly allow
> non-compliant embedded data to use the two former to reduce redundant
> documentation.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> ---
>  .../userspace-api/media/drivers/camera-sensor.rst     | 11 +++++++++++
>  1 file changed, 11 insertions(+)
> 
> diff --git a/Documentation/userspace-api/media/drivers/camera-sensor.rst b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> index 9f3b0da3ad0d..dc415b8f6c8e 100644
> --- a/Documentation/userspace-api/media/drivers/camera-sensor.rst
> +++ b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> @@ -123,3 +123,14 @@ In general, changing the embedded data format from the driver-configured values
>  is not supported. The height of the metadata is device-specific and the width
>  is that (or less of that) of the image width, as configured on the pixel data
>  stream.
> +
> +CCS and non-CCS embedded data
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Embedded data which is fully compliant with CCS definitions uses ``CCS embedded
> +data format <MEDIA-BUS-FMT-CCS-EMBEDDED>`` media bus code (level
> +3). Device-specific embedded data compliant with either MIPI CCS embedded data
> +levels 1 or 2 only shall not use CCS embedded data mbus code, but may refer to
> +CCS embedded data documentation with the level of conformance specified and omit
> +documenting these aspects of the format. The rest of the device-specific
> +embedded data format is documented in the context of the data format itself.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 29/46] media: Documentation: ccs: Document routing

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Tue, Apr 16, 2024 at 10:33:02PM +0300, Sakari Ailus wrote:
> Document which routes are available for the CCS driver (source) sub-device
> and what configuration are possible.
> 
> Also update copyright.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../userspace-api/media/drivers/ccs.rst       | 38 ++++++++++++++++++-
>  .../media/v4l/subdev-formats.rst              |  2 +
>  2 files changed, 39 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/userspace-api/media/drivers/ccs.rst b/Documentation/userspace-api/media/drivers/ccs.rst
> index 03015b33d5ab..53890ac54dab 100644
> --- a/Documentation/userspace-api/media/drivers/ccs.rst
> +++ b/Documentation/userspace-api/media/drivers/ccs.rst
> @@ -111,4 +111,40 @@ than in the centre.
>  Shading correction needs to be enabled for luminance correction level to have an
>  effect.
>  
> -**Copyright** |copy| 2020 Intel Corporation
> +.. _media-ccs-routes:
> +
> +Routes
> +------
> +
> +The CCS driver implements one or two :ref:`routes <subdev-routing>` in
> +its source sub-device (scaler sub-device if exists for the device, otherwise

s/if exists/if it exists/

> +binner) depending on whether the sensor supports embedded data. (All CCS
> +compliant sensors do but the CCS driver supports preceding standards that did
> +not require embedded data support, too.)
> +
> +The first route of the CCS source sub-device is for pixel data (sink pad
> +0/stream 0 -> source pad 1/stream 0) and the second one is for embedded data
> +(internal sink pad 2/stream 0 -> source pad 1/stream 1).
> +
> +Embedded data
> +~~~~~~~~~~~~~
> +
> +MIPI CCS supports generation of camera sensor embedded data. The media bus code
> +used for this format on the internal sink pad is
> +:ref:`MEDIA_BUS_FMT_CCS_EMBEDDDED <MEDIA-BUS-FMT-CCS-EMBEDDED>`.
> +
> +The bit depth of the CCS pixel data affects how the sensor will output the
> +embedded data, adding padding to align with CSI-2 bus :ref:`Data Units
> +<media-glossary-data-unit>` for that particular bit depth. This is indicated by
> +the generic metadata format on the sensor's source sub-device's source pad.

That's a bit hard to parse, maybe "on the source pad of the sensor's
source sub-device" ?

> +
> +Devices supporting embedded data for bit depths greater than or equal to 16 may
> +support more dense packing or legacy single metadata byte per data unit, or both
> +of these. The supported embedded data formats can be enumerated and configured
> +on stream 1 of the source pad (1) of the CCS source sub-device.
> +
> +The use of the denser packing results in embedded data lines being longer than
> +the pixel data in data units since the data units are smaller. In bytes the
> +embedded data lines are still not longer than the image data lines.
> +
> +**Copyright** |copy| 2020, 2023 Intel Corporation

2024 ?

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> index fa181ce8f48c..a04756092238 100644
> --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
> +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> @@ -8594,3 +8594,5 @@ levels above.
>  This mbus code are only used for "2-byte simplified tagged data format" (code
>  0xa) but their use may be extended further in the future, to cover other CCS
>  embedded data format codes.
> +
> +Also see :ref:`CCS driver documentation <media-ccs-routes>`.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 30/46] media: ccs: Add support for embedded data stream

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Tue, Apr 16, 2024 at 10:33:03PM +0300, Sakari Ailus wrote:
> Add support for embedded data stream, in UAPI and frame descriptor.
> 
> This patch adds also a new embedded data pad (2) to the source sub-device.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  drivers/media/i2c/ccs/ccs-core.c | 372 +++++++++++++++++++++++++++++--
>  drivers/media/i2c/ccs/ccs.h      |  18 +-
>  2 files changed, 362 insertions(+), 28 deletions(-)
> 
> diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
> index 3ca2415fca3b..ba629eafec43 100644
> --- a/drivers/media/i2c/ccs/ccs-core.c
> +++ b/drivers/media/i2c/ccs/ccs-core.c
> @@ -1903,6 +1903,13 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
>  	if (rval < 0)
>  		goto err_pm_put;
>  
> +	/* Configure embedded data */
> +	if (sensor->csi_format->compressed >= 16) {
> +		rval = ccs_write(sensor, EMB_DATA_CTRL, sensor->emb_data_ctrl);
> +		if (rval < 0)
> +			goto err_pm_put;
> +	}
> +
>  	if (CCS_LIM(sensor, FLASH_MODE_CAPABILITY) &
>  	    (CCS_FLASH_MODE_CAPABILITY_SINGLE_STROBE |
>  	     SMIAPP_FLASH_MODE_CAPABILITY_MULTIPLE_STROBE) &&
> @@ -2022,6 +2029,57 @@ static const struct ccs_csi_data_format
>  	return sensor->csi_format;
>  }
>  
> +#define CCS_EMBEDDED_CODE_DEPTH(depth, half_depth)			\
> +	depth,								\
> +	CCS_EMB_DATA_CAPABILITY_TWO_BYTES_PER_RAW##depth,		\
> +	CCS_EMB_DATA_CAPABILITY_NO_ONE_BYTE_PER_RAW##depth,		\
> +	CCS_EMB_DATA_CTRL_RAW##half_depth##_PACKING_FOR_RAW##depth,	\
> +	MEDIA_BUS_FMT_META_##half_depth,				\
> +	MEDIA_BUS_FMT_META_##depth,					\
> +
> +static const struct ccs_embedded_code {
> +	u8 depth;
> +	u8 cap_two_bytes_per_sample;
> +	u8 cap_no_legacy;
> +	u8 ctrl;
> +	u32 code_two_bytes;
> +	u32 code_legacy;
> +} ccs_embedded_codes[] = {
> +	{ CCS_EMBEDDED_CODE_DEPTH(16, 8) },
> +	{ CCS_EMBEDDED_CODE_DEPTH(20, 10) },
> +	{ CCS_EMBEDDED_CODE_DEPTH(24, 12) },
> +};
> +
> +static const struct ccs_embedded_code *ccs_embedded_code(unsigned int bpp)
> +{
> +	unsigned int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(ccs_embedded_codes); i++)
> +		if (ccs_embedded_codes[i].depth == bpp)
> +			return ccs_embedded_codes + i;
> +
> +	WARN_ON(1);
> +
> +	return ccs_embedded_codes;
> +}
> +
> +static u32
> +ccs_default_embedded_code(struct ccs_sensor *sensor,
> +			  const struct ccs_embedded_code *embedded_code)
> +{
> +	if (CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
> +	    BIT(embedded_code->cap_two_bytes_per_sample))
> +		return embedded_code->code_two_bytes;
> +
> +	if (!(CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
> +	      BIT(embedded_code->cap_no_legacy)))
> +		return embedded_code->code_legacy;
> +
> +	WARN_ON(1);
> +
> +	return embedded_code->code_legacy;
> +}
> +
>  static int ccs_enum_mbus_code(struct v4l2_subdev *subdev,
>  			      struct v4l2_subdev_state *sd_state,
>  			      struct v4l2_subdev_mbus_code_enum *code)
> @@ -2037,6 +2095,69 @@ static int ccs_enum_mbus_code(struct v4l2_subdev *subdev,
>  	dev_err(&client->dev, "subdev %s, pad %u, index %u\n",
>  		subdev->name, code->pad, code->index);
>  
> +	if (subdev == &sensor->src->sd) {
> +		if (code->pad == CCS_PAD_META) {
> +			if (code->index)
> +				goto out;
> +
> +			code->code = MEDIA_BUS_FMT_CCS_EMBEDDED;
> +
> +			rval = 0;
> +			goto out;
> +		}
> +		if (code->stream == CCS_STREAM_META) {
> +			struct v4l2_mbus_framefmt *pix_fmt =
> +				v4l2_subdev_state_get_format(sd_state,
> +							     CCS_PAD_SRC,
> +							     CCS_STREAM_PIXEL);
> +			const struct ccs_csi_data_format *csi_format =
> +				ccs_validate_csi_data_format(sensor,
> +							     pix_fmt->code);
> +			unsigned int i = 0;
> +			u32 codes[2];
> +
> +			switch (csi_format->compressed) {
> +			case 8:
> +				codes[i++] = MEDIA_BUS_FMT_META_8;
> +				break;
> +			case 10:
> +				codes[i++] = MEDIA_BUS_FMT_META_10;
> +				break;
> +			case 12:
> +				codes[i++] = MEDIA_BUS_FMT_META_12;
> +				break;
> +			case 14:
> +				codes[i++] = MEDIA_BUS_FMT_META_14;
> +				break;
> +			case 16:
> +			case 20:
> +			case 24: {
> +				const struct ccs_embedded_code *embedded_code =
> +					ccs_embedded_code(csi_format->compressed);
> +
> +				if (CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
> +				    BIT(embedded_code->cap_two_bytes_per_sample))
> +					codes[i++] =
> +						embedded_code->code_two_bytes;
> +
> +				if (!(CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
> +				      BIT(embedded_code->cap_no_legacy)))
> +					codes[i++] = embedded_code->code_legacy;
> +				break;
> +			}
> +			default:
> +				WARN_ON(1);
> +			}
> +
> +			if (WARN_ON(i > ARRAY_SIZE(codes)) || code->index >= i)
> +				goto out;
> +
> +			code->code = codes[code->index];
> +			rval = 0;
> +			goto out;
> +		}
> +	}
> +
>  	if (subdev != &sensor->src->sd || code->pad != CCS_PAD_SRC) {
>  		if (code->index)
>  			goto out;
> @@ -2079,8 +2200,11 @@ static int __ccs_get_format(struct v4l2_subdev *subdev,
>  			    struct v4l2_subdev_state *sd_state,
>  			    struct v4l2_subdev_format *fmt)
>  {
> -	fmt->format = *v4l2_subdev_state_get_format(sd_state, fmt->pad);
> -	fmt->format.code = __ccs_get_mbus_code(subdev, fmt->pad);
> +	fmt->format = *v4l2_subdev_state_get_format(sd_state, fmt->pad,
> +						    fmt->stream);
> +
> +	if (fmt->pad != CCS_PAD_META && fmt->stream != CCS_STREAM_META)
> +		fmt->format.code = __ccs_get_mbus_code(subdev, fmt->pad);
>  
>  	return 0;
>  }
> @@ -2110,10 +2234,11 @@ static void ccs_get_crop_compose(struct v4l2_subdev *subdev,
>  	if (crops)
>  		for (i = 0; i < subdev->entity.num_pads; i++)
>  			crops[i] =
> -				v4l2_subdev_state_get_crop(sd_state, i);
> +				v4l2_subdev_state_get_crop(sd_state, i,
> +							   CCS_STREAM_PIXEL);
>  	if (comps)
> -		*comps = v4l2_subdev_state_get_compose(sd_state,
> -						       ssd->sink_pad);
> +		*comps = v4l2_subdev_state_get_compose(sd_state, ssd->sink_pad,
> +						       CCS_STREAM_PIXEL);

This hunk and the next one could have been moved to the patch that
introduced CCS_STREAM_PIXEL. Same for the change in __ccs_init_state()
below.

>  }
>  
>  /* Changes require propagation only on sink pad. */
> @@ -2146,7 +2271,8 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
>  		fallthrough;
>  	case V4L2_SEL_TGT_COMPOSE:
>  		*crops[CCS_PAD_SRC] = *comp;
> -		fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC);
> +		fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC,
> +						   CCS_STREAM_PIXEL);
>  		fmt->width = comp->width;
>  		fmt->height = comp->height;
>  		if (which == V4L2_SUBDEV_FORMAT_ACTIVE && ssd == sensor->src)
> @@ -2210,6 +2336,83 @@ static int ccs_set_format_source(struct v4l2_subdev *subdev,
>  	return ccs_pll_update(sensor);
>  }
>  
> +static int ccs_set_format_meta(struct v4l2_subdev *subdev,
> +			       struct v4l2_subdev_state *sd_state,
> +			       struct v4l2_mbus_framefmt *fmt)
> +{
> +	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
> +	const struct ccs_csi_data_format *csi_format;
> +	struct v4l2_mbus_framefmt *pix_fmt;
> +	struct v4l2_mbus_framefmt *meta_fmt;
> +	struct v4l2_mbus_framefmt *meta_out_fmt;
> +	u32 code;
> +
> +	pix_fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC,
> +					       CCS_STREAM_PIXEL);
> +	meta_fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_META, 0);
> +	meta_out_fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC,
> +						    CCS_STREAM_META);
> +
> +	code = fmt ? fmt->code : meta_out_fmt->code;

When this function is called from __ccs_init_state(), fmt will be NULL,
and meta_out_fmt will be uninitialized. Is this intended ?

> +
> +	meta_out_fmt->width = meta_fmt->width = pix_fmt->width;
> +	meta_out_fmt->height = meta_fmt->height =
> +		sensor->embedded_end - sensor->embedded_start;
> +	meta_fmt->code = MEDIA_BUS_FMT_CCS_EMBEDDED;
> +
> +	csi_format = ccs_validate_csi_data_format(sensor, pix_fmt->code);
> +
> +	switch (csi_format->compressed) {
> +	case 8:
> +		meta_out_fmt->code = MEDIA_BUS_FMT_META_8;
> +		break;
> +	case 10:
> +		meta_out_fmt->code = MEDIA_BUS_FMT_META_10;
> +		break;
> +	case 12:
> +		meta_out_fmt->code = MEDIA_BUS_FMT_META_12;
> +		break;
> +	case 14:
> +		meta_out_fmt->code = MEDIA_BUS_FMT_META_14;
> +		break;
> +	case 16:
> +	case 20:
> +	case 24: {
> +		const struct ccs_embedded_code *embedded_code;
> +
> +		embedded_code = ccs_embedded_code(csi_format->compressed);
> +		meta_out_fmt->code =
> +			ccs_default_embedded_code(sensor, embedded_code);
> +
> +		if (!(CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
> +		      BIT(embedded_code->cap_no_legacy)) &&
> +		    code == embedded_code->code_legacy) {
> +			meta_out_fmt->code = embedded_code->code_legacy;
> +			sensor->emb_data_ctrl = 0;
> +		}
> +
> +		if (CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
> +		    BIT(embedded_code->cap_two_bytes_per_sample) &&
> +		    code == embedded_code->code_two_bytes) {
> +			meta_out_fmt->code = embedded_code->code_two_bytes;
> +			sensor->emb_data_ctrl = embedded_code->ctrl;
> +			meta_fmt->width <<= 1;
> +			meta_out_fmt->width <<= 1;
> +		}
> +
> +		break;
> +	}
> +	default:
> +		WARN_ON(1);
> +		return 0;
> +	}
> +
> +	if (fmt)
> +		*fmt = *meta_out_fmt;
> +
> +	return 0;
> +}
> +
>  static int ccs_set_format(struct v4l2_subdev *subdev,
>  			  struct v4l2_subdev_state *sd_state,
>  			  struct v4l2_subdev_format *fmt)
> @@ -2218,12 +2421,25 @@ static int ccs_set_format(struct v4l2_subdev *subdev,
>  	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
>  	struct v4l2_rect *crops[CCS_PADS];
>  
> +	if (subdev == &sensor->src->sd && fmt->pad == CCS_PAD_META)

You could also write

	if (ssd == sensor->src && fmt->pad == CCS_PAD_META)

Same below.

> +		return ccs_get_format(subdev, sd_state, fmt);
> +
>  	mutex_lock(&sensor->mutex);

Is this needed, shouldn't the state lock be enough ?

>  
> +	if (subdev == &sensor->src->sd && fmt->stream == CCS_STREAM_META) {
> +		ccs_set_format_meta(subdev, sd_state, &fmt->format);
> +
> +		mutex_unlock(&sensor->mutex);
> +
> +		return 0;
> +	}
> +
>  	if (fmt->pad == ssd->source_pad) {
>  		int rval;
>  
>  		rval = ccs_set_format_source(subdev, sd_state, fmt);
> +		if (sensor->embedded_start != sensor->embedded_end)

A ccs_sensor_has_embedded_data() (name bikeshedding allowed) inline
helper could be nice to replace this manual check could be nice, as you
do the same in many locations below.

> +			ccs_set_format_meta(subdev, sd_state, NULL);

This doesn't seem correct, you shouldn't set the metadata format on
subdevs that are not the source subdev.

A comment to explain how the metadata format is propagated would also be
useful.

>  
>  		mutex_unlock(&sensor->mutex);
>  
> @@ -2498,6 +2714,12 @@ static int ccs_sel_supported(struct v4l2_subdev *subdev,
>  	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
>  	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
>  
> +	if (sel->stream != CCS_STREAM_PIXEL)
> +		return -EINVAL;
> +
> +	if (sel->pad == CCS_PAD_META)
> +		return -EINVAL;
> +
>  	/* We only implement crop in three places. */
>  	switch (sel->target) {
>  	case V4L2_SEL_TGT_CROP:
> @@ -2542,7 +2764,8 @@ static int ccs_set_crop(struct v4l2_subdev *subdev,
>  
>  	if (sel->pad == ssd->sink_pad) {
>  		struct v4l2_mbus_framefmt *mfmt =
> -			v4l2_subdev_state_get_format(sd_state, sel->pad);
> +			v4l2_subdev_state_get_format(sd_state, sel->pad,
> +						     CCS_STREAM_PIXEL);
>  
>  		src_size.width = mfmt->width;
>  		src_size.height = mfmt->height;
> @@ -2603,7 +2826,9 @@ static int ccs_get_selection(struct v4l2_subdev *subdev,
>  		} else if (sel->pad == ssd->sink_pad) {
>  			struct v4l2_mbus_framefmt *sink_fmt =
>  				v4l2_subdev_state_get_format(sd_state,
> -							     ssd->sink_pad);
> +							     ssd->sink_pad,
> +							     CCS_STREAM_PIXEL);
> +
>  			sel->r.top = sel->r.left = 0;
>  			sel->r.width = sink_fmt->width;
>  			sel->r.height = sink_fmt->height;
> @@ -2686,6 +2911,14 @@ static int ccs_get_frame_desc(struct v4l2_subdev *subdev, unsigned int pad,
>  	entry++;
>  	desc->num_entries++;
>  
> +	if (sensor->embedded_start != sensor->embedded_end) {
> +		entry->pixelcode = MEDIA_BUS_FMT_CCS_EMBEDDED;

I think you need to report the generic pixel code here.

> +		entry->stream = CCS_STREAM_META;
> +		entry->bus.csi2.dt = MIPI_CSI2_DT_EMBEDDED_8B;
> +		entry++;
> +		desc->num_entries++;
> +	}
> +
>  	return 0;
>  }
>  
> @@ -3004,6 +3237,8 @@ static void ccs_cleanup(struct ccs_sensor *sensor)
>  	ccs_free_controls(sensor);
>  }
>  
> +static const struct v4l2_subdev_internal_ops ccs_internal_ops;
> +
>  static int ccs_init_subdev(struct ccs_sensor *sensor,
>  			   struct ccs_subdev *ssd, const char *name,
>  			   unsigned short num_pads, u32 function,
> @@ -3016,15 +3251,18 @@ static int ccs_init_subdev(struct ccs_sensor *sensor,
>  	if (!ssd)
>  		return 0;
>  
> -	if (ssd != sensor->src)
> +	if (ssd != sensor->src) {
>  		v4l2_subdev_init(&ssd->sd, &ccs_ops);
> +		ssd->sd.internal_ops = &ccs_internal_ops;
> +	}
>  
>  	ssd->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
>  	ssd->sd.entity.function = function;
>  	ssd->sensor = sensor;
>  
>  	ssd->npads = num_pads;
> -	ssd->source_pad = num_pads - 1;
> +	ssd->source_pad =
> +		ssd == sensor->pixel_array ? CCS_PA_PAD_SRC : CCS_PAD_SRC;
>  
>  	v4l2_i2c_subdev_set_name(&ssd->sd, client, sensor->minfo.name, name);
>  
> @@ -3038,6 +3276,10 @@ static int ccs_init_subdev(struct ccs_sensor *sensor,
>  		ssd->sd.owner = THIS_MODULE;
>  		ssd->sd.dev = &client->dev;
>  		v4l2_set_subdevdata(&ssd->sd, client);
> +	} else {
> +		ssd->sd.flags |= V4L2_SUBDEV_FL_STREAMS;
> +		ssd->pads[CCS_PAD_META].flags =
> +			MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL;
>  	}
>  
>  	rval = media_entity_pads_init(&ssd->sd.entity, ssd->npads, ssd->pads);
> @@ -3055,21 +3297,19 @@ static int ccs_init_subdev(struct ccs_sensor *sensor,
>  	return 0;
>  }
>  
> -static int ccs_init_state(struct v4l2_subdev *sd,
> -			  struct v4l2_subdev_state *sd_state)
> +static int __ccs_init_state(struct v4l2_subdev *sd,
> +			    struct v4l2_subdev_state *sd_state)
>  {
>  	struct ccs_subdev *ssd = to_ccs_subdev(sd);
>  	struct ccs_sensor *sensor = ssd->sensor;
>  	unsigned int pad = ssd == sensor->pixel_array ?
>  		CCS_PA_PAD_SRC : CCS_PAD_SINK;
>  	struct v4l2_mbus_framefmt *fmt =
> -		v4l2_subdev_state_get_format(sd_state, pad);
> +		v4l2_subdev_state_get_format(sd_state, pad, CCS_STREAM_PIXEL);
>  	struct v4l2_rect *crop =
> -		v4l2_subdev_state_get_crop(sd_state, pad);
> +		v4l2_subdev_state_get_crop(sd_state, pad, CCS_STREAM_PIXEL);
>  	bool is_active = !sd->active_state || sd->active_state == sd_state;
>  
> -	mutex_lock(&sensor->mutex);
> -
>  	ccs_get_native_size(ssd, crop);
>  
>  	fmt->width = crop->width;
> @@ -3081,20 +3321,78 @@ static int ccs_init_state(struct v4l2_subdev *sd,
>  		if (is_active)
>  			sensor->pa_src = *crop;
>  
> -		mutex_unlock(&sensor->mutex);
>  		return 0;
>  	}
>  
> -	fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC);
> +	fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC,
> +					   CCS_STREAM_PIXEL);
>  	fmt->code = ssd == sensor->src ?
>  		sensor->csi_format->code : sensor->internal_csi_format->code;
>  	fmt->field = V4L2_FIELD_NONE;
>  
>  	ccs_propagate(sd, sd_state, is_active, V4L2_SEL_TGT_CROP);
>  
> +	return 0;
> +}
> +
> +static int ccs_init_state(struct v4l2_subdev *sd,
> +			  struct v4l2_subdev_state *sd_state)
> +{
> +	struct ccs_subdev *ssd = to_ccs_subdev(sd);
> +	struct ccs_sensor *sensor = ssd->sensor;
> +	int rval;
> +
> +	mutex_lock(&sensor->mutex);
> +	rval = __ccs_init_state(sd, sd_state);
>  	mutex_unlock(&sensor->mutex);
>  
> -	return 0;
> +	return rval;
> +}
> +
> +static int ccs_src_init_state(struct v4l2_subdev *sd,
> +			      struct v4l2_subdev_state *sd_state)
> +{
> +	struct v4l2_subdev_route routes[] = {
> +		{
> +			.sink_pad = CCS_PAD_SINK,
> +			.source_pad = CCS_PAD_SRC,
> +			.source_stream = CCS_STREAM_PIXEL,
> +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> +		}, {
> +			.sink_pad = CCS_PAD_META,
> +			.source_pad = CCS_PAD_SRC,
> +			.source_stream = CCS_STREAM_META,
> +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> +		}
> +	};
> +	struct v4l2_subdev_krouting routing = {
> +		.routes = routes,
> +		.num_routes = 1,
> +	};
> +	struct ccs_subdev *ssd = to_ccs_subdev(sd);
> +	struct ccs_sensor *sensor = ssd->sensor;
> +	int rval;
> +
> +	mutex_lock(&sensor->mutex);

Is this needed, shouldn't the state lock be enough ?

> +
> +	if (sensor->embedded_start != sensor->embedded_end)
> +		routing.num_routes++;
> +
> +	rval = v4l2_subdev_set_routing(sd, sd_state, &routing);
> +	if (rval)
> +		goto out;
> +
> +	rval = __ccs_init_state(sd, sd_state);
> +	if (rval)
> +		goto out;
> +
> +	if (sensor->embedded_start != sensor->embedded_end)
> +		ccs_set_format_meta(sd, sd_state, NULL);
> +
> +out:
> +	mutex_unlock(&sensor->mutex);
> +
> +	return rval;
>  }
>  
>  static const struct v4l2_subdev_video_ops ccs_video_ops = {
> @@ -3109,6 +3407,14 @@ static const struct v4l2_subdev_pad_ops ccs_pad_ops = {
>  	.set_fmt = ccs_set_format,
>  	.get_selection = ccs_get_selection,
>  	.set_selection = ccs_set_selection,
> +};
> +
> +static const struct v4l2_subdev_pad_ops ccs_src_pad_ops = {
> +	.enum_mbus_code = ccs_enum_mbus_code,
> +	.get_fmt = ccs_get_format,

I'm surprised you need to implement .get_fmt(). The
v4l2_subdev_get_fmt() helper should have been enough.

> +	.set_fmt = ccs_set_format,
> +	.get_selection = ccs_get_selection,
> +	.set_selection = ccs_set_selection,
>  	.enable_streams = ccs_enable_streams,
>  	.disable_streams = ccs_disable_streams,
>  	.get_frame_desc = ccs_get_frame_desc,
> @@ -3125,12 +3431,22 @@ static const struct v4l2_subdev_ops ccs_ops = {
>  	.sensor = &ccs_sensor_ops,
>  };
>  
> +static const struct v4l2_subdev_ops ccs_src_ops = {
> +	.video = &ccs_video_ops,
> +	.pad = &ccs_src_pad_ops,
> +	.sensor = &ccs_sensor_ops,
> +};
> +
>  static const struct media_entity_operations ccs_entity_ops = {
>  	.link_validate = v4l2_subdev_link_validate,
>  };
>  
> -static const struct v4l2_subdev_internal_ops ccs_internal_src_ops = {
> +static const struct v4l2_subdev_internal_ops ccs_internal_ops = {
>  	.init_state = ccs_init_state,
> +};
> +
> +static const struct v4l2_subdev_internal_ops ccs_src_internal_ops = {
> +	.init_state = ccs_src_init_state,
>  	.registered = ccs_registered,
>  	.unregistered = ccs_unregistered,
>  };
> @@ -3281,8 +3597,8 @@ static int ccs_probe(struct i2c_client *client)
>  
>  	sensor->src = &sensor->ssds[sensor->ssds_used];
>  
> -	v4l2_i2c_subdev_init(&sensor->src->sd, client, &ccs_ops);
> -	sensor->src->sd.internal_ops = &ccs_internal_src_ops;
> +	v4l2_i2c_subdev_init(&sensor->src->sd, client, &ccs_src_ops);
> +	sensor->src->sd.internal_ops = &ccs_src_internal_ops;
>  
>  	sensor->regulators = devm_kcalloc(&client->dev,
>  					  ARRAY_SIZE(ccs_regulators),
> @@ -3553,12 +3869,20 @@ static int ccs_probe(struct i2c_client *client)
>  		goto out_cleanup;
>  	}
>  
> -	rval = ccs_init_subdev(sensor, sensor->scaler, " scaler", 2,
> +	rval = ccs_init_subdev(sensor, sensor->scaler, " scaler",
> +			       sensor->ssds_used != CCS_SUBDEVS ?
> +			       CCS_PADS_NOMETA :
> +			       sensor->embedded_start == sensor->embedded_end ?
> +			       CCS_PADS_NOMETA : CCS_PADS,
>  			       MEDIA_ENT_F_PROC_VIDEO_SCALER,
>  			       "ccs scaler mutex", &scaler_lock_key);
>  	if (rval)
>  		goto out_cleanup;
> -	rval = ccs_init_subdev(sensor, sensor->binner, " binner", 2,
> +	rval = ccs_init_subdev(sensor, sensor->binner, " binner",
> +			       sensor->ssds_used == CCS_SUBDEVS ?
> +			       CCS_PADS_NOMETA :
> +			       sensor->embedded_start == sensor->embedded_end ?
> +			       CCS_PADS_NOMETA : CCS_PADS,
>  			       MEDIA_ENT_F_PROC_VIDEO_SCALER,
>  			       "ccs binner mutex", &binner_lock_key);
>  	if (rval)
> diff --git a/drivers/media/i2c/ccs/ccs.h b/drivers/media/i2c/ccs/ccs.h
> index 90b442a3d53e..477b2fb99aa0 100644
> --- a/drivers/media/i2c/ccs/ccs.h
> +++ b/drivers/media/i2c/ccs/ccs.h
> @@ -173,11 +173,18 @@ struct ccs_csi_data_format {
>  #define CCS_SUBDEVS			3
>  
>  #define CCS_PA_PAD_SRC			0
> -#define CCS_PAD_SINK			0
> -#define CCS_PAD_SRC			1
> -#define CCS_PADS			2
> +enum {
> +	CCS_PAD_SINK,
> +	CCS_PAD_SRC,
> +	CCS_PAD_META,
> +	CCS_PADS_NOMETA = CCS_PAD_META,

This doesn't seem to improve readability of the code above :-S

> +	CCS_PADS,
> +};
>  
> -#define CCS_STREAM_PIXEL		0
> +enum {
> +	CCS_STREAM_PIXEL,
> +	CCS_STREAM_META,
> +};
>  
>  struct ccs_binning_subtype {
>  	u8 horizontal:4;
> @@ -228,6 +235,9 @@ struct ccs_sensor {
>  	int default_pixel_order;
>  	struct ccs_data_container sdata, mdata;
>  
> +	u32 embedded_mbus_code;

Not used.

> +	u8 emb_data_ctrl;

The general direction I'd like to take with v4l2_subdev_state is to
avoid storing state information in the device private structure. Could
this be dropped and computed in ccs_enable_streams() instead ?

> +
>  	u8 binning_horizontal;
>  	u8 binning_vertical;
>  

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 31/46] media: ccs: Remove ccs_get_crop_compose helper

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Tue, Apr 16, 2024 at 10:33:04PM +0300, Sakari Ailus wrote:
> As it's now easier to obtain the necessary information on crop and compose
> rectangles after moving to sub-device state, remove the
> ccs_get_crop_compose helper.
> 
> Also remove the comp arguments of the compose goodness calculators and
> make related local variables and function arguments const where
> applicable.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> ---
>  drivers/media/i2c/ccs/ccs-core.c | 164 ++++++++++++++-----------------
>  1 file changed, 72 insertions(+), 92 deletions(-)
> 
> diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
> index ba629eafec43..299b5ac6538c 100644
> --- a/drivers/media/i2c/ccs/ccs-core.c
> +++ b/drivers/media/i2c/ccs/ccs-core.c
> @@ -2223,24 +2223,6 @@ static int ccs_get_format(struct v4l2_subdev *subdev,
>  	return rval;
>  }
>  
> -static void ccs_get_crop_compose(struct v4l2_subdev *subdev,
> -				 struct v4l2_subdev_state *sd_state,
> -				 struct v4l2_rect **crops,
> -				 struct v4l2_rect **comps)
> -{
> -	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
> -	unsigned int i;
> -
> -	if (crops)
> -		for (i = 0; i < subdev->entity.num_pads; i++)
> -			crops[i] =
> -				v4l2_subdev_state_get_crop(sd_state, i,
> -							   CCS_STREAM_PIXEL);
> -	if (comps)
> -		*comps = v4l2_subdev_state_get_compose(sd_state, ssd->sink_pad,
> -						       CCS_STREAM_PIXEL);
> -}
> -
>  /* Changes require propagation only on sink pad. */
>  static void ccs_propagate(struct v4l2_subdev *subdev,
>  			  struct v4l2_subdev_state *sd_state, int which,
> @@ -2248,15 +2230,17 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
>  {
>  	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
>  	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
> -	struct v4l2_rect *comp, *crops[CCS_PADS];
> +	struct v4l2_rect *comp, *crop;
>  	struct v4l2_mbus_framefmt *fmt;
>  
> -	ccs_get_crop_compose(subdev, sd_state, crops, &comp);
> -
> +	comp = v4l2_subdev_state_get_compose(sd_state, ssd->sink_pad,
> +					     CCS_STREAM_PIXEL);
>  	switch (target) {
>  	case V4L2_SEL_TGT_CROP:
> -		comp->width = crops[CCS_PAD_SINK]->width;
> -		comp->height = crops[CCS_PAD_SINK]->height;
> +		crop = v4l2_subdev_state_get_crop(sd_state, CCS_PAD_SINK,
> +						  CCS_STREAM_PIXEL);
> +		comp->width = crop->width;
> +		comp->height = crop->height;
>  		if (which == V4L2_SUBDEV_FORMAT_ACTIVE) {
>  			if (ssd == sensor->scaler) {
>  				sensor->scale_m = CCS_LIM(sensor, SCALER_N_MIN);
> @@ -2270,13 +2254,15 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
>  		}
>  		fallthrough;
>  	case V4L2_SEL_TGT_COMPOSE:
> -		*crops[CCS_PAD_SRC] = *comp;
> +		crop = v4l2_subdev_state_get_crop(sd_state, CCS_PAD_SRC,
> +						  CCS_STREAM_PIXEL);
> +		*crop = *comp;
>  		fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC,
>  						   CCS_STREAM_PIXEL);
>  		fmt->width = comp->width;
>  		fmt->height = comp->height;
>  		if (which == V4L2_SUBDEV_FORMAT_ACTIVE && ssd == sensor->src)
> -			sensor->src_src = *crops[CCS_PAD_SRC];
> +			sensor->src_src = *crop;
>  		break;
>  	default:
>  		WARN_ON_ONCE(1);
> @@ -2419,7 +2405,7 @@ static int ccs_set_format(struct v4l2_subdev *subdev,
>  {
>  	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
>  	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
> -	struct v4l2_rect *crops[CCS_PADS];
> +	struct v4l2_rect *crop;
>  
>  	if (subdev == &sensor->src->sd && fmt->pad == CCS_PAD_META)
>  		return ccs_get_format(subdev, sd_state, fmt);
> @@ -2461,12 +2447,13 @@ static int ccs_set_format(struct v4l2_subdev *subdev,
>  		      CCS_LIM(sensor, MIN_Y_OUTPUT_SIZE),
>  		      CCS_LIM(sensor, MAX_Y_OUTPUT_SIZE));
>  
> -	ccs_get_crop_compose(subdev, sd_state, crops, NULL);
> +	crop = v4l2_subdev_state_get_crop(sd_state, ssd->sink_pad,
> +					  CCS_STREAM_PIXEL);
>  
> -	crops[ssd->sink_pad]->left = 0;
> -	crops[ssd->sink_pad]->top = 0;
> -	crops[ssd->sink_pad]->width = fmt->format.width;
> -	crops[ssd->sink_pad]->height = fmt->format.height;
> +	crop->left = 0;
> +	crop->top = 0;
> +	crop->width = fmt->format.width;
> +	crop->height = fmt->format.height;
>  	ccs_propagate(subdev, sd_state, fmt->which, V4L2_SEL_TGT_CROP);
>  
>  	mutex_unlock(&sensor->mutex);
> @@ -2521,26 +2508,23 @@ static int scaling_goodness(struct v4l2_subdev *subdev, int w, int ask_w,
>  static void ccs_set_compose_binner(struct v4l2_subdev *subdev,
>  				   struct v4l2_subdev_state *sd_state,
>  				   struct v4l2_subdev_selection *sel,
> -				   struct v4l2_rect **crops,
> -				   struct v4l2_rect *comp)
> +				   const struct v4l2_rect *sink_crop)
>  {
>  	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
>  	unsigned int i;
>  	unsigned int binh = 1, binv = 1;
> -	int best = scaling_goodness(
> -		subdev,
> -		crops[CCS_PAD_SINK]->width, sel->r.width,
> -		crops[CCS_PAD_SINK]->height, sel->r.height, sel->flags);
> +	int best = scaling_goodness(subdev, sink_crop->width, sel->r.width,
> +				    sink_crop->height, sel->r.height,
> +				    sel->flags);
>  
>  	for (i = 0; i < sensor->nbinning_subtypes; i++) {
> -		int this = scaling_goodness(
> -			subdev,
> -			crops[CCS_PAD_SINK]->width
> -			/ sensor->binning_subtypes[i].horizontal,
> -			sel->r.width,
> -			crops[CCS_PAD_SINK]->height
> -			/ sensor->binning_subtypes[i].vertical,
> -			sel->r.height, sel->flags);
> +		int this = scaling_goodness(subdev,
> +					    sink_crop->width
> +					    / sensor->binning_subtypes[i].horizontal,
> +					    sel->r.width,
> +					    sink_crop->height
> +					    / sensor->binning_subtypes[i].vertical,
> +					    sel->r.height, sel->flags);
>  
>  		if (this > best) {
>  			binh = sensor->binning_subtypes[i].horizontal;
> @@ -2553,8 +2537,8 @@ static void ccs_set_compose_binner(struct v4l2_subdev *subdev,
>  		sensor->binning_horizontal = binh;
>  	}
>  
> -	sel->r.width = (crops[CCS_PAD_SINK]->width / binh) & ~1;
> -	sel->r.height = (crops[CCS_PAD_SINK]->height / binv) & ~1;
> +	sel->r.width = (sink_crop->width / binh) & ~1;
> +	sel->r.height = (sink_crop->height / binv) & ~1;
>  }
>  
>  /*
> @@ -2569,8 +2553,7 @@ static void ccs_set_compose_binner(struct v4l2_subdev *subdev,
>  static void ccs_set_compose_scaler(struct v4l2_subdev *subdev,
>  				   struct v4l2_subdev_state *sd_state,
>  				   struct v4l2_subdev_selection *sel,
> -				   struct v4l2_rect **crops,
> -				   struct v4l2_rect *comp)
> +				   const struct v4l2_rect *sink_crop)
>  {
>  	struct i2c_client *client = v4l2_get_subdevdata(subdev);
>  	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
> @@ -2582,17 +2565,12 @@ static void ccs_set_compose_scaler(struct v4l2_subdev *subdev,
>  	unsigned int i;
>  	int best = INT_MIN;
>  
> -	sel->r.width = min_t(unsigned int, sel->r.width,
> -			     crops[CCS_PAD_SINK]->width);
> -	sel->r.height = min_t(unsigned int, sel->r.height,
> -			      crops[CCS_PAD_SINK]->height);
> -
> -	a = crops[CCS_PAD_SINK]->width
> -		* CCS_LIM(sensor, SCALER_N_MIN) / sel->r.width;
> -	b = crops[CCS_PAD_SINK]->height
> -		* CCS_LIM(sensor, SCALER_N_MIN) / sel->r.height;
> -	max_m = crops[CCS_PAD_SINK]->width
> -		* CCS_LIM(sensor, SCALER_N_MIN)
> +	sel->r.width = min_t(unsigned int, sel->r.width, sink_crop->width);
> +	sel->r.height = min_t(unsigned int, sel->r.height, sink_crop->height);
> +
> +	a = sink_crop->width * CCS_LIM(sensor, SCALER_N_MIN) / sel->r.width;
> +	b = sink_crop->height * CCS_LIM(sensor, SCALER_N_MIN) / sel->r.height;
> +	max_m = sink_crop->width * CCS_LIM(sensor, SCALER_N_MIN)
>  		/ CCS_LIM(sensor, MIN_X_OUTPUT_SIZE);
>  
>  	a = clamp(a, CCS_LIM(sensor, SCALER_M_MIN),
> @@ -2623,14 +2601,12 @@ static void ccs_set_compose_scaler(struct v4l2_subdev *subdev,
>  	}
>  
>  	for (i = 0; i < ntry; i++) {
> -		int this = scaling_goodness(
> -			subdev,
> -			crops[CCS_PAD_SINK]->width
> -			/ try[i] * CCS_LIM(sensor, SCALER_N_MIN),
> -			sel->r.width,
> -			crops[CCS_PAD_SINK]->height,
> -			sel->r.height,
> -			sel->flags);
> +		int this = scaling_goodness(subdev,
> +					    sink_crop->width
> +					    / try[i]
> +					    * CCS_LIM(sensor, SCALER_N_MIN),
> +					    sel->r.width, sink_crop->height,
> +					    sel->r.height, sel->flags);
>  
>  		dev_dbg(&client->dev, "trying factor %u (%u)\n", try[i], i);
>  
> @@ -2645,12 +2621,10 @@ static void ccs_set_compose_scaler(struct v4l2_subdev *subdev,
>  			continue;
>  
>  		this = scaling_goodness(
> -			subdev, crops[CCS_PAD_SINK]->width
> -			/ try[i]
> +			subdev, sink_crop->width / try[i]
>  			* CCS_LIM(sensor, SCALER_N_MIN),
>  			sel->r.width,
> -			crops[CCS_PAD_SINK]->height
> -			/ try[i]
> +			sink_crop->height / try[i]
>  			* CCS_LIM(sensor, SCALER_N_MIN),
>  			sel->r.height,
>  			sel->flags);
> @@ -2662,18 +2636,13 @@ static void ccs_set_compose_scaler(struct v4l2_subdev *subdev,
>  		}
>  	}
>  
> -	sel->r.width =
> -		(crops[CCS_PAD_SINK]->width
> -		 / scale_m
> -		 * CCS_LIM(sensor, SCALER_N_MIN)) & ~1;
> +	sel->r.width = (sink_crop->width / scale_m
> +			* CCS_LIM(sensor, SCALER_N_MIN)) & ~1;
>  	if (mode == SMIAPP_SCALING_MODE_BOTH)
> -		sel->r.height =
> -			(crops[CCS_PAD_SINK]->height
> -			 / scale_m
> -			 * CCS_LIM(sensor, SCALER_N_MIN))
> -			& ~1;
> +		sel->r.height = (sink_crop->height / scale_m
> +				 * CCS_LIM(sensor, SCALER_N_MIN)) & ~1;
>  	else
> -		sel->r.height = crops[CCS_PAD_SINK]->height;
> +		sel->r.height = sink_crop->height;
>  
>  	if (sel->which == V4L2_SUBDEV_FORMAT_ACTIVE) {
>  		sensor->scale_m = scale_m;
> @@ -2687,17 +2656,21 @@ static int ccs_set_compose(struct v4l2_subdev *subdev,
>  {
>  	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
>  	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
> -	struct v4l2_rect *comp, *crops[CCS_PADS];
> +	const struct v4l2_rect *sink_crop;
> +	struct v4l2_rect *comp;
>  
> -	ccs_get_crop_compose(subdev, sd_state, crops, &comp);
> +	sink_crop = v4l2_subdev_state_get_crop(sd_state, CCS_PAD_SINK,
> +					       CCS_STREAM_PIXEL);
> +	comp = v4l2_subdev_state_get_compose(sd_state, ssd->sink_pad,
> +					     CCS_STREAM_PIXEL);
>  
>  	sel->r.top = 0;
>  	sel->r.left = 0;
>  
>  	if (ssd == sensor->binner)
> -		ccs_set_compose_binner(subdev, sd_state, sel, crops, comp);
> +		ccs_set_compose_binner(subdev, sd_state, sel, sink_crop);
>  	else
> -		ccs_set_compose_scaler(subdev, sd_state, sel, crops, comp);
> +		ccs_set_compose_scaler(subdev, sd_state, sel, sink_crop);
>  
>  	*comp = sel->r;
>  	ccs_propagate(subdev, sd_state, sel->which, V4L2_SEL_TGT_COMPOSE);
> @@ -2758,9 +2731,13 @@ static int ccs_set_crop(struct v4l2_subdev *subdev,
>  {
>  	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
>  	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
> -	struct v4l2_rect src_size = { 0 }, *crops[CCS_PADS], *comp;
> +	struct v4l2_rect src_size = { 0 }, *crop;
> +	const struct v4l2_rect *comp;
>  
> -	ccs_get_crop_compose(subdev, sd_state, crops, &comp);
> +	crop = v4l2_subdev_state_get_crop(sd_state, sel->pad,
> +					  CCS_STREAM_PIXEL);
> +	comp = v4l2_subdev_state_get_compose(sd_state, ssd->sink_pad,
> +					     CCS_STREAM_PIXEL);
>  
>  	if (sel->pad == ssd->sink_pad) {
>  		struct v4l2_mbus_framefmt *mfmt =
> @@ -2784,7 +2761,7 @@ static int ccs_set_crop(struct v4l2_subdev *subdev,
>  	sel->r.left = min_t(int, sel->r.left, src_size.width - sel->r.width);
>  	sel->r.top = min_t(int, sel->r.top, src_size.height - sel->r.height);
>  
> -	*crops[sel->pad] = sel->r;
> +	*crop = sel->r;
>  
>  	if (ssd != sensor->pixel_array && sel->pad == CCS_PAD_SINK)
>  		ccs_propagate(subdev, sd_state, sel->which, V4L2_SEL_TGT_CROP);
> @@ -2809,14 +2786,17 @@ static int ccs_get_selection(struct v4l2_subdev *subdev,
>  {
>  	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
>  	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
> -	struct v4l2_rect *comp, *crops[CCS_PADS];
> +	const struct v4l2_rect *crop, *comp;
>  	int ret;
>  
>  	ret = ccs_sel_supported(subdev, sel);
>  	if (ret)
>  		return ret;
>  
> -	ccs_get_crop_compose(subdev, sd_state, crops, &comp);
> +	crop = v4l2_subdev_state_get_crop(sd_state, sel->pad,
> +					  CCS_STREAM_PIXEL);
> +	comp = v4l2_subdev_state_get_compose(sd_state, ssd->sink_pad,
> +					     CCS_STREAM_PIXEL);
>  
>  	switch (sel->target) {
>  	case V4L2_SEL_TGT_CROP_BOUNDS:
> @@ -2838,7 +2818,7 @@ static int ccs_get_selection(struct v4l2_subdev *subdev,
>  		break;
>  	case V4L2_SEL_TGT_CROP:
>  	case V4L2_SEL_TGT_COMPOSE_BOUNDS:
> -		sel->r = *crops[sel->pad];
> +		sel->r = *crop;
>  		break;
>  	case V4L2_SEL_TGT_COMPOSE:
>  		sel->r = *comp;

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 32/46] media: ccs: Rely on sub-device state locking

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Tue, Apr 16, 2024 at 10:33:05PM +0300, Sakari Ailus wrote:
> Rely on sub-device state locking to serialise access to driver's data
> structures. The driver-provided mutex is used as the state lock for all
> driver sub-devices.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  drivers/media/i2c/ccs/ccs-core.c | 192 +++++++++++++------------------
>  drivers/media/i2c/ccs/ccs.h      |   1 -
>  2 files changed, 81 insertions(+), 112 deletions(-)

The diffstat doesn't do justice to this patch, it's a really nice
simplification.

> 
> diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
> index 299b5ac6538c..f82f3ec37c7c 100644
> --- a/drivers/media/i2c/ccs/ccs-core.c
> +++ b/drivers/media/i2c/ccs/ccs-core.c
> @@ -541,12 +541,13 @@ static int ccs_pll_update(struct ccs_sensor *sensor)
>   *
>   */
>  
> -static void __ccs_update_exposure_limits(struct ccs_sensor *sensor)
> +static void __ccs_update_exposure_limits(struct ccs_sensor *sensor,
> +					 struct v4l2_rect *pa_src)

const

>  {
>  	struct v4l2_ctrl *ctrl = sensor->exposure;
>  	int max;
>  
> -	max = sensor->pa_src.height + sensor->vblank->val -
> +	max = pa_src->height + sensor->vblank->val -
>  		CCS_LIM(sensor, COARSE_INTEGRATION_TIME_MAX_MARGIN);
>  
>  	__v4l2_ctrl_modify_range(ctrl, ctrl->minimum, max, ctrl->step, max);
> @@ -649,12 +650,20 @@ static int ccs_set_ctrl(struct v4l2_ctrl *ctrl)
>  		container_of(ctrl->handler, struct ccs_subdev, ctrl_handler)
>  			->sensor;
>  	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
> +	struct v4l2_subdev_state *state;
> +	struct v4l2_rect *pa_src;

const

>  	int pm_status;
>  	u32 orient = 0;
>  	unsigned int i;
>  	int exposure;
>  	int rval;
>  
> +	if (ctrl->id == V4L2_CID_VBLANK || ctrl->id == V4L2_CID_HBLANK) {
> +		state = v4l2_subdev_get_locked_active_state(&sensor->pixel_array->sd);
> +		pa_src = v4l2_subdev_state_get_crop(state, CCS_PA_PAD_SRC,
> +						    CCS_STREAM_PIXEL);
> +	}
> +
>  	switch (ctrl->id) {
>  	case V4L2_CID_HFLIP:
>  	case V4L2_CID_VFLIP:
> @@ -673,7 +682,7 @@ static int ccs_set_ctrl(struct v4l2_ctrl *ctrl)
>  	case V4L2_CID_VBLANK:
>  		exposure = sensor->exposure->val;
>  
> -		__ccs_update_exposure_limits(sensor);
> +		__ccs_update_exposure_limits(sensor, pa_src);
>  
>  		if (exposure > sensor->exposure->maximum) {
>  			sensor->exposure->val =	sensor->exposure->maximum;
> @@ -765,12 +774,12 @@ static int ccs_set_ctrl(struct v4l2_ctrl *ctrl)
>  		break;
>  	case V4L2_CID_VBLANK:
>  		rval = ccs_write(sensor, FRAME_LENGTH_LINES,
> -				 sensor->pa_src.height + ctrl->val);
> +				 pa_src->height + ctrl->val);
>  
>  		break;
>  	case V4L2_CID_HBLANK:
>  		rval = ccs_write(sensor, LINE_LENGTH_PCK,
> -				 sensor->pa_src.width + ctrl->val);
> +				 pa_src->width + ctrl->val);
>  
>  		break;
>  	case V4L2_CID_TEST_PATTERN:
> @@ -1225,7 +1234,8 @@ static int ccs_get_mbus_formats(struct ccs_sensor *sensor)
>  	return 0;
>  }
>  
> -static void ccs_update_blanking(struct ccs_sensor *sensor)
> +static void ccs_update_blanking(struct ccs_sensor *sensor,
> +				struct v4l2_rect *pa_src)

const

>  {
>  	struct v4l2_ctrl *vblank = sensor->vblank;
>  	struct v4l2_ctrl *hblank = sensor->hblank;
> @@ -1248,21 +1258,26 @@ static void ccs_update_blanking(struct ccs_sensor *sensor)
>  
>  	min = max_t(int,
>  		    CCS_LIM(sensor, MIN_FRAME_BLANKING_LINES),
> -		    min_fll - sensor->pa_src.height);
> -	max = max_fll -	sensor->pa_src.height;
> +		    min_fll - pa_src->height);
> +	max = max_fll -	pa_src->height;
>  
>  	__v4l2_ctrl_modify_range(vblank, min, max, vblank->step, min);
>  
> -	min = max_t(int, min_llp - sensor->pa_src.width, min_lbp);
> -	max = max_llp - sensor->pa_src.width;
> +	min = max_t(int, min_llp - pa_src->width, min_lbp);
> +	max = max_llp - pa_src->width;
>  
>  	__v4l2_ctrl_modify_range(hblank, min, max, hblank->step, min);
>  
> -	__ccs_update_exposure_limits(sensor);
> +	__ccs_update_exposure_limits(sensor, pa_src);
>  }
>  
>  static int ccs_pll_blanking_update(struct ccs_sensor *sensor)
>  {
> +	struct v4l2_subdev_state *state =
> +		v4l2_subdev_get_locked_active_state(&sensor->pixel_array->sd);
> +	struct v4l2_rect *pa_src =
> +		v4l2_subdev_state_get_crop(state, CCS_PA_PAD_SRC,
> +					   CCS_STREAM_PIXEL);

You can guess what I will say :-)

>  	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
>  	int rval;
>  
> @@ -1271,15 +1286,15 @@ static int ccs_pll_blanking_update(struct ccs_sensor *sensor)
>  		return rval;
>  
>  	/* Output from pixel array, including blanking */
> -	ccs_update_blanking(sensor);
> +	ccs_update_blanking(sensor, pa_src);
>  
>  	dev_dbg(&client->dev, "vblank\t\t%d\n", sensor->vblank->val);
>  	dev_dbg(&client->dev, "hblank\t\t%d\n", sensor->hblank->val);
>  
>  	dev_dbg(&client->dev, "real timeperframe\t100/%d\n",
>  		sensor->pll.pixel_rate_pixel_array /
> -		((sensor->pa_src.width + sensor->hblank->val) *
> -		 (sensor->pa_src.height + sensor->vblank->val) / 100));
> +		((pa_src->width + sensor->hblank->val) *
> +		 (pa_src->height + sensor->vblank->val) / 100));
>  
>  	return 0;
>  }
> @@ -1788,6 +1803,16 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
>  			      u64 streams_mask)
>  {
>  	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
> +	struct v4l2_subdev_state *pa_state =
> +		v4l2_subdev_get_locked_active_state(&sensor->pixel_array->sd);
> +	struct v4l2_subdev_state *src_state =
> +		v4l2_subdev_get_locked_active_state(&sensor->src->sd);
> +	struct v4l2_rect *pa_src =
> +		v4l2_subdev_state_get_crop(pa_state, CCS_PA_PAD_SRC,
> +					   CCS_STREAM_PIXEL);
> +	struct v4l2_rect *src_src =
> +		v4l2_subdev_state_get_crop(src_state, CCS_PAD_SRC,
> +					   CCS_STREAM_PIXEL);

Same for pa_src and src_src.

>  	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
>  	unsigned int binning_mode;
>  	int rval;
> @@ -1835,22 +1860,20 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
>  		goto err_pm_put;
>  
>  	/* Analog crop start coordinates */
> -	rval = ccs_write(sensor, X_ADDR_START, sensor->pa_src.left);
> +	rval = ccs_write(sensor, X_ADDR_START, pa_src->left);
>  	if (rval < 0)
>  		goto err_pm_put;
>  
> -	rval = ccs_write(sensor, Y_ADDR_START, sensor->pa_src.top);
> +	rval = ccs_write(sensor, Y_ADDR_START, pa_src->top);
>  	if (rval < 0)
>  		goto err_pm_put;
>  
>  	/* Analog crop end coordinates */
> -	rval = ccs_write(sensor, X_ADDR_END,
> -			 sensor->pa_src.left + sensor->pa_src.width - 1);
> +	rval = ccs_write(sensor, X_ADDR_END, pa_src->left + pa_src->width - 1);
>  	if (rval < 0)
>  		goto err_pm_put;
>  
> -	rval = ccs_write(sensor, Y_ADDR_END,
> -			 sensor->pa_src.top + sensor->pa_src.height - 1);
> +	rval = ccs_write(sensor, Y_ADDR_END, pa_src->top + pa_src->height - 1);
>  	if (rval < 0)
>  		goto err_pm_put;
>  
> @@ -1862,23 +1885,30 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
>  	/* Digital crop */
>  	if (CCS_LIM(sensor, DIGITAL_CROP_CAPABILITY)
>  	    == CCS_DIGITAL_CROP_CAPABILITY_INPUT_CROP) {
> +		struct v4l2_subdev_state *scaler_state =
> +			v4l2_subdev_get_locked_active_state(&sensor->scaler->sd);
> +		struct v4l2_rect *scaler_sink =
> +			v4l2_subdev_state_get_crop(scaler_state,
> +						   sensor->scaler->sink_pad,
> +						   CCS_STREAM_PIXEL);

Here too.

> +
>  		rval = ccs_write(sensor, DIGITAL_CROP_X_OFFSET,
> -				 sensor->scaler_sink.left);
> +				 scaler_sink->left);
>  		if (rval < 0)
>  			goto err_pm_put;
>  
>  		rval = ccs_write(sensor, DIGITAL_CROP_Y_OFFSET,
> -				 sensor->scaler_sink.top);
> +				 scaler_sink->top);
>  		if (rval < 0)
>  			goto err_pm_put;
>  
>  		rval = ccs_write(sensor, DIGITAL_CROP_IMAGE_WIDTH,
> -				 sensor->scaler_sink.width);
> +				 scaler_sink->width);
>  		if (rval < 0)
>  			goto err_pm_put;
>  
>  		rval = ccs_write(sensor, DIGITAL_CROP_IMAGE_HEIGHT,
> -				 sensor->scaler_sink.height);
> +				 scaler_sink->height);
>  		if (rval < 0)
>  			goto err_pm_put;
>  	}
> @@ -1896,10 +1926,10 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
>  	}
>  
>  	/* Output size from sensor */
> -	rval = ccs_write(sensor, X_OUTPUT_SIZE, sensor->src_src.width);
> +	rval = ccs_write(sensor, X_OUTPUT_SIZE, src_src->width);
>  	if (rval < 0)
>  		goto err_pm_put;
> -	rval = ccs_write(sensor, Y_OUTPUT_SIZE, sensor->src_src.height);
> +	rval = ccs_write(sensor, Y_OUTPUT_SIZE, src_src->height);
>  	if (rval < 0)
>  		goto err_pm_put;
>  
> @@ -2088,9 +2118,6 @@ static int ccs_enum_mbus_code(struct v4l2_subdev *subdev,
>  	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
>  	unsigned int i;
>  	int idx = -1;
> -	int rval = -EINVAL;
> -
> -	mutex_lock(&sensor->mutex);
>  
>  	dev_err(&client->dev, "subdev %s, pad %u, index %u\n",
>  		subdev->name, code->pad, code->index);
> @@ -2098,12 +2125,11 @@ static int ccs_enum_mbus_code(struct v4l2_subdev *subdev,
>  	if (subdev == &sensor->src->sd) {
>  		if (code->pad == CCS_PAD_META) {
>  			if (code->index)
> -				goto out;
> +				return -EINVAL;
>  
>  			code->code = MEDIA_BUS_FMT_CCS_EMBEDDED;
>  
> -			rval = 0;
> -			goto out;
> +			return 0;
>  		}
>  		if (code->stream == CCS_STREAM_META) {
>  			struct v4l2_mbus_framefmt *pix_fmt =
> @@ -2150,21 +2176,21 @@ static int ccs_enum_mbus_code(struct v4l2_subdev *subdev,
>  			}
>  
>  			if (WARN_ON(i > ARRAY_SIZE(codes)) || code->index >= i)
> -				goto out;
> +				return -EINVAL;
>  
>  			code->code = codes[code->index];
> -			rval = 0;
> -			goto out;
> +
> +			return 0;
>  		}
>  	}
>  
>  	if (subdev != &sensor->src->sd || code->pad != CCS_PAD_SRC) {
>  		if (code->index)
> -			goto out;
> +			return -EINVAL;
>  
>  		code->code = sensor->internal_csi_format->code;
> -		rval = 0;
> -		goto out;
> +
> +		return 0;
>  	}
>  
>  	for (i = 0; i < ARRAY_SIZE(ccs_csi_data_formats); i++) {
> @@ -2175,18 +2201,14 @@ static int ccs_enum_mbus_code(struct v4l2_subdev *subdev,
>  			code->code = ccs_csi_data_formats[i].code;
>  			dev_err(&client->dev, "found index %u, i %u, code %x\n",
>  				code->index, i, code->code);
> -			rval = 0;
> -			break;
> +			return 0;
>  		}
>  	}
>  
> -out:
> -	mutex_unlock(&sensor->mutex);
> -
> -	return rval;
> +	return -EINVAL;
>  }
>  
> -static u32 __ccs_get_mbus_code(struct v4l2_subdev *subdev, unsigned int pad)
> +static u32 ccs_get_mbus_code(struct v4l2_subdev *subdev, unsigned int pad)
>  {
>  	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
>  
> @@ -2196,33 +2218,19 @@ static u32 __ccs_get_mbus_code(struct v4l2_subdev *subdev, unsigned int pad)
>  		return sensor->internal_csi_format->code;
>  }
>  
> -static int __ccs_get_format(struct v4l2_subdev *subdev,
> -			    struct v4l2_subdev_state *sd_state,
> -			    struct v4l2_subdev_format *fmt)
> +static int ccs_get_format(struct v4l2_subdev *subdev,
> +			  struct v4l2_subdev_state *sd_state,
> +			  struct v4l2_subdev_format *fmt)
>  {
>  	fmt->format = *v4l2_subdev_state_get_format(sd_state, fmt->pad,
>  						    fmt->stream);
>  
>  	if (fmt->pad != CCS_PAD_META && fmt->stream != CCS_STREAM_META)
> -		fmt->format.code = __ccs_get_mbus_code(subdev, fmt->pad);
> +		fmt->format.code = ccs_get_mbus_code(subdev, fmt->pad);
>  
>  	return 0;
>  }

Any chance to switch to v4l2_subdev_get_fmt() ?

>  
> -static int ccs_get_format(struct v4l2_subdev *subdev,
> -			  struct v4l2_subdev_state *sd_state,
> -			  struct v4l2_subdev_format *fmt)
> -{
> -	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
> -	int rval;
> -
> -	mutex_lock(&sensor->mutex);
> -	rval = __ccs_get_format(subdev, sd_state, fmt);
> -	mutex_unlock(&sensor->mutex);
> -
> -	return rval;
> -}
> -
>  /* Changes require propagation only on sink pad. */
>  static void ccs_propagate(struct v4l2_subdev *subdev,
>  			  struct v4l2_subdev_state *sd_state, int which,
> @@ -2246,7 +2254,6 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
>  				sensor->scale_m = CCS_LIM(sensor, SCALER_N_MIN);
>  				sensor->scaling_mode =
>  					CCS_SCALING_MODE_NO_SCALING;
> -				sensor->scaler_sink = *comp;
>  			} else if (ssd == sensor->binner) {
>  				sensor->binning_horizontal = 1;
>  				sensor->binning_vertical = 1;
> @@ -2261,8 +2268,6 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
>  						   CCS_STREAM_PIXEL);
>  		fmt->width = comp->width;
>  		fmt->height = comp->height;
> -		if (which == V4L2_SUBDEV_FORMAT_ACTIVE && ssd == sensor->src)
> -			sensor->src_src = *crop;
>  		break;
>  	default:
>  		WARN_ON_ONCE(1);
> @@ -2281,7 +2286,7 @@ static int ccs_set_format_source(struct v4l2_subdev *subdev,
>  	unsigned int i;
>  	int rval;
>  
> -	rval = __ccs_get_format(subdev, sd_state, fmt);
> +	rval = ccs_get_format(subdev, sd_state, fmt);
>  	if (rval)
>  		return rval;
>  
> @@ -2410,13 +2415,9 @@ static int ccs_set_format(struct v4l2_subdev *subdev,
>  	if (subdev == &sensor->src->sd && fmt->pad == CCS_PAD_META)
>  		return ccs_get_format(subdev, sd_state, fmt);
>  
> -	mutex_lock(&sensor->mutex);
> -
>  	if (subdev == &sensor->src->sd && fmt->stream == CCS_STREAM_META) {
>  		ccs_set_format_meta(subdev, sd_state, &fmt->format);
>  
> -		mutex_unlock(&sensor->mutex);
> -
>  		return 0;
>  	}
>  
> @@ -2427,13 +2428,12 @@ static int ccs_set_format(struct v4l2_subdev *subdev,
>  		if (sensor->embedded_start != sensor->embedded_end)
>  			ccs_set_format_meta(subdev, sd_state, NULL);
>  
> -		mutex_unlock(&sensor->mutex);
> -
>  		return rval;
>  	}
>  
>  	/* Sink pad. Width and height are changeable here. */
> -	fmt->format.code = __ccs_get_mbus_code(subdev, fmt->pad);
> +	fmt->format.code = ccs_get_mbus_code(subdev, fmt->pad);
> +
>  	fmt->format.width &= ~1;
>  	fmt->format.height &= ~1;
>  	fmt->format.field = V4L2_FIELD_NONE;
> @@ -2456,8 +2456,6 @@ static int ccs_set_format(struct v4l2_subdev *subdev,
>  	crop->height = fmt->format.height;
>  	ccs_propagate(subdev, sd_state, fmt->which, V4L2_SEL_TGT_CROP);
>  
> -	mutex_unlock(&sensor->mutex);
> -
>  	return 0;
>  }
>  
> @@ -2765,9 +2763,6 @@ static int ccs_set_crop(struct v4l2_subdev *subdev,
>  
>  	if (ssd != sensor->pixel_array && sel->pad == CCS_PAD_SINK)
>  		ccs_propagate(subdev, sd_state, sel->which, V4L2_SEL_TGT_CROP);
> -	else if (sel->which == V4L2_SUBDEV_FORMAT_ACTIVE &&
> -		 ssd == sensor->pixel_array)
> -		sensor->pa_src = sel->r;
>  
>  	return 0;
>  }
> @@ -2839,8 +2834,6 @@ static int ccs_set_selection(struct v4l2_subdev *subdev,
>  	if (ret)
>  		return ret;
>  
> -	mutex_lock(&sensor->mutex);
> -
>  	sel->r.left = max(0, sel->r.left & ~1);
>  	sel->r.top = max(0, sel->r.top & ~1);
>  	sel->r.width = CCS_ALIGN_DIM(sel->r.width, sel->flags);
> @@ -2862,7 +2855,6 @@ static int ccs_set_selection(struct v4l2_subdev *subdev,
>  		ret = -EINVAL;
>  	}
>  
> -	mutex_unlock(&sensor->mutex);
>  	return ret;
>  }
>  
> @@ -3238,6 +3230,7 @@ static int ccs_init_subdev(struct ccs_sensor *sensor,
>  
>  	ssd->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
>  	ssd->sd.entity.function = function;
> +	ssd->sd.state_lock = &sensor->mutex;
>  	ssd->sensor = sensor;
>  
>  	ssd->npads = num_pads;
> @@ -3277,8 +3270,8 @@ static int ccs_init_subdev(struct ccs_sensor *sensor,
>  	return 0;
>  }
>  
> -static int __ccs_init_state(struct v4l2_subdev *sd,
> -			    struct v4l2_subdev_state *sd_state)
> +static int ccs_init_state(struct v4l2_subdev *sd,
> +			  struct v4l2_subdev_state *sd_state)
>  {
>  	struct ccs_subdev *ssd = to_ccs_subdev(sd);
>  	struct ccs_sensor *sensor = ssd->sensor;
> @@ -3297,12 +3290,8 @@ static int __ccs_init_state(struct v4l2_subdev *sd,
>  	fmt->code = sensor->internal_csi_format->code;
>  	fmt->field = V4L2_FIELD_NONE;
>  
> -	if (ssd == sensor->pixel_array) {
> -		if (is_active)
> -			sensor->pa_src = *crop;
> -
> +	if (ssd == sensor->pixel_array)
>  		return 0;
> -	}
>  
>  	fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC,
>  					   CCS_STREAM_PIXEL);
> @@ -3315,20 +3304,6 @@ static int __ccs_init_state(struct v4l2_subdev *sd,
>  	return 0;
>  }
>  
> -static int ccs_init_state(struct v4l2_subdev *sd,
> -			  struct v4l2_subdev_state *sd_state)
> -{
> -	struct ccs_subdev *ssd = to_ccs_subdev(sd);
> -	struct ccs_sensor *sensor = ssd->sensor;
> -	int rval;
> -
> -	mutex_lock(&sensor->mutex);
> -	rval = __ccs_init_state(sd, sd_state);
> -	mutex_unlock(&sensor->mutex);
> -
> -	return rval;
> -}
> -
>  static int ccs_src_init_state(struct v4l2_subdev *sd,
>  			      struct v4l2_subdev_state *sd_state)
>  {
> @@ -3353,25 +3328,20 @@ static int ccs_src_init_state(struct v4l2_subdev *sd,
>  	struct ccs_sensor *sensor = ssd->sensor;
>  	int rval;
>  
> -	mutex_lock(&sensor->mutex);
> -
>  	if (sensor->embedded_start != sensor->embedded_end)
>  		routing.num_routes++;
>  
>  	rval = v4l2_subdev_set_routing(sd, sd_state, &routing);
>  	if (rval)
> -		goto out;
> +		return 0;
>  
> -	rval = __ccs_init_state(sd, sd_state);
> +	rval = ccs_init_state(sd, sd_state);
>  	if (rval)
> -		goto out;
> +		return 0;
>  
>  	if (sensor->embedded_start != sensor->embedded_end)
>  		ccs_set_format_meta(sd, sd_state, NULL);
>  
> -out:
> -	mutex_unlock(&sensor->mutex);
> -
>  	return rval;
>  }
>  
> diff --git a/drivers/media/i2c/ccs/ccs.h b/drivers/media/i2c/ccs/ccs.h
> index 477b2fb99aa0..dcca3f88ea67 100644
> --- a/drivers/media/i2c/ccs/ccs.h
> +++ b/drivers/media/i2c/ccs/ccs.h
> @@ -230,7 +230,6 @@ struct ccs_sensor {
>  	u32 mbus_frame_fmts;
>  	const struct ccs_csi_data_format *csi_format;
>  	const struct ccs_csi_data_format *internal_csi_format;
> -	struct v4l2_rect pa_src, scaler_sink, src_src;
>  	u32 default_mbus_frame_fmts;
>  	int default_pixel_order;
>  	struct ccs_data_container sdata, mdata;

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 33/46] media: ccs: Compute binning configuration from sub-device state

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Tue, Apr 16, 2024 at 10:33:06PM +0300, Sakari Ailus wrote:
> Calculate binning configuration from sub-device state so the state related
> configuration can be removed from the driver's device context struct.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> ---
>  drivers/media/i2c/ccs/ccs-core.c | 56 +++++++++++++++++++-------------
>  drivers/media/i2c/ccs/ccs.h      |  3 --
>  2 files changed, 33 insertions(+), 26 deletions(-)
> 
> diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
> index f82f3ec37c7c..08e719d611fb 100644
> --- a/drivers/media/i2c/ccs/ccs-core.c
> +++ b/drivers/media/i2c/ccs/ccs-core.c
> @@ -511,13 +511,36 @@ static int ccs_pll_try(struct ccs_sensor *sensor, struct ccs_pll *pll)
>  	return ccs_pll_calculate(&client->dev, &lim, pll);
>  }
>  
> +static void
> +ccs_get_binning(struct ccs_sensor *sensor, u8 *binning_mode, u8 *binh, u8 *binv)
> +{
> +	struct v4l2_subdev_state *state =
> +		v4l2_subdev_get_locked_active_state(&sensor->binner->sd);
> +	const struct v4l2_rect *sink_crop =
> +		v4l2_subdev_state_get_crop(state, CCS_PAD_SINK,
> +					   CCS_STREAM_PIXEL);
> +	const struct v4l2_rect *sink_comp =
> +		v4l2_subdev_state_get_compose(state, CCS_PAD_SINK,
> +					      CCS_STREAM_PIXEL);
> +
> +	if (binning_mode)
> +		*binning_mode =	sink_crop->width == sink_comp->width &&
> +				sink_crop->height == sink_comp->height ? 0 : 1;
> +
> +	*binh = sink_crop->width / sink_comp->width;
> +	*binv = sink_crop->height / sink_comp->height;
> +}
> +
>  static int ccs_pll_update(struct ccs_sensor *sensor)
>  {
>  	struct ccs_pll *pll = &sensor->pll;
> +	u8 binh, binv;
>  	int rval;
>  
> -	pll->binning_horizontal = sensor->binning_horizontal;
> -	pll->binning_vertical = sensor->binning_vertical;
> +	ccs_get_binning(sensor, NULL, &binh, &binv);
> +
> +	pll->binning_horizontal = binh;
> +	pll->binning_vertical = binv;
>  	pll->link_freq =
>  		sensor->link_freq->qmenu_int[sensor->link_freq->val];
>  	pll->scale_m = sensor->scale_m;
> @@ -1241,8 +1264,11 @@ static void ccs_update_blanking(struct ccs_sensor *sensor,
>  	struct v4l2_ctrl *hblank = sensor->hblank;
>  	u16 min_fll, max_fll, min_llp, max_llp, min_lbp;
>  	int min, max;
> +	u8 binh, binv;
> +
> +	ccs_get_binning(sensor, NULL, &binh, &binv);
>  
> -	if (sensor->binning_vertical > 1 || sensor->binning_horizontal > 1) {
> +	if (binv > 1 || binh > 1) {
>  		min_fll = CCS_LIM(sensor, MIN_FRAME_LENGTH_LINES_BIN);
>  		max_fll = CCS_LIM(sensor, MAX_FRAME_LENGTH_LINES_BIN);
>  		min_llp = CCS_LIM(sensor, MIN_LINE_LENGTH_PCK_BIN);
> @@ -1814,7 +1840,7 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
>  		v4l2_subdev_state_get_crop(src_state, CCS_PAD_SRC,
>  					   CCS_STREAM_PIXEL);
>  	struct i2c_client *client = v4l2_get_subdevdata(&sensor->src->sd);
> -	unsigned int binning_mode;
> +	u8 binning_mode, binh, binv;
>  	int rval;
>  
>  	if (pad != CCS_PAD_SRC)
> @@ -1836,19 +1862,12 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
>  		goto err_pm_put;
>  
>  	/* Binning configuration */
> -	if (sensor->binning_horizontal == 1 &&
> -	    sensor->binning_vertical == 1) {
> -		binning_mode = 0;
> -	} else {
> -		u8 binning_type =
> -			(sensor->binning_horizontal << 4)
> -			| sensor->binning_vertical;
> +	ccs_get_binning(sensor,	&binning_mode, &binh, &binv);
>  
> -		rval = ccs_write(sensor, BINNING_TYPE, binning_type);
> +	if (binning_mode) {
> +		rval = ccs_write(sensor, BINNING_TYPE, (binh << 4) | binv);
>  		if (rval < 0)
>  			goto err_pm_put;
> -
> -		binning_mode = 1;
>  	}
>  	rval = ccs_write(sensor, BINNING_MODE, binning_mode);
>  	if (rval < 0)
> @@ -2254,9 +2273,6 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
>  				sensor->scale_m = CCS_LIM(sensor, SCALER_N_MIN);
>  				sensor->scaling_mode =
>  					CCS_SCALING_MODE_NO_SCALING;
> -			} else if (ssd == sensor->binner) {
> -				sensor->binning_horizontal = 1;
> -				sensor->binning_vertical = 1;
>  			}
>  		}
>  		fallthrough;
> @@ -2530,10 +2546,6 @@ static void ccs_set_compose_binner(struct v4l2_subdev *subdev,
>  			best = this;
>  		}
>  	}
> -	if (sel->which == V4L2_SUBDEV_FORMAT_ACTIVE) {
> -		sensor->binning_vertical = binv;
> -		sensor->binning_horizontal = binh;
> -	}
>  
>  	sel->r.width = (sink_crop->width / binh) & ~1;
>  	sel->r.height = (sink_crop->height / binv) & ~1;
> @@ -3715,8 +3727,6 @@ static int ccs_probe(struct i2c_client *client)
>  				sensor->binning_subtypes[i].vertical);
>  		}
>  	}
> -	sensor->binning_horizontal = 1;
> -	sensor->binning_vertical = 1;
>  
>  	if (device_create_file(&client->dev, &dev_attr_ident) != 0) {
>  		dev_err(&client->dev, "sysfs ident entry creation failed\n");
> diff --git a/drivers/media/i2c/ccs/ccs.h b/drivers/media/i2c/ccs/ccs.h
> index dcca3f88ea67..aadbd4302607 100644
> --- a/drivers/media/i2c/ccs/ccs.h
> +++ b/drivers/media/i2c/ccs/ccs.h
> @@ -237,9 +237,6 @@ struct ccs_sensor {
>  	u32 embedded_mbus_code;
>  	u8 emb_data_ctrl;
>  
> -	u8 binning_horizontal;
> -	u8 binning_vertical;
> -
>  	u8 scale_m;
>  	u8 scaling_mode;
>  

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 34/46] media: ccs: Compute scaling configuration from sub-device state

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Tue, Apr 16, 2024 at 10:33:07PM +0300, Sakari Ailus wrote:
> Compute scaling configuration from sub-device state instead of storing it
> to the driver's device context struct.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  drivers/media/i2c/ccs/ccs-core.c | 59 ++++++++++++++++++++++----------
>  drivers/media/i2c/ccs/ccs.h      |  3 --
>  2 files changed, 40 insertions(+), 22 deletions(-)
> 
> diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
> index 08e719d611fb..541faa7d84a6 100644
> --- a/drivers/media/i2c/ccs/ccs-core.c
> +++ b/drivers/media/i2c/ccs/ccs-core.c
> @@ -531,19 +531,51 @@ ccs_get_binning(struct ccs_sensor *sensor, u8 *binning_mode, u8 *binh, u8 *binv)
>  	*binv = sink_crop->height / sink_comp->height;
>  }
>  
> +static void ccs_get_scaling(struct ccs_sensor *sensor, u8 *scaling_mode,
> +			    u8 *scale_m)
> +{
> +	struct v4l2_subdev_state *state =
> +		v4l2_subdev_get_locked_active_state(&sensor->scaler->sd);
> +	const struct v4l2_rect *sink_crop =
> +		v4l2_subdev_state_get_crop(state, CCS_PAD_SINK,
> +					   CCS_STREAM_PIXEL);
> +	const struct v4l2_rect *sink_comp =
> +		v4l2_subdev_state_get_compose(state, CCS_PAD_SINK,
> +					      CCS_STREAM_PIXEL);
> +
> +	*scale_m = sink_crop->width * CCS_LIM(sensor, SCALER_N_MIN) /
> +		sink_comp->width;
> +
> +	if (!scaling_mode)
> +		return;
> +
> +	if (sink_crop->width == sink_comp->width)
> +		*scaling_mode = CCS_SCALING_MODE_NO_SCALING;
> +	else if (sink_crop->height == sink_comp->height)
> +		*scaling_mode = CCS_SCALING_MODE_HORIZONTAL;
> +	else
> +		*scaling_mode = SMIAPP_SCALING_MODE_BOTH;
> +}
> +
>  static int ccs_pll_update(struct ccs_sensor *sensor)
>  {
>  	struct ccs_pll *pll = &sensor->pll;
>  	u8 binh, binv;
> +	u8 scale_m;
>  	int rval;
>  
>  	ccs_get_binning(sensor, NULL, &binh, &binv);
>  
> +	if (sensor->scaler)
> +		ccs_get_scaling(sensor, NULL, &scale_m);
> +	else
> +		scale_m = CCS_LIM(sensor, SCALER_N_MIN);
> +
>  	pll->binning_horizontal = binh;
>  	pll->binning_vertical = binv;
>  	pll->link_freq =
>  		sensor->link_freq->qmenu_int[sensor->link_freq->val];
> -	pll->scale_m = sensor->scale_m;
> +	pll->scale_m = scale_m;
>  	pll->bits_per_pixel = sensor->csi_format->compressed;
>  
>  	rval = ccs_pll_try(sensor, pll);
> @@ -1186,7 +1218,7 @@ static int ccs_get_mbus_formats(struct ccs_sensor *sensor)
>  	/* Figure out which BPP values can be used with which formats. */
>  	pll->binning_horizontal = 1;
>  	pll->binning_vertical = 1;
> -	pll->scale_m = sensor->scale_m;
> +	pll->scale_m = CCS_LIM(sensor, SCALER_N_MIN);
>  
>  	for (i = 0; i < ARRAY_SIZE(ccs_csi_data_formats); i++) {
>  		sensor->compressed_min_bpp =
> @@ -1935,11 +1967,15 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
>  	/* Scaling */
>  	if (CCS_LIM(sensor, SCALING_CAPABILITY)
>  	    != CCS_SCALING_CAPABILITY_NONE) {
> -		rval = ccs_write(sensor, SCALING_MODE, sensor->scaling_mode);
> +		u8 scaling_mode, scale_m;
> +
> +		ccs_get_scaling(sensor, &scaling_mode, &scale_m);
> +
> +		rval = ccs_write(sensor, SCALING_MODE, scaling_mode);
>  		if (rval < 0)
>  			goto err_pm_put;
>  
> -		rval = ccs_write(sensor, SCALE_M, sensor->scale_m);
> +		rval = ccs_write(sensor, SCALE_M, scale_m);
>  		if (rval < 0)
>  			goto err_pm_put;
>  	}
> @@ -2255,7 +2291,6 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
>  			  struct v4l2_subdev_state *sd_state, int which,

You can now drop the which parameter to this function \o/ With this, and
the is_active variable removed from the caller,

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

>  			  int target)
>  {
> -	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
>  	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
>  	struct v4l2_rect *comp, *crop;
>  	struct v4l2_mbus_framefmt *fmt;
> @@ -2268,13 +2303,6 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
>  						  CCS_STREAM_PIXEL);
>  		comp->width = crop->width;
>  		comp->height = crop->height;
> -		if (which == V4L2_SUBDEV_FORMAT_ACTIVE) {
> -			if (ssd == sensor->scaler) {
> -				sensor->scale_m = CCS_LIM(sensor, SCALER_N_MIN);
> -				sensor->scaling_mode =
> -					CCS_SCALING_MODE_NO_SCALING;
> -			}
> -		}
>  		fallthrough;
>  	case V4L2_SEL_TGT_COMPOSE:
>  		crop = v4l2_subdev_state_get_crop(sd_state, CCS_PAD_SRC,
> @@ -2653,11 +2681,6 @@ static void ccs_set_compose_scaler(struct v4l2_subdev *subdev,
>  				 * CCS_LIM(sensor, SCALER_N_MIN)) & ~1;
>  	else
>  		sel->r.height = sink_crop->height;
> -
> -	if (sel->which == V4L2_SUBDEV_FORMAT_ACTIVE) {
> -		sensor->scale_m = scale_m;
> -		sensor->scaling_mode = mode;
> -	}
>  }
>  /* We're only called on source pads. This function sets scaling. */
>  static int ccs_set_compose(struct v4l2_subdev *subdev,
> @@ -3763,8 +3786,6 @@ static int ccs_probe(struct i2c_client *client)
>  	sensor->pixel_array = &sensor->ssds[sensor->ssds_used];
>  	sensor->ssds_used++;
>  
> -	sensor->scale_m = CCS_LIM(sensor, SCALER_N_MIN);
> -
>  	/* prepare PLL configuration input values */
>  	sensor->pll.bus_type = CCS_PLL_BUS_TYPE_CSI2_DPHY;
>  	sensor->pll.csi2.lanes = sensor->hwcfg.lanes;
> diff --git a/drivers/media/i2c/ccs/ccs.h b/drivers/media/i2c/ccs/ccs.h
> index aadbd4302607..1c30fa85bed6 100644
> --- a/drivers/media/i2c/ccs/ccs.h
> +++ b/drivers/media/i2c/ccs/ccs.h
> @@ -237,9 +237,6 @@ struct ccs_sensor {
>  	u32 embedded_mbus_code;
>  	u8 emb_data_ctrl;
>  
> -	u8 scale_m;
> -	u8 scaling_mode;
> -
>  	u8 frame_skip;
>  	u16 embedded_start; /* embedded data start line */
>  	u16 embedded_end;

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 36/46] media: uapi: Add media bus code for ov2740 embedded data

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Tue, Apr 16, 2024 at 10:33:09PM +0300, Sakari Ailus wrote:
> Add a media bus code for ov2740 camera sensor embedded data and document
> it.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> Reviewed-by: Julien Massot <julien.massot@collabora.com>
> ---
>  .../media/v4l/subdev-formats.rst              | 70 +++++++++++++++++++
>  include/uapi/linux/media-bus-format.h         |  3 +-
>  2 files changed, 72 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> index a04756092238..c99b58cb8c7b 100644
> --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
> +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> @@ -8596,3 +8596,73 @@ This mbus code are only used for "2-byte simplified tagged data format" (code
>  embedded data format codes.
>  
>  Also see :ref:`CCS driver documentation <media-ccs-routes>`.
> +
> +Omnivision OV2740 Embedded Data Format
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +The Omnivision OV2740 camera sensor produces the following embedded data format,
> +indicated by mbus code MEDIA_BUS_FMT_OV2740_EMBEDDED. The format conforms to
> +:ref:`CCS embedded data format <MEDIA-BUS-FMT-CCS-EMBEDDED>` up to level 1.
> +
> +.. flat-table:: Omnivision OV2740 Embedded Data Format. Octets at indices marked
> +                reserved or unused have been omitted from the table. The values
> +                are big endian byte order.

s/are big endian/are in big endian/

> +    :header-rows: 1
> +
> +    * - Offset
> +      - Size in bits (active bits if not the same as size)
> +      - Content description
> +    * - 4
> +      - 16 (10--0)
> +      - Analogue gain
> +    * - 6
> +      - 16
> +      - Coarse integration time
> +    * - 10
> +      - 8
> +      - Dpc correction threshold

v8 indicated this contained bits 9:2 of the value, was that wrong, or
did it get dropped by mistake ?

> +    * - 15
> +      - 16
> +      - Output image width
> +    * - 17
> +      - 16
> +      - Output image height
> +    * - 23
> +      - 8
> +      - MIPI header revision number (2)
> +    * - 31
> +      - 8
> +      - Vertical (bit 1) and horizontal flip (bit 0)
> +    * - 32
> +      - 8
> +      - Frame duration A
> +    * - 33
> +      - 8
> +      - Frame duration B
> +    * - 34
> +      - 8
> +      - Context count (2)
> +    * - 35
> +      - 8
> +      - Context select

I'm not sure to understand what those four values are.

> +    * - 54
> +      - 8
> +      - Data pedestal bits 9--2
> +    * - 63
> +      - 8
> +      - Frame average bits 9--2
> +    * - 64
> +      - 16
> +      - Digital gain red
> +    * - 66
> +      - 16
> +      - Digital gain greenr
> +    * - 68
> +      - 16
> +      - Digital gain blue
> +    * - 70
> +      - 16
> +      - Digital gain greenb
> +    * - 89
> +      - 8
> +      - Frame counter (starts at 1, wraps to 0 after 255)
> diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
> index 03f7e9ab517b..13e68c2ccb61 100644
> --- a/include/uapi/linux/media-bus-format.h
> +++ b/include/uapi/linux/media-bus-format.h
> @@ -183,7 +183,8 @@
>  #define MEDIA_BUS_FMT_META_20			0x8006
>  #define MEDIA_BUS_FMT_META_24			0x8007
>  
> -/* Specific metadata formats. Next is 0x9002. */
> +/* Specific metadata formats. Next is 0x9003. */
>  #define MEDIA_BUS_FMT_CCS_EMBEDDED		0x9001
> +#define MEDIA_BUS_FMT_OV2740_EMBEDDED		0x9002
>  
>  #endif /* __LINUX_MEDIA_BUS_FORMAT_H */

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 38/46] media: ov2740: Remove shorthand variables

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Tue, Apr 16, 2024 at 10:33:11PM +0300, Sakari Ailus wrote:
> Remove two variables in ov2740_init_control() that are used as a shorthand
> for where the information is really located. Make the code more readable
> by removing them.

Dropping size is nice. I don't know if removing cur_mode makes the code
more readable, but if it does for you,

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  drivers/media/i2c/ov2740.c | 18 +++++++-----------
>  1 file changed, 7 insertions(+), 11 deletions(-)
> 
> diff --git a/drivers/media/i2c/ov2740.c b/drivers/media/i2c/ov2740.c
> index 57906df7be4e..196a111516b0 100644
> --- a/drivers/media/i2c/ov2740.c
> +++ b/drivers/media/i2c/ov2740.c
> @@ -754,10 +754,8 @@ static const struct v4l2_ctrl_ops ov2740_ctrl_ops = {
>  static int ov2740_init_controls(struct ov2740 *ov2740)
>  {
>  	struct v4l2_ctrl_handler *ctrl_hdlr;
> -	const struct ov2740_mode *cur_mode;
>  	s64 exposure_max, h_blank, pixel_rate;
>  	u32 vblank_min, vblank_max, vblank_default;
> -	int size;
>  	int ret;
>  
>  	ctrl_hdlr = &ov2740->ctrl_handler;
> @@ -765,12 +763,10 @@ static int ov2740_init_controls(struct ov2740 *ov2740)
>  	if (ret)
>  		return ret;
>  
> -	cur_mode = ov2740->cur_mode;
> -	size = ARRAY_SIZE(link_freq_menu_items);
> -
>  	ov2740->link_freq =
>  		v4l2_ctrl_new_int_menu(ctrl_hdlr, &ov2740_ctrl_ops,
> -				       V4L2_CID_LINK_FREQ, size - 1,
> +				       V4L2_CID_LINK_FREQ,
> +				       ARRAY_SIZE(link_freq_menu_items) - 1,
>  				       ov2740->supported_modes->link_freq_index,
>  				       link_freq_menu_items);
>  	if (ov2740->link_freq)
> @@ -781,14 +777,14 @@ static int ov2740_init_controls(struct ov2740 *ov2740)
>  					       V4L2_CID_PIXEL_RATE, 0,
>  					       pixel_rate, 1, pixel_rate);
>  
> -	vblank_min = cur_mode->vts_min - cur_mode->height;
> -	vblank_max = cur_mode->vts_max - cur_mode->height;
> -	vblank_default = cur_mode->vts_def - cur_mode->height;
> +	vblank_min = ov2740->cur_mode->vts_min - ov2740->cur_mode->height;
> +	vblank_max = ov2740->cur_mode->vts_max - ov2740->cur_mode->height;
> +	vblank_default = ov2740->cur_mode->vts_def - ov2740->cur_mode->height;
>  	ov2740->vblank = v4l2_ctrl_new_std(ctrl_hdlr, &ov2740_ctrl_ops,
>  					   V4L2_CID_VBLANK, vblank_min,
>  					   vblank_max, 1, vblank_default);
>  
> -	h_blank = cur_mode->hts - cur_mode->width;
> +	h_blank = ov2740->cur_mode->hts - ov2740->cur_mode->width;
>  	ov2740->hblank = v4l2_ctrl_new_std(ctrl_hdlr, &ov2740_ctrl_ops,
>  					   V4L2_CID_HBLANK, h_blank, h_blank, 1,
>  					   h_blank);
> @@ -801,7 +797,7 @@ static int ov2740_init_controls(struct ov2740 *ov2740)
>  	v4l2_ctrl_new_std(ctrl_hdlr, &ov2740_ctrl_ops, V4L2_CID_DIGITAL_GAIN,
>  			  OV2740_DGTL_GAIN_MIN, OV2740_DGTL_GAIN_MAX,
>  			  OV2740_DGTL_GAIN_STEP, OV2740_DGTL_GAIN_DEFAULT);
> -	exposure_max = cur_mode->vts_def - OV2740_EXPOSURE_MAX_MARGIN;
> +	exposure_max = ov2740->cur_mode->vts_def - OV2740_EXPOSURE_MAX_MARGIN;
>  	ov2740->exposure = v4l2_ctrl_new_std(ctrl_hdlr, &ov2740_ctrl_ops,
>  					     V4L2_CID_EXPOSURE,
>  					     OV2740_EXPOSURE_MIN, exposure_max,

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 41/46] media: ov2740: Add support for embedded data

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Tue, Apr 16, 2024 at 10:33:14PM +0300, Sakari Ailus wrote:
> Add support for embedded data. This introduces two internal pads for pixel
> and embedded data streams. As the driver supports a single mode only,
> there's no need for backward compatibility in mode selection.
> 
> The embedded data is configured to be placed before the image data whereas
> after the image data is the default.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  drivers/media/i2c/ov2740.c | 156 +++++++++++++++++++++++++++++++++----
>  1 file changed, 141 insertions(+), 15 deletions(-)
> 
> diff --git a/drivers/media/i2c/ov2740.c b/drivers/media/i2c/ov2740.c
> index c29e9b8cde15..dc0931308053 100644
> --- a/drivers/media/i2c/ov2740.c
> +++ b/drivers/media/i2c/ov2740.c
> @@ -11,6 +11,7 @@
>  #include <linux/pm_runtime.h>
>  #include <linux/nvmem-provider.h>
>  #include <linux/regmap.h>
> +#include <media/mipi-csi2.h>
>  #include <media/v4l2-ctrls.h>
>  #include <media/v4l2-device.h>
>  #include <media/v4l2-fwnode.h>
> @@ -71,11 +72,31 @@
>  #define OV2740_REG_ISP_CTRL00		0x5000
>  /* ISP CTRL01 */
>  #define OV2740_REG_ISP_CTRL01		0x5001
> +
> +/* Embedded data line location control */
> +#define OV2740_REG_EMBEDDED_FLAG	0x5a08
> +#define OV2740_EMBEDDED_FLAG_FOOTER	BIT(2) /* otherwise it's in header */
> +#define OV2740_EMBEDDED_FLAG_MYSTERY	BIT(1)
>  /* Customer Addresses: 0x7010 - 0x710F */
>  #define CUSTOMER_USE_OTP_SIZE		0x100
>  /* OTP registers from sensor */
>  #define OV2740_REG_OTP_CUSTOMER		0x7010
>  
> +enum {
> +	OV2740_PAD_SOURCE,
> +	OV2740_PAD_PIXEL,
> +	OV2740_PAD_META,
> +	OV2740_NUM_PADS,
> +};
> +
> +enum {
> +	OV2740_STREAM_PIXEL,
> +	OV2740_STREAM_META,
> +};
> +
> +#define OV2740_META_WIDTH		100U
> +#define OV2740_META_HEIGHT		1U
> +
>  struct nvm_data {
>  	struct nvmem_device *nvmem;
>  	struct regmap *regmap;
> @@ -149,6 +170,7 @@ static const struct ov2740_reg mode_1932x1092_regs_360mhz[] = {
>  	{0x3000, 0x00},
>  	{0x3018, 0x32},
>  	{0x3031, 0x0a},
> +	{0x3036, 0x12},

Please name this register and document the value.

>  	{0x3080, 0x08},
>  	{0x3083, 0xB4},
>  	{0x3103, 0x00},
> @@ -253,7 +275,7 @@ static const struct ov2740_reg mode_1932x1092_regs_360mhz[] = {
>  	{0x4017, 0x10},
>  	{0x4044, 0x02},
>  	{0x4304, 0x08},
> -	{0x4307, 0x30},
> +	{0x4307, 0x31},

Same here.

>  	{0x4320, 0x80},
>  	{0x4322, 0x00},
>  	{0x4323, 0x00},
> @@ -302,6 +324,7 @@ static const struct ov2740_reg mode_1932x1092_regs_180mhz[] = {
>  	{0x3000, 0x00},
>  	{0x3018, 0x32},	/* 0x32 for 2 lanes, 0x12 for 1 lane */
>  	{0x3031, 0x0a},
> +	{0x3036, 0x12},
>  	{0x3080, 0x08},
>  	{0x3083, 0xB4},
>  	{0x3103, 0x00},
> @@ -406,7 +429,7 @@ static const struct ov2740_reg mode_1932x1092_regs_180mhz[] = {
>  	{0x4017, 0x10},
>  	{0x4044, 0x02},
>  	{0x4304, 0x08},
> -	{0x4307, 0x30},
> +	{0x4307, 0x31},
>  	{0x4320, 0x80},
>  	{0x4322, 0x00},
>  	{0x4323, 0x00},
> @@ -513,7 +536,7 @@ static const struct ov2740_mode supported_modes_180mhz[] = {
>  
>  struct ov2740 {
>  	struct v4l2_subdev sd;
> -	struct media_pad pad;
> +	struct media_pad pads[OV2740_NUM_PADS];
>  	struct v4l2_ctrl_handler ctrl_handler;
>  
>  	/* V4L2 Controls */
> @@ -973,6 +996,11 @@ static int ov2740_enable_streams(struct v4l2_subdev *sd,
>  	if (ret)
>  		goto out_pm_put;
>  
> +	ret = ov2740_write_reg(ov2740, OV2740_REG_EMBEDDED_FLAG, 1,
> +			       OV2740_EMBEDDED_FLAG_MYSTERY);
> +	if (ret)
> +		goto out_pm_put;
> +
>  	ret = ov2740_write_reg(ov2740, OV2740_REG_MODE_SELECT, 1,
>  			       OV2740_MODE_STREAMING);
>  	if (ret) {
> @@ -1010,23 +1038,49 @@ static int ov2740_disable_streams(struct v4l2_subdev *sd,
>  	return ret;
>  }
>  
> -static int ov2740_set_format(struct v4l2_subdev *sd,
> -			     struct v4l2_subdev_state *sd_state,
> -			     struct v4l2_subdev_format *fmt)
> +static int __ov2740_set_format(struct v4l2_subdev *sd,
> +			       struct v4l2_subdev_state *sd_state,
> +			       struct v4l2_mbus_framefmt *format,
> +			       enum v4l2_subdev_format_whence which,
> +			       unsigned int pad, unsigned int stream)
>  {
> +	struct v4l2_mbus_framefmt *src_pix_fmt, *src_meta_fmt, *pix_fmt,
> +		*meta_fmt;
>  	struct ov2740 *ov2740 = to_ov2740(sd);
>  	const struct ov2740_mode *mode;
>  	s32 vblank_def, h_blank;
>  
> +	/*
> +	 * Allow setting format on internal pixel pad as well as the source
> +	 * pad's pixel stream (for compatibility).
> +	 */

As mentioned in the review of v8, the internal image pad represents the
pixel array, it should thus report the full pixel array size, and be
unaffected by the selected mode. We need to discuss how to expose raw
sensor parameters to userspace, which will affect this patch. I don't
think it's a good idea to expose a non-standard behaviour to userspace
before we agree on how to standardize it.

> +	if ((pad == OV2740_PAD_SOURCE && stream == OV2740_STREAM_META) ||
> +	    pad == OV2740_PAD_META) {
> +		*format = *v4l2_subdev_state_get_format(sd_state, pad, stream);
> +		return 0;
> +	}
> +
> +	pix_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_PIXEL, 0);
> +	meta_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_META, 0);
> +	src_pix_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_SOURCE,
> +						   OV2740_STREAM_PIXEL);
> +	src_meta_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_SOURCE,
> +						    OV2740_STREAM_META);
> +
>  	mode = v4l2_find_nearest_size(ov2740->supported_modes,
>  				      ov2740->supported_modes_count,
>  				      width, height,
> -				      fmt->format.width, fmt->format.height);
> +				      format->width, format->height);
> +	ov2740_update_pad_format(mode, pix_fmt);
> +	*format = *src_pix_fmt = *pix_fmt;
>  
> -	ov2740_update_pad_format(mode, &fmt->format);
> -	*v4l2_subdev_state_get_format(sd_state, fmt->pad) = fmt->format;
> +	meta_fmt->code = MEDIA_BUS_FMT_OV2740_EMBEDDED;
> +	meta_fmt->width = OV2740_META_WIDTH;
> +	meta_fmt->height = OV2740_META_HEIGHT;
> +	*src_meta_fmt = *meta_fmt;
> +	src_meta_fmt->code = MEDIA_BUS_FMT_META_10;
>  
> -	if (fmt->which == V4L2_SUBDEV_FORMAT_TRY)
> +	if (which == V4L2_SUBDEV_FORMAT_TRY)
>  		return 0;
>  
>  	ov2740->cur_mode = mode;
> @@ -1046,6 +1100,14 @@ static int ov2740_set_format(struct v4l2_subdev *sd,
>  	return 0;
>  }
>  
> +static int ov2740_set_format(struct v4l2_subdev *sd,
> +			     struct v4l2_subdev_state *sd_state,
> +			     struct v4l2_subdev_format *fmt)
> +{
> +	return __ov2740_set_format(sd, sd_state, &fmt->format, fmt->which,
> +				   fmt->pad, fmt->stream);
> +}
> +
>  static int ov2740_enum_mbus_code(struct v4l2_subdev *sd,
>  				 struct v4l2_subdev_state *sd_state,
>  				 struct v4l2_subdev_mbus_code_enum *code)
> @@ -1082,10 +1144,68 @@ static int ov2740_enum_frame_size(struct v4l2_subdev *sd,
>  static int ov2740_init_state(struct v4l2_subdev *sd,
>  			     struct v4l2_subdev_state *sd_state)
>  {
> +	struct v4l2_subdev_route routes[] = {
> +		{
> +			.sink_pad = OV2740_PAD_PIXEL,
> +			.source_pad = OV2740_PAD_SOURCE,
> +			.source_stream = OV2740_STREAM_PIXEL,
> +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> +		}, {
> +			.sink_pad = OV2740_PAD_META,
> +			.source_pad = OV2740_PAD_SOURCE,
> +			.source_stream = OV2740_STREAM_META,
> +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> +		},
> +	};
> +	struct v4l2_subdev_krouting routing = {
> +		.routes = routes,
> +		.num_routes = ARRAY_SIZE(routes),
> +	};
> +	struct v4l2_subdev_state *active_state;
> +	struct v4l2_mbus_framefmt format = { 0 };
>  	struct ov2740 *ov2740 = to_ov2740(sd);
> +	int ret;
> +
> +	ret = v4l2_subdev_set_routing(sd, sd_state, &routing);
> +	if (ret)
> +		return ret;
> +
> +	active_state = v4l2_subdev_get_locked_active_state(sd);

There's a lockdep assertion that will trip when initializing any try
state.

I'll stop reviewing this as comments on v8 haven't been taken into
account. Please revisit them for v10.

> +
> +	ov2740_update_pad_format(&ov2740->supported_modes[0], &format);
> +
> +	return __ov2740_set_format(sd, sd_state, &format,
> +				   active_state == sd_state ?
> +				   V4L2_SUBDEV_FORMAT_ACTIVE :
> +				   V4L2_SUBDEV_FORMAT_TRY, OV2740_PAD_PIXEL, 0);
> +}
> +
> +static int ov2740_get_frame_desc(struct v4l2_subdev *sd, unsigned int pad,
> +				 struct v4l2_mbus_frame_desc *desc)
> +{
> +	struct v4l2_mbus_frame_desc_entry *entry = desc->entry;
> +	struct v4l2_subdev_state *sd_state;
> +	struct v4l2_mbus_framefmt *fmt;
> +
> +	desc->type = V4L2_MBUS_FRAME_DESC_TYPE_CSI2;
> +
> +	sd_state = v4l2_subdev_lock_and_get_active_state(sd);
> +	fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_SOURCE,
> +					   OV2740_STREAM_PIXEL);
> +	entry->pixelcode = fmt->code;
> +	v4l2_subdev_unlock_state(sd_state);
> +
> +	entry->stream = OV2740_STREAM_PIXEL;
> +	entry->bus.csi2.dt = MIPI_CSI2_DT_RAW10;
> +	entry++;
> +	desc->num_entries++;
> +
> +	entry->pixelcode = MEDIA_BUS_FMT_META_10;
> +	entry->stream = OV2740_STREAM_META;
> +	entry->bus.csi2.dt = MIPI_CSI2_DT_EMBEDDED_8B;
> +	entry++;
> +	desc->num_entries++;
>  
> -	ov2740_update_pad_format(&ov2740->supported_modes[0],
> -				 v4l2_subdev_state_get_format(sd_state, 0));
>  	return 0;
>  }
>  
> @@ -1100,6 +1220,7 @@ static const struct v4l2_subdev_pad_ops ov2740_pad_ops = {
>  	.enum_frame_size = ov2740_enum_frame_size,
>  	.enable_streams = ov2740_enable_streams,
>  	.disable_streams = ov2740_disable_streams,
> +	.get_frame_desc = ov2740_get_frame_desc,
>  };
>  
>  static const struct v4l2_subdev_ops ov2740_subdev_ops = {
> @@ -1366,11 +1487,16 @@ static int ov2740_probe(struct i2c_client *client)
>  	}
>  
>  	ov2740->sd.state_lock = ov2740->ctrl_handler.lock;
> -	ov2740->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
> +	ov2740->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_STREAMS;
>  	ov2740->sd.entity.ops = &ov2740_subdev_entity_ops;
>  	ov2740->sd.entity.function = MEDIA_ENT_F_CAM_SENSOR;
> -	ov2740->pad.flags = MEDIA_PAD_FL_SOURCE;
> -	ret = media_entity_pads_init(&ov2740->sd.entity, 1, &ov2740->pad);
> +	ov2740->pads[OV2740_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
> +	ov2740->pads[OV2740_PAD_PIXEL].flags =
> +		MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL;
> +	ov2740->pads[OV2740_PAD_META].flags =
> +		MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL;
> +	ret = media_entity_pads_init(&ov2740->sd.entity,
> +				     ARRAY_SIZE(ov2740->pads), ov2740->pads);
>  	if (ret) {
>  		dev_err_probe(dev, ret, "failed to init entity pads\n");
>  		goto probe_error_v4l2_ctrl_handler_free;

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 42/46] media: ov2740: Add generic sensor fwnode properties as controls

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Tue, Apr 16, 2024 at 10:33:15PM +0300, Sakari Ailus wrote:
> Add generic sensor property information as controĺs by using
> v4l2_fwnode_device_parse() and v4l2_ctrl_new_fwnode_properties().
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  drivers/media/i2c/ov2740.c | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/drivers/media/i2c/ov2740.c b/drivers/media/i2c/ov2740.c
> index dc0931308053..e37d824291fe 100644
> --- a/drivers/media/i2c/ov2740.c
> +++ b/drivers/media/i2c/ov2740.c
> @@ -779,6 +779,8 @@ static const struct v4l2_ctrl_ops ov2740_ctrl_ops = {
>  
>  static int ov2740_init_controls(struct ov2740 *ov2740)
>  {
> +	struct i2c_client *client = v4l2_get_subdevdata(&ov2740->sd);
> +	struct v4l2_fwnode_device_properties props;
>  	struct v4l2_ctrl_handler *ctrl_hdlr;
>  	s64 exposure_max, h_blank, pixel_rate;
>  	u32 vblank_min, vblank_max, vblank_default;
> @@ -789,6 +791,10 @@ static int ov2740_init_controls(struct ov2740 *ov2740)
>  	if (ret)
>  		return ret;
>  
> +	if (!v4l2_fwnode_device_parse(&client->dev, &props))

If you moved the parsing earlier, you could set the right number of
controls when initializing the handler. This being said, maybe we should
instead try to get rid of the controls count hint to the handler
initialization function.

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> +		v4l2_ctrl_new_fwnode_properties(ctrl_hdlr, &ov2740_ctrl_ops,
> +						&props);
> +
>  	ov2740->link_freq =
>  		v4l2_ctrl_new_int_menu(ctrl_hdlr, &ov2740_ctrl_ops,
>  				       V4L2_CID_LINK_FREQ,

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 37/46] media: ov2740: Fix LINK_FREQ and PIXEL_RATE control value reporting

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Tue, Apr 16, 2024 at 10:33:10PM +0300, Sakari Ailus wrote:
> The driver dug the supported link frequency up from the V4L2 fwnode

s/dug/digs/

> endpoint and used it internally, but failed to report this in the

s/used/uses/
s/failed/fails/

> LINK_FREQ and PIXEL_RATE controls. Fix this.
> 
> Fixes: 0677a2d9b735 ("media: ov2740: Add support for 180 MHz link frequency")
> Cc: stable@vger.kernel.org # for v6.8 and later
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>

You're missing the tags given by Hans and Bingbu. As this patch is
unrelated to the rest of the series, it should be split off and merged
in v6.10.

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> ---
>  drivers/media/i2c/ov2740.c | 11 ++++++-----
>  1 file changed, 6 insertions(+), 5 deletions(-)
> 
> diff --git a/drivers/media/i2c/ov2740.c b/drivers/media/i2c/ov2740.c
> index 552935ccb4a9..57906df7be4e 100644
> --- a/drivers/media/i2c/ov2740.c
> +++ b/drivers/media/i2c/ov2740.c
> @@ -768,14 +768,15 @@ static int ov2740_init_controls(struct ov2740 *ov2740)
>  	cur_mode = ov2740->cur_mode;
>  	size = ARRAY_SIZE(link_freq_menu_items);
>  
> -	ov2740->link_freq = v4l2_ctrl_new_int_menu(ctrl_hdlr, &ov2740_ctrl_ops,
> -						   V4L2_CID_LINK_FREQ,
> -						   size - 1, 0,
> -						   link_freq_menu_items);
> +	ov2740->link_freq =
> +		v4l2_ctrl_new_int_menu(ctrl_hdlr, &ov2740_ctrl_ops,
> +				       V4L2_CID_LINK_FREQ, size - 1,
> +				       ov2740->supported_modes->link_freq_index,
> +				       link_freq_menu_items);
>  	if (ov2740->link_freq)
>  		ov2740->link_freq->flags |= V4L2_CTRL_FLAG_READ_ONLY;
>  
> -	pixel_rate = to_pixel_rate(OV2740_LINK_FREQ_360MHZ_INDEX);
> +	pixel_rate = to_pixel_rate(ov2740->supported_modes->link_freq_index);
>  	ov2740->pixel_rate = v4l2_ctrl_new_std(ctrl_hdlr, &ov2740_ctrl_ops,
>  					       V4L2_CID_PIXEL_RATE, 0,
>  					       pixel_rate, 1, pixel_rate);

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 43/46] media: ov2740: Add support for G_SELECTION IOCTL

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Tue, Apr 16, 2024 at 10:33:16PM +0300, Sakari Ailus wrote:
> Add support for the G_SELECTION IOCTL in the ov2740 driver.

We need to first define and document how G_SELECTION should behave for
raw sensors.

> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  drivers/media/i2c/ov2740.c | 31 +++++++++++++++++++++++++++++++
>  1 file changed, 31 insertions(+)
> 
> diff --git a/drivers/media/i2c/ov2740.c b/drivers/media/i2c/ov2740.c
> index e37d824291fe..6e355e986b88 100644
> --- a/drivers/media/i2c/ov2740.c
> +++ b/drivers/media/i2c/ov2740.c
> @@ -23,6 +23,11 @@
>  #define OV2740_DATA_LANES		2
>  #define OV2740_RGB_DEPTH		10
>  
> +#define OV2740_DUMMY_LINES_PRE		8U
> +#define OV2740_DUMMY_LINES_POST		8U
> +#define OV2740_ACTIVE_WIDTH		1932U
> +#define OV2740_ACTIVE_HEIGHT		1092U
> +
>  #define OV2740_REG_CHIP_ID		0x300a
>  #define OV2740_CHIP_ID			0x2740
>  
> @@ -1114,6 +1119,31 @@ static int ov2740_set_format(struct v4l2_subdev *sd,
>  				   fmt->pad, fmt->stream);
>  }
>  
> +static int ov2740_get_selection(struct v4l2_subdev *sd,
> +				struct v4l2_subdev_state *state,
> +				struct v4l2_subdev_selection *sel)
> +{
> +	switch (sel->target) {
> +	case V4L2_SEL_TGT_NATIVE_SIZE:
> +	case V4L2_SEL_TGT_CROP_BOUNDS:
> +		sel->r.top = 0;
> +		sel->r.left = 0;
> +		sel->r.width = OV2740_ACTIVE_WIDTH;
> +		sel->r.height = OV2740_DUMMY_LINES_PRE + OV2740_ACTIVE_HEIGHT +
> +			OV2740_DUMMY_LINES_POST;
> +		return 0;
> +	case V4L2_SEL_TGT_CROP:
> +	case V4L2_SEL_TGT_CROP_DEFAULT:
> +		sel->r.top = OV2740_DUMMY_LINES_PRE;
> +		sel->r.left = 0;
> +		sel->r.width = OV2740_ACTIVE_WIDTH;
> +		sel->r.height = OV2740_ACTIVE_HEIGHT;
> +		return 0;
> +	}
> +
> +	return -EINVAL;
> +}
> +
>  static int ov2740_enum_mbus_code(struct v4l2_subdev *sd,
>  				 struct v4l2_subdev_state *sd_state,
>  				 struct v4l2_subdev_mbus_code_enum *code)
> @@ -1222,6 +1252,7 @@ static const struct v4l2_subdev_video_ops ov2740_video_ops = {
>  static const struct v4l2_subdev_pad_ops ov2740_pad_ops = {
>  	.get_fmt = v4l2_subdev_get_fmt,
>  	.set_fmt = ov2740_set_format,
> +	.get_selection = ov2740_get_selection,
>  	.enum_mbus_code = ov2740_enum_mbus_code,
>  	.enum_frame_size = ov2740_enum_frame_size,
>  	.enable_streams = ov2740_enable_streams,

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 44/46] media: v4l: Add V4L2_SUBDEV_ROUTE_FL_IMMUTABLE sub-device routing flag

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Tue, Apr 16, 2024 at 10:33:17PM +0300, Sakari Ailus wrote:
> Add a flag to denote immutable routes, V4L2_SUBDEV_ROUTE_FL_IMMUTABLE.
> Such routes cannot be changed and they're always active.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  Documentation/userspace-api/media/v4l/dev-subdev.rst         | 4 +++-
>  .../userspace-api/media/v4l/vidioc-subdev-g-routing.rst      | 5 +++++
>  include/uapi/linux/v4l2-subdev.h                             | 5 +++++
>  3 files changed, 13 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> index de8dfd4f11a5..5c63c8c24108 100644
> --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> @@ -572,7 +572,9 @@ internal pad always has a single stream only (0).
>  Routes from an internal sink pad to an external source pad are typically not
>  modifiable but they can be activated and deactivated using the
>  :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE <v4l2-subdev-routing-flags>` flag, depending
> -on driver capabilities.
> +on driver capabilities. The :ref:`V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
> +<v4l2-subdev-routing-flags>` flag indicates that the
> +``V4L2_SUBDEV_ROUTE_FLAG_ACTIVE`` of the route may not be unset.
>  
>  Interaction between routes, streams, formats and selections
>  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> index 88df7bf80b00..c0f6d49cac6d 100644
> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> @@ -150,6 +150,11 @@ Also ``VIDIOC_SUBDEV_S_ROUTING`` may return more route than the user provided in
>      * - V4L2_SUBDEV_ROUTE_FL_ACTIVE
>        - 0x0001
>        - The route is enabled. Set by applications.
> +    * - V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
> +      - 0x0002
> +      - The route is immutable. Set by the driver. Indicates that the
> +	``V4L2_SUBDEV_ROUTE_FL_ACTIVE`` flag of an immutable route may not be
> +	unset.
>  
>  Return Value
>  ============
> diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
> index 6a39128d0606..c6f1228d43b1 100644
> --- a/include/uapi/linux/v4l2-subdev.h
> +++ b/include/uapi/linux/v4l2-subdev.h
> @@ -204,6 +204,11 @@ struct v4l2_subdev_capability {
>   * on a video node.
>   */
>  #define V4L2_SUBDEV_ROUTE_FL_ACTIVE		(1U << 0)
> +/*
> + * Is the route immutable. The ACTIVE flag of an immutable route may not be
> + * changed.

s/changed/unset/

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> + */
> +#define V4L2_SUBDEV_ROUTE_FL_IMMUTABLE		(1U << 1)
>  
>  /**
>   * struct v4l2_subdev_route - A route inside a subdev

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 00/46] Generic line based metadata support, internal pads

[Laurent Pinchart]

Hi Sakari,

Thank you for the patches. This is progressing nicely, and I think we
can merge part of this series for v6.10. Patches from 01/46 to 08/46 and
from 11/46 to 17/46 are nearly there, just a few of them will need a new
(and hopefully final) version. You could additionally merge the
CCS-specific patches 18/46 to 24/46 if desired. Some ov2740 rework could
go in too.

The rest of the patches (and in particular 09/46 and 10/46) depend on
internal pads support, which still needs discussions and additional
documentation for raw sensors.

When posting v10, please move 09/46 and 10/46 after 24/46.

On Tue, Apr 16, 2024 at 10:32:33PM +0300, Sakari Ailus wrote:
> Hi folks,
> 
> Here are a few patches to add support generic, line based metadata as well
> as internal pads. While the amount of code is not very large, to the
> contrary it is quite small actually IMO, I presume what this is about and
> why it is being proposed requires some explaining.
> 
> Metadata mbus codes and formats have existed for some time in V4L2. They
> however have been only used by drivers that produce the data itself and
> effectively this metadata has always been statistics of some sort (at
> least when it comes to ISPs). What is different here is that we intend to
> add support for metadata originating from camera sensors.
> 
> Camera sensors produce different kinds of metadata, embedded data (usually
> register address--value pairs used to capture the frame, in a more or less
> sensor specific format), histograms (in a very sensor specific format),
> dark pixels etc. The number of these formats is probably going to be about
> as large as image data formats if not larger, as the image data formats
> are much better standardised but a smaller subset of them will be
> supported by V4L2, at least initially but possibly much more in the long
> run.
> 
> Having this many device specific formats would be a major problem for all
> the other drivers along that pipeline (not to mention the users of those
> drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2
> receiver drivers that have DMA: the poor driver developer would not only
> need to know camera sensor specific formats but to choose the specific
> packing of that format suitable for the DMA used by the hardware. It is
> unlikely many of these would ever get tested while being present on the
> driver API. Also adding new sensors with new embedded data formats would
> involve updating all bridge and CSI-2 receiver drivers. I don't expect
> this to be a workable approach.
> 
> Instead what I'm proposing is to use specific metadata formats on the
> sensor devices only, on internal pads (more about those soon) of the
> sensors, only visible in the UAPI, and then generic mbus formats along the
> pipeline and finally generic V4L2 metadata formats on the DMAs (specific
> to bit depth and packing). This would unsnarl the two, defining what data
> there is (specific mbus code) and how that is transported and packed
> (generic mbus codes and V4L2 formats).
> 
> The user space would be required to "know" the path of that data from the
> sensor's internal pad to the V4L2 video node. I do not see this as these
> devices require at least some knowledge of the pipeline, i.e. hardware at
> hand. Separating what the data means and how it is packed may even be
> beneficial: it allows separating code that interprets the data (sensor
> internal mbus code) from the code that accesses it (packing).
> 
> These formats are in practice line based, meaning that there may be
> padding at the end of the line, depending on the bus as well as the DMA.
> If non-line based formats are needed, it is always possible to set the
> "height" field to 1.
> 
> The internal sink pads are an alternative to source routes [1]. The source
> routes were not universally liked and I do have to say I like re-using
> existing interface concepts (pads and everything you can do with pads,
> including access format, selections etc.) wherever it makes sense, instead
> of duplicating functionality.
> 
> Effectively internal sink pads behave mostly just like sink pads, but they
> describe a flow of data that originates from a sub-device instead of
> arriving to a sub-device. The SUBDEV_S_ROUTING IOCTLs are used to enable
> and disable routes from internal sink pads to sub-device's source pads.
> The subdev format IOCTLs are usable, too, so one can find which subdev
> format is available on given internal sink pad.
> 
> I've also pushed these here and I'll keep updating the branch, I've also
> included untested OV2740 patches:
> 
> <URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=metadata>
> 
> Questions and comments are most welcome.
> 
> Preliminary media-ctl and yavta patches can be found here:
> 
> <URL:https://git.retiisi.eu/?p=~sailus/yavta.git;a=shortlog;h=refs/heads/metadata>
> <URL:https://git.retiisi.eu/?p=~sailus/v4l-utils.git;a=shortlog;h=refs/heads/metadata>
> 
> I have used IMX219 as an example on routing in a sensor driver in this
> version.
> 
> The patches are on my master branch
> <URL:https://git.linuxtv.org/sailus/media_tree.git/log/>.
> 
> [1] <URL:https://lore.kernel.org/linux-media/20220831141357.1396081-20-tomi.valkeinen@ideasonboard.com/>
> 
> since v8:
> 
> - Move the patch adding internal pad flag past the routing API reworks, as
>   well as a few other patches, in order to separate the patches to those
>   that could still be merged for v6.10 (routing changes) and those that
>   couldn't (sensor API related). The patch on the edge is "media: uapi:
>   v4l: subdev: Enable streams API".
> 
> - Include Laurent's two patches to address crop API issues wrt. streams.
> 
> - Add two patches to prepare for CCS driver rework (media: ccs: Move
>   ccs_pm_get_init function up and media: ccs: Rename out label of
>   ccs_start_streaming).
> 
> - Address issues in the ov2740 driver patches (as well as the driver
>   itself), 4 more patches towards the end of the set.
> 
> - Improved generic metadata format names, align with other existing
>   formats.
> 
> - Improved ov2740 embedded data documentation.
> 
> - Reworked streams and camera sensor documentation based on Laurent's
>   comments mainly. In particular, the contradictory concept of internal
>   source pads no longer should exist in the patches.
> 
> - Fixed pad numbering in the CCS example.
> 
> - Fixed S_ROUTING behaviour when len_routes is too small and when
>   S_ROUTING isn't implemented by the driver.
> 
> - Reorder sections in meta-formats.rst alphabetically.
> 
> - Add a note per struct fields that certain struct v4l2_subdev_format are
>   zero for metadata mbus codes.
> 
> - CCS driver patch cleanups.
> 
> - CCS driver metadata width fix for space-efficient embedded data at 16
>   bpp and over.
> 
> - Postpone CCS frame descriptor quirk for now.
> 
> - Use MIPI_CSI2_DT_USER_DEFINED(0) instead of a numerical value for
>   compressed data datatype.
> 
> since v7:
> 
> - Add embedded data support for the ov2740 driver.
> 
> - Add three patches on top, to add an IMMUTABLE flag to source streams
>   when they cannot be disabled.
> 
> - Improved documentation of len_routes and num_routes arguments of
>   [GS]_ROUTING.
> 
> - Remove one inclusion of twice-included media/v4l2-fwnode.h in
>   drivers/media/i2c/ccs/ccs-core.c .
> 
> - Add missing forward declaration of ccs_internal_ops in
>   drivers/media/i2c/ccs/ccs-core.c .
> 
> since v6:
> 
> - Improve embedded data UAPI documentation on camera sensors.
> 
> - Improve wording of stream glossary entry.
> 
> - Improve internal pad flag documentation.
> 
> - Fix definition of "data unit" and remove an extra "only" in INTERNAL pad
>   flag description (1st patch).
> 
> - Use IMX219 driver in examples consistently.
> 
> - Remove the CSI-2 to parallel bridge from the example to simplify the
>   example.
> 
> - Minor rewording of some parts of the routing examples.
> 
> - Rebase on unified sub-device state information access functions:
>   <URL:https://lore.kernel.org/linux-media/20231027095913.1010187-1-sakari.ailus@linux.intel.com/T/#t>
> 
> - In CCS driver, do not maintain current active configuration in driver's
>   device context struct (apart from mbus codes). Rely on sub-device state
>   locking and clean up the code. (Multiple patches towards the end of the
>   set.)
> 
> - Arrange the CCS patches early in the set towards the end of the set.
> 
> - Move the patch enabling streams API to the end of the set.
> 
> - Rework IOCTL argument copying condition for [GS]_ROUTING).
> 
> - Handle copying back routes in S_ROUTING, do not rely on G_ROUTING
>   IOCTL implementation.
> 
> - Rebase on metadata preparation patchset v6:
>   <URL:https://lore.kernel.org/linux-media/20231106121805.1266696-1-sakari.ailus@linux.intel.com/T/#t>.
> 
> since v5:
> 
> - Rebase on new set of preparation patches.
> 
> - Switch CCS driver from s_stream to enable_streams/disable_streams. Keep
>   streaming state information --- the sensor remains in streaming state if
>   any of the streams is enabled.
> 
> - Fix setting mbus code on embedded data in get_frame_desc() op in the CCS
>   driver.
> 
> since v4:
> 
> - Add a patch to acquire two sub-device states that may use the same lock.
> 
> - Add a patch for CCS driver to remove ccs_get_crop_compose() helper.
> 
> - Add a patch for CCS driver moving acquiring and releasing the mutex to
>   the s_stream callback.
> 
> - Add a patch for CCS driver to rely on sub-device state locking using a
>   single driver-provided lock.
> 
> - Fixed calculating minimum number of routes in copying the routes
>   (thanks, Laurent).
> 
> - Moved a label in S_ROUTING handling to make Clang happy (hopefully).
> 
> - Fixed setting emb_data_ctrl register for CCS embedded data support.
> 
> - Rebase on Laurent's cleanup patches.
> 
> - Wrap a few long lines.
> 
> - Write in embedded data documentation sensor drivers generally don't
>   allow configuring it.
> 
> since v3:
> 
> - Separate preparation patches from this set.
> 
> - Add a definition for "Data unit", a pixel that is not image data and use
>   it instead in format documentation.
> 
> - Fix more numbered lists in dev-subdev.rst.
> 
> - Remove a redundant definition for V4L2_META_FMT_GENERIC_CSI2_2_24 ---
>   V4L2_META_FMT_GENERIC_CSI2_12 can be used instead.
> 
> - Use "X" instead of "p" to denote padding in format documentation.
> 
> - Use IMX219 in examples instead of CCS.
> 
> - Document that the generic V4L2 CSI-2 metadata formats use padding
>   defined in CSI-2 spec and packing defined in CCS spec.
> 
> - Add patches to align [GS]_ROUTING behaviour with V4L2. This means mainly
>   returning configured routes as part of S_ROUTING as well. "len_routes"
>   field is added to denote the length of the array and having more routes
>   than fits in the array is no longer an error. Also added more reserved
>   fields.
> 
> - Added trivial support for S_ROUTING (via G_ROUTING implementation) for
>   use in drivers with static-only routes.
> 
> - Added helper functions to obtain mbus format as well as crop and compose
>   rectangles that are streams-independent.
> 
> - Added a patch to define generic CSI-2 long packet types.
> 
> - Removed MEDIA_BUS_FMT_IS_META() macro. It didn't seem useful in the end.
> 
> - Use a single CCS embedded data format. The bit depth can be selected
>   using the meta stream on the source pad.
> 
> - Fix mbus code numbers (there were holes due to removed redundant
>   formats).
> 
> - Fix generic mbus code documentation (byte was being used instead of
>   bit).
> 
> - Fix spelling of "length".
> 
> - Added a patch to remove v4l2_subdev_enable_streams_api that disables
>   streams API. This should be merged once libcamera support for streams
>   works nicely.
> 
> - Don't use strings in printing frame descriptor flags.
> 
> - Warn on string truncation in printing frame descriptor.
> 
> since v2:
> 
> - Add a better example, with formats.
> 
> - Add CCS static data media bus codes.
> 
> - Added an example demonstrating the use of internal pads. --- Is the
>   level of detail enough for the purpose?
> 
> - Improved documentation.
> 
> - Added a macro to tell whether a format is a metadata format.
>   (Documentation could be added.)
> 
> - A small ReST syntax fix in the same section.
> 
> - Drop leftovers of a patch checking for the INTERNAL_SOURCE flag.
> 
> since v1:
> 
> - Make the new pad flag just "INTERNAL", requiring either SINK or SOURCE
>   pad flag to accompany it. Removed the union in struct v4l2_subdev_route.
> 
> - Add the term "stream" to MC glossary.
> 
> - Improved and fixed documentation (according to comments).
> 
> - Note these formats are little endian.
> 
> - Remove 1X8 from the names of the mbus codes. These formats have generally
>   8 bits per pixel.
> 
> - Fix mbus code numbering (had holes in RFC).
> 
> - Add new metadata fields to debug prints.
> 
> - Fix a minor documentation build issue.
> 
> Laurent Pinchart (2):
>   media: v4l2-subdev: Fix stream handling for crop API
>   media: v4l2-subdev: Clearly document that the crop API won't be
>     extended
> 
> Sakari Ailus (44):
>   media: Documentation: Add "stream" into glossary
>   media: uapi: Add generic serial metadata mbus formats
>   media: uapi: Document which mbus format fields are valid for metadata
>   media: uapi: v4l: Add generic 8-bit metadata format definitions
>   media: v4l: Support line-based metadata capture
>   media: Documentation: Additional streams generally don't harm capture
>   media: Documentation: Document embedded data guidelines for camera
>     sensors
>   media: Documentation: v4l: Document internal sink pads
>   media: Documentation: Document S_ROUTING behaviour
>   media: v4l: subdev: Add a function to lock two sub-device states, use
>     it
>   media: v4l: subdev: Move G_ROUTING handling below S_ROUTING
>   media: v4l: subdev: Copy argument back to user also for S_ROUTING
>   media: v4l: subdev: Add len_routes field to struct v4l2_subdev_routing
>   media: v4l: subdev: Return routes set using S_ROUTING
>   media: v4l: subdev: Add trivial set_routing support
>   media: ccs: No need to set streaming to false in power off
>   media: ccs: Move ccs_pm_get_init function up
>   media: ccs: Rename out label of ccs_start_streaming
>   media: ccs: Use {enable,disable}_streams operations
>   media: ccs: Track streaming state
>   media: ccs: Move ccs_validate_csi_data_format up
>   media: ccs: Support frame descriptors
>   media: uapi: v4l: subdev: Enable streams API
>   media: mc: Add INTERNAL pad flag
>   media: uapi: ccs: Add media bus code for MIPI CCS embedded data
>   media: Documentation: Document non-CCS use of CCS embedded data format
>   media: Documentation: ccs: Document routing
>   media: ccs: Add support for embedded data stream
>   media: ccs: Remove ccs_get_crop_compose helper
>   media: ccs: Rely on sub-device state locking
>   media: ccs: Compute binning configuration from sub-device state
>   media: ccs: Compute scaling configuration from sub-device state
>   media: ccs: Remove which parameter from ccs_propagate
>   media: uapi: Add media bus code for ov2740 embedded data
>   media: ov2740: Fix LINK_FREQ and PIXEL_RATE control value reporting
>   media: ov2740: Remove shorthand variables
>   media: ov2740: Switch to {enable,disable}_streams
>   media: ov2740: Track streaming state
>   media: ov2740: Add support for embedded data
>   media: ov2740: Add generic sensor fwnode properties as controls
>   media: ov2740: Add support for G_SELECTION IOCTL
>   media: v4l: Add V4L2_SUBDEV_ROUTE_FL_IMMUTABLE sub-device routing flag
>   media: ccs: Add IMMUTABLE route flag
>   media: ov2740: Add IMMUTABLE route flag
> 
>  .../media/drivers/camera-sensor.rst           |   32 +
>  .../userspace-api/media/drivers/ccs.rst       |   38 +-
>  .../userspace-api/media/glossary.rst          |   15 +
>  .../media/mediactl/media-types.rst            |    9 +
>  .../userspace-api/media/v4l/dev-meta.rst      |   21 +
>  .../userspace-api/media/v4l/dev-subdev.rst    |  179 ++-
>  .../userspace-api/media/v4l/meta-formats.rst  |    3 +-
>  .../media/v4l/metafmt-generic.rst             |  328 +++++
>  .../media/v4l/subdev-formats.rst              |  374 +++++-
>  .../media/v4l/vidioc-enum-fmt.rst             |    7 +
>  .../media/v4l/vidioc-subdev-g-crop.rst        |    6 +-
>  .../media/v4l/vidioc-subdev-g-routing.rst     |   60 +-
>  .../media/videodev2.h.rst.exceptions          |    1 +
>  drivers/media/i2c/ccs/ccs-core.c              | 1050 +++++++++++------
>  drivers/media/i2c/ccs/ccs.h                   |   27 +-
>  drivers/media/i2c/ov2740.c                    |  304 +++--
>  drivers/media/mc/mc-entity.c                  |   15 +-
>  drivers/media/v4l2-core/v4l2-ioctl.c          |   25 +-
>  drivers/media/v4l2-core/v4l2-subdev.c         |  118 +-
>  include/media/v4l2-subdev.h                   |   42 +
>  include/uapi/linux/media-bus-format.h         |   13 +
>  include/uapi/linux/media.h                    |    1 +
>  include/uapi/linux/v4l2-mediabus.h            |   18 +-
>  include/uapi/linux/v4l2-subdev.h              |   18 +-
>  include/uapi/linux/videodev2.h                |   18 +
>  25 files changed, 2183 insertions(+), 539 deletions(-)
>  create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 06/46] media: uapi: v4l: Add generic 8-bit metadata format definitions

[Sakari Ailus]

Hi Laurent,

On Fri, Apr 19, 2024 at 07:26:19PM +0300, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.
> 
> On Tue, Apr 16, 2024 at 10:32:39PM +0300, Sakari Ailus wrote:
> > Generic 8-bit metadata formats define the in-memory data layout but not
> > the format of the data itself. The reasoning for having such formats is to
> > allow CSI-2 receiver drivers to receive and DMA drivers to write the data
> > to memory without knowing a large number of device-specific formats.
> > 
> > These formats may be used only in conjunction with a Media controller
> 
> "Media Controller" or "media controller"

Yes.

> 
> > pipeline where the internal pad of the source sub-device defines the
> > specific format of the data (using an mbus code).
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  .../userspace-api/media/v4l/dev-subdev.rst    |   2 +
> >  .../userspace-api/media/v4l/meta-formats.rst  |   3 +-
> >  .../media/v4l/metafmt-generic.rst             | 328 ++++++++++++++++++
> >  drivers/media/v4l2-core/v4l2-ioctl.c          |   7 +
> >  include/uapi/linux/videodev2.h                |   8 +
> >  5 files changed, 347 insertions(+), 1 deletion(-)
> >  create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > index 43988516acdd..f375b820ab68 100644
> > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > @@ -506,6 +506,8 @@ source pads.
> >  
> >      subdev-formats
> >  
> > +.. _subdev-routing:
> > +
> >  Streams, multiplexed media pads and internal routing
> >  ----------------------------------------------------
> >  
> > diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
> > index 0bb61fc5bc00..c23aac823d2c 100644
> > --- a/Documentation/userspace-api/media/v4l/meta-formats.rst
> > +++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
> > @@ -13,9 +13,10 @@ These formats are used for the :ref:`metadata` interface only.
> >      :maxdepth: 1
> >  
> >      metafmt-d4xx
> > +    metafmt-generic
> >      metafmt-intel-ipu3
> >      metafmt-rkisp1
> >      metafmt-uvc
> > +    metafmt-vivid
> >      metafmt-vsp1-hgo
> >      metafmt-vsp1-hgt
> > -    metafmt-vivid
> > diff --git a/Documentation/userspace-api/media/v4l/metafmt-generic.rst b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > new file mode 100644
> > index 000000000000..cba34be64dfe
> > --- /dev/null
> > +++ b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > @@ -0,0 +1,328 @@
> > +.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
> > +
> > +********************************************************************************************************************************************************************************************************************************************************************************
> > +V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O')
> > +********************************************************************************************************************************************************************************************************************************************************************************
> > +
> > +
> > +Generic line-based metadata formats
> > +
> > +
> > +Description
> > +===========
> > +
> > +These generic line-based metadata formats define the memory layout of the data
> > +without defining the format or meaning of the metadata itself. These formats may
> > +only be used with a Media Controller pipeline where the more specific format is
> > +reported by an :ref:`internal sink pad <MEDIA-PAD-FL-INTERNAL>` of the source
> > +sub-device. See also :ref:`source routes <subdev-routing>`.
> > +
> > +.. _v4l2-meta-fmt-generic-8:
> > +
> > +V4L2_META_FMT_GENERIC_8
> > +-----------------------
> > +
> > +The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format.
> > +
> > +This format is also used on CSI-2 for both 8 bits per ``Data Unit
> > +<media-glossary-data-unit>`` as well as for 16 bits per Data Unit when two bytes
> > +of metadata are packed into one 16-bit Data Unit.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_8.**
> > +Each cell is one byte. "M" denotes a byte of metadata.
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|
> > +
> > +.. flat-table:: Sample 4x2 Metadata Frame
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - M\ :sub:`10`
> > +      - M\ :sub:`20`
> > +      - M\ :sub:`30`
> > +    * - start + 4:
> > +      - M\ :sub:`01`
> > +      - M\ :sub:`11`
> > +      - M\ :sub:`21`
> > +      - M\ :sub:`31`
> > +
> > +.. _v4l2-meta-fmt-generic-csi2-10:
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_10
> > +-----------------------------
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_10 contains 8-bit generic metadata packed in 10-bit
> > +Data Units, with one padding byte after every four bytes of metadata. This
> > +format is typically used by CSI-2 receivers with a source that transmits
> > +MEDIA_BUS_FMT_META_10 and the CSI-2 receiver writes the received data to memory
> > +as-is.
> > +
> > +The packing of the data follows the MIPI CSI-2 specification and the padding of
> > +the data is defined in the MIPI CCS specification.
> > +
> > +This format is also used in conjunction with 20 bits per ``Data Unit
> 
> s/in conjunction with/for/
> 
> Same below.

That is shorter but is it better?

> 
> > +<media-glossary-data-unit>`` formats that pack two bytes of metadata into one
> > +Data Unit.
> 
> How about adding a mention of MEDIA_BUS_FMT_META_20 here ? The CCS
> specification calls this "RAW20 (with more optimal RAW10 packing)",
> using the same vocabulary could help. Same for
> V4L2_META_FMT_GENERIC_CSI2_12, and actually V4L2_META_FMT_GENERIC_8 too.

Is it useful? It's the same as 10-bit packing after all.

RAW20 in CCS refers to image data.

> 
> > +
> > +This format is little endian.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
> > +Each cell is one byte. "M" denotes a byte of metadata and "x" a byte of padding.
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
> > +
> > +.. flat-table:: Sample 4x2 Metadata Frame
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - M\ :sub:`10`
> > +      - M\ :sub:`20`
> > +      - M\ :sub:`30`
> > +      - x
> > +    * - start + 5:
> > +      - M\ :sub:`01`
> > +      - M\ :sub:`11`
> > +      - M\ :sub:`21`
> > +      - M\ :sub:`31`
> > +      - x
> > +
> > +.. _v4l2-meta-fmt-generic-csi2-12:
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_12
> > +-----------------------------
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_12 contains 8-bit generic metadata packed in 12-bit
> > +Data Units, with two padding bytes after every four bytes of metadata. This
> 
> Isn't it one padding byte every two bytes of metadata ?

Uh, yes. This was mistakenly copied from text for a different bpp.

> 
> > +format is typically used by CSI-2 receivers with a source that transmits
> > +MEDIA_BUS_FMT_META_12 and the CSI-2 receiver writes the received data to memory
> > +as-is.
> > +
> > +The packing of the data follows the MIPI CSI-2 specification and the padding of
> > +the data is defined in the MIPI CCS specification.
> > +
> > +This format is also used in conjunction with 24 bits per ``Data Unit
> > +<media-glossary-data-unit>`` formats that pack two bytes of metadata into one
> > +Data Unit.
> > +
> > +This format is little endian.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_12.**
> > +Each cell is one byte. "M" denotes a byte of metadata and "x" a byte of padding.
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> > +
> > +.. flat-table:: Sample 4x2 Metadata Frame
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - M\ :sub:`10`
> > +      - x
> > +      - M\ :sub:`20`
> > +      - M\ :sub:`30`
> > +      - x
> > +    * - start + 6:
> > +      - M\ :sub:`01`
> > +      - M\ :sub:`11`
> > +      - x
> > +      - M\ :sub:`21`
> > +      - M\ :sub:`31`
> > +      - x
> > +
> > +.. _v4l2-meta-fmt-generic-csi2-14:
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_14
> > +-----------------------------
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_14 contains 8-bit generic metadata packed in 14-bit
> > +Data Units, with three padding bytes after every four bytes of metadata. This
> > +format is typically used by CSI-2 receivers with a source that transmits
> > +MEDIA_BUS_FMT_META_14 and the CSI-2 receiver writes the received data to memory
> > +as-is.
> > +
> > +The packing of the data follows the MIPI CSI-2 specification and the padding of
> > +the data is defined in the MIPI CCS specification.
> > +
> > +This format is little endian.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_14.**
> > +Each cell is one byte. "M" denotes a byte of metadata and "x" a byte of padding.
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{.8cm}|
> > +
> > +.. flat-table:: Sample 4x2 Metadata Frame
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - M\ :sub:`10`
> > +      - M\ :sub:`20`
> > +      - M\ :sub:`30`
> > +      - x
> > +      - x
> > +      - x
> > +    * - start + 7:
> > +      - M\ :sub:`01`
> > +      - M\ :sub:`11`
> > +      - M\ :sub:`21`
> > +      - M\ :sub:`31`
> > +      - x
> > +      - x
> > +      - x
> > +
> > +.. _v4l2-meta-fmt-generic-csi2-16:
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_16
> > +-----------------------------
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_16 contains 8-bit generic metadata packed in 16-bit
> > +Data Units, with one padding byte after every byte of metadata. This format is
> > +typically used by CSI-2 receivers with a source that transmits
> > +MEDIA_BUS_FMT_META_16 and the CSI-2 receiver writes the received data to memory
> > +as-is.
> > +
> > +The packing of the data follows the MIPI CSI-2 specification and the padding of
> > +the data is defined in the MIPI CCS specification.
> 
> Here you could also mention that more optimal packing schemes for
> MEDIA_BUS_FMT_META_16 use V4L2_META_FMT_GENERIC_8. Same below.

The "more optimal packing scheme" is really a different format. There's no
other relation between these, really. I wouldn't mention it here. A driver
would enumerate both for metadata when they're supported.

> 
> > +
> > +This format is little endian.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_16.**
> > +Each cell is one byte. "M" denotes a byte of metadata and "x" a byte of padding.
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|
> > +
> > +.. flat-table:: Sample 4x2 Metadata Frame
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - x
> > +      - M\ :sub:`10`
> > +      - x
> > +      - M\ :sub:`20`
> > +      - x
> > +      - M\ :sub:`30`
> > +      - x
> > +    * - start + 8:
> > +      - M\ :sub:`01`
> > +      - x
> > +      - M\ :sub:`11`
> > +      - x
> > +      - M\ :sub:`21`
> > +      - x
> > +      - M\ :sub:`31`
> > +      - x
> > +
> > +.. _v4l2-meta-fmt-generic-csi2-20:
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_20
> > +-----------------------------
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_20 contains 8-bit generic metadata packed in 20-bit
> > +Data Units, with alternating one or two padding bytes after every byte of
> > +metadata. This format is typically used by CSI-2 receivers with a source that
> > +transmits MEDIA_BUS_FMT_META_20 and the CSI-2 receiver writes the received data
> > +to memory as-is.
> > +
> > +The packing of the data follows the MIPI CSI-2 specification and the padding of
> > +the data is defined in the MIPI CCS specification.
> > +
> > +This format is little endian.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_20.**
> > +Each cell is one byte. "M" denotes a byte of metadata and "x" a byte of padding.
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> > +
> > +.. flat-table:: Sample 4x2 Metadata Frame
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8 8 8 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - x
> > +      - M\ :sub:`10`
> > +      - x
> > +      - x
> > +      - M\ :sub:`20`
> > +      - x
> > +      - M\ :sub:`30`
> > +      - x
> > +      - x
> > +    * - start + 10:
> > +      - M\ :sub:`01`
> > +      - x
> > +      - M\ :sub:`11`
> > +      - x
> > +      - x
> > +      - M\ :sub:`21`
> > +      - x
> > +      - M\ :sub:`31`
> > +      - x
> > +      - x
> > +
> > +.. _v4l2-meta-fmt-generic-csi2-24:
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_24
> > +-----------------------------
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_24 contains 8-bit generic metadata packed in 24-bit
> > +Data Units, with two padding bytes after every byte of metadata. This format is
> > +typically used by CSI-2 receivers with a source that transmits
> > +MEDIA_BUS_FMT_META_24 and the CSI-2 receiver writes the received data to memory
> > +as-is.
> > +
> > +The packing of the data follows the MIPI CSI-2 specification and the padding of
> > +the data is defined in the MIPI CCS specification.
> > +
> > +This format is little endian.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_24.**
> > +Each cell is one byte. "M" denotes a byte of metadata and "x" a byte of padding.
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> > +
> > +.. flat-table:: Sample 4x2 Metadata Frame
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8 8 8 8 8 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - x
> > +      - x
> > +      - M\ :sub:`10`
> > +      - x
> > +      - x
> > +      - M\ :sub:`20`
> > +      - x
> > +      - x
> > +      - M\ :sub:`30`
> > +      - x
> > +      - x
> > +    * - start + 12:
> > +      - M\ :sub:`01`
> > +      - x
> > +      - x
> > +      - M\ :sub:`11`
> > +      - x
> > +      - x
> > +      - M\ :sub:`21`
> > +      - x
> > +      - x
> > +      - M\ :sub:`31`
> > +      - x
> > +      - x
> > diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> > index 5aeff5519407..ae2dca7f2817 100644
> > --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> > +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> > @@ -1460,6 +1460,13 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
> >  	case V4L2_PIX_FMT_Y210:		descr = "10-bit YUYV Packed"; break;
> >  	case V4L2_PIX_FMT_Y212:		descr = "12-bit YUYV Packed"; break;
> >  	case V4L2_PIX_FMT_Y216:		descr = "16-bit YUYV Packed"; break;
> > +	case V4L2_META_FMT_GENERIC_8:	descr = "8-bit Generic Metadata"; break;
> > +	case V4L2_META_FMT_GENERIC_CSI2_10:	descr = "8-bit Generic Meta, 10b CSI-2"; break;
> > +	case V4L2_META_FMT_GENERIC_CSI2_12:	descr = "8-bit Generic Meta, 12b CSI-2"; break;
> > +	case V4L2_META_FMT_GENERIC_CSI2_14:	descr = "8-bit Generic Meta, 14b CSI-2"; break;
> > +	case V4L2_META_FMT_GENERIC_CSI2_16:	descr = "8-bit Generic Meta, 16b CSI-2"; break;
> > +	case V4L2_META_FMT_GENERIC_CSI2_20:	descr = "8-bit Generic Meta, 20b CSI-2"; break;
> > +	case V4L2_META_FMT_GENERIC_CSI2_24:	descr = "8-bit Generic Meta, 24b CSI-2"; break;
> >  
> >  	default:
> >  		/* Compressed formats */
> > diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> > index 2663213b76a4..c7cf20b5da67 100644
> > --- a/include/uapi/linux/videodev2.h
> > +++ b/include/uapi/linux/videodev2.h
> > @@ -839,6 +839,14 @@ struct v4l2_pix_format {
> >  #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
> >  #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
> >  
> > +#define V4L2_META_FMT_GENERIC_8		v4l2_fourcc('M', 'E', 'T', '8') /* Generic 8-bit metadata */
> > +#define V4L2_META_FMT_GENERIC_CSI2_10	v4l2_fourcc('M', 'C', '1', 'A') /* 10-bit CSI-2 packed 8-bit metadata */
> > +#define V4L2_META_FMT_GENERIC_CSI2_12	v4l2_fourcc('M', 'C', '1', 'C') /* 12-bit CSI-2 packed 8-bit metadata */
> > +#define V4L2_META_FMT_GENERIC_CSI2_14	v4l2_fourcc('M', 'C', '1', 'E') /* 14-bit CSI-2 packed 8-bit metadata */
> > +#define V4L2_META_FMT_GENERIC_CSI2_16	v4l2_fourcc('M', 'C', '1', 'G') /* 16-bit CSI-2 packed 8-bit metadata */
> > +#define V4L2_META_FMT_GENERIC_CSI2_20	v4l2_fourcc('M', 'C', '1', 'K') /* 20-bit CSI-2 packed 8-bit metadata */
> > +#define V4L2_META_FMT_GENERIC_CSI2_24	v4l2_fourcc('M', 'C', '1', 'O') /* 24-bit CSI-2 packed 8-bit metadata */
> > +
> >  /* priv field value to indicates that subsequent fields are valid. */
> >  #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
> >  
> 

-- 
Kind regards,

Sakari Ailus

Re: [PATCH v9 07/46] media: v4l: Support line-based metadata capture

[Sakari Ailus]

Hi Laurent,

On Fri, Apr 19, 2024 at 07:30:09PM +0300, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.

Thanks for the review.

> 
> On Tue, Apr 16, 2024 at 10:32:40PM +0300, Sakari Ailus wrote:
> > Many camera sensors, among other devices, transmit embedded data and image
> > data for each CSI-2 frame. This embedded data typically contains register
> > configuration of the sensor that has been used to capture the image data
> > of the same frame.
> > 
> > The embedded data is received by the CSI-2 receiver and has the same
> > properties as the image data, including that it is line based: it has
> > width, height and bytesperline (stride).
> > 
> > Add these fields to struct v4l2_meta_format and document them.
> > 
> > Also add V4L2_FMT_FLAG_META_LINE_BASED to tell a given format is
> > line-based i.e. these fields of struct v4l2_meta_format are valid for it.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  .../userspace-api/media/v4l/dev-meta.rst      | 21 +++++++++++++++++++
> >  .../media/v4l/vidioc-enum-fmt.rst             |  7 +++++++
> >  .../media/videodev2.h.rst.exceptions          |  1 +
> >  drivers/media/v4l2-core/v4l2-ioctl.c          |  5 +++--
> >  include/uapi/linux/videodev2.h                | 10 +++++++++
> >  5 files changed, 42 insertions(+), 2 deletions(-)
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/dev-meta.rst b/Documentation/userspace-api/media/v4l/dev-meta.rst
> > index 0e7e1ee1471a..5eee9ab60395 100644
> > --- a/Documentation/userspace-api/media/v4l/dev-meta.rst
> > +++ b/Documentation/userspace-api/media/v4l/dev-meta.rst
> > @@ -47,6 +47,12 @@ member of the ``fmt`` union as needed per the desired operation. Both drivers
> >  and applications must set the remainder of the :c:type:`v4l2_format` structure
> >  to 0.
> >  
> > +Devices that capture metadata by line have the struct v4l2_fmtdesc
> > +``V4L2_FMT_FLAG_META_LINE_BASED`` flag set for :c:func:`VIDIOC_ENUM_FMT`. Such
> > +devices can typically also :ref:`capture image data <capture>`. This primarily
> > +involves devices that receive the data from a different devices such as a camera
> > +sensor.
> > +
> >  .. c:type:: v4l2_meta_format
> >  
> >  .. tabularcolumns:: |p{1.4cm}|p{2.4cm}|p{13.5cm}|
> > @@ -65,3 +71,18 @@ to 0.
> >        - ``buffersize``
> >        - Maximum buffer size in bytes required for data. The value is set by the
> >          driver.
> > +    * - __u32
> > +      - ``width``
> > +      - Width of a line of metadata in Data Units. Valid when
> > +	:c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set,
> > +	otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
> > +    * - __u32
> > +      - ``height``
> > +      - Number of rows of metadata. Valid when :c:type`v4l2_fmtdesc` flag
> > +	``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
> > +	:c:func:`VIDIOC_ENUM_FMT`.
> > +    * - __u32
> > +      - ``bytesperline``
> > +      - Offset in bytes between the beginning of two consecutive lines. Valid
> > +	when :c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is
> > +	set, otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
> > diff --git a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> > index 000c154b0f98..a439be1b15d1 100644
> > --- a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> > +++ b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> > @@ -227,6 +227,13 @@ the ``mbus_code`` field is handled differently:
> >  	The application can ask to configure the quantization of the capture
> >  	device when calling the :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl with
> >  	:ref:`V4L2_PIX_FMT_FLAG_SET_CSC <v4l2-pix-fmt-flag-set-csc>` set.
> > +    * - ``V4L2_FMT_FLAG_META_LINE_BASED``
> > +      - 0x0200
> > +      - The metadata format is line-based. In this case the ``width``,
> > +	``height`` and ``bytesperline`` fields of :c:type:`v4l2_meta_format` are
> > +	valid. The buffer consists of ``height`` lines, each having ``width``
> > +	Data Units of data and offset (in bytes) between the beginning of each
> 
> s/and offset/, and the offset/

Sounds good.

> 
> > +	two consecutive lines is ``bytesperline``.
> >  
> >  Return Value
> >  ============
> > diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b/Documentation/userspace-api/media/videodev2.h.rst.exceptions
> > index 3e58aac4ef0b..bdc628e8c1d6 100644
> > --- a/Documentation/userspace-api/media/videodev2.h.rst.exceptions
> > +++ b/Documentation/userspace-api/media/videodev2.h.rst.exceptions
> > @@ -215,6 +215,7 @@ replace define V4L2_FMT_FLAG_CSC_XFER_FUNC fmtdesc-flags
> >  replace define V4L2_FMT_FLAG_CSC_YCBCR_ENC fmtdesc-flags
> >  replace define V4L2_FMT_FLAG_CSC_HSV_ENC fmtdesc-flags
> >  replace define V4L2_FMT_FLAG_CSC_QUANTIZATION fmtdesc-flags
> > +replace define V4L2_FMT_FLAG_META_LINE_BASED fmtdesc-flags
> >  
> >  # V4L2 timecode types
> >  replace define V4L2_TC_TYPE_24FPS timecode-type
> > diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> > index ae2dca7f2817..2cfc9106857a 100644
> > --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> > +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> > @@ -343,8 +343,9 @@ static void v4l_print_format(const void *arg, bool write_only)
> >  	case V4L2_BUF_TYPE_META_OUTPUT:
> >  		meta = &p->fmt.meta;
> >  		pixelformat = meta->dataformat;
> > -		pr_cont(", dataformat=%p4cc, buffersize=%u\n",
> > -			&pixelformat, meta->buffersize);
> > +		pr_cont(", dataformat=%p4cc, buffersize=%u, width=%u, height=%u, bytesperline=%u\n",
> > +			&pixelformat, meta->buffersize, meta->width,
> > +			meta->height, meta->bytesperline);
> >  		break;
> >  	}
> >  }
> > diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> > index c7cf20b5da67..37112dfebd0c 100644
> > --- a/include/uapi/linux/videodev2.h
> > +++ b/include/uapi/linux/videodev2.h
> > @@ -877,6 +877,7 @@ struct v4l2_fmtdesc {
> >  #define V4L2_FMT_FLAG_CSC_YCBCR_ENC		0x0080
> >  #define V4L2_FMT_FLAG_CSC_HSV_ENC		V4L2_FMT_FLAG_CSC_YCBCR_ENC
> >  #define V4L2_FMT_FLAG_CSC_QUANTIZATION		0x0100
> > +#define V4L2_FMT_FLAG_META_LINE_BASED		0x0200
> 
> Could the V4L2_FMT_FLAG_META_LINE_BASED flag be set by the V4L2 core for
> line-based metadata formats, instead of requiring drivers to set it ? It
> would ensure consistency.

This would essentially need to be set after driver callbacks dealing with a
format-related IOCTL has returned, drivers don't need this information as
such.

I can add a separate patch for it.

> 
> >  
> >  	/* Frame Size and frame rate enumeration */
> >  /*
> > @@ -2424,10 +2425,19 @@ struct v4l2_sdr_format {
> >   * struct v4l2_meta_format - metadata format definition
> >   * @dataformat:		little endian four character code (fourcc)
> >   * @buffersize:		maximum size in bytes required for data
> > + * @width:		number of data units of data per line (valid for line
> > + *			based formats only, see format documentation)
> > + * @height:		number of lines of data per buffer (valid for line based
> > + *			formats only)
> > + * @bytesperline:	offset between the beginnings of two adjacent lines in
> > + *			bytes (valid for line based formats only)
> >   */
> >  struct v4l2_meta_format {
> >  	__u32				dataformat;
> >  	__u32				buffersize;
> > +	__u32				width;
> > +	__u32				height;
> > +	__u32				bytesperline;
> >  } __attribute__ ((packed));
> >  
> >  /**
> 

-- 
Regards,

Sakari Ailus

Re: [PATCH v9 06/46] media: uapi: v4l: Add generic 8-bit metadata format definitions

[Laurent Pinchart]

Hi Sakari,

On Tue, Apr 23, 2024 at 07:04:58AM +0000, Sakari Ailus wrote:
> On Fri, Apr 19, 2024 at 07:26:19PM +0300, Laurent Pinchart wrote:
> > On Tue, Apr 16, 2024 at 10:32:39PM +0300, Sakari Ailus wrote:
> > > Generic 8-bit metadata formats define the in-memory data layout but not
> > > the format of the data itself. The reasoning for having such formats is to
> > > allow CSI-2 receiver drivers to receive and DMA drivers to write the data
> > > to memory without knowing a large number of device-specific formats.
> > > 
> > > These formats may be used only in conjunction with a Media controller
> > 
> > "Media Controller" or "media controller"
> 
> Yes.
> 
> > > pipeline where the internal pad of the source sub-device defines the
> > > specific format of the data (using an mbus code).
> > > 
> > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > ---
> > >  .../userspace-api/media/v4l/dev-subdev.rst    |   2 +
> > >  .../userspace-api/media/v4l/meta-formats.rst  |   3 +-
> > >  .../media/v4l/metafmt-generic.rst             | 328 ++++++++++++++++++
> > >  drivers/media/v4l2-core/v4l2-ioctl.c          |   7 +
> > >  include/uapi/linux/videodev2.h                |   8 +
> > >  5 files changed, 347 insertions(+), 1 deletion(-)
> > >  create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > > 
> > > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > index 43988516acdd..f375b820ab68 100644
> > > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > @@ -506,6 +506,8 @@ source pads.
> > >  
> > >      subdev-formats
> > >  
> > > +.. _subdev-routing:
> > > +
> > >  Streams, multiplexed media pads and internal routing
> > >  ----------------------------------------------------
> > >  
> > > diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
> > > index 0bb61fc5bc00..c23aac823d2c 100644
> > > --- a/Documentation/userspace-api/media/v4l/meta-formats.rst
> > > +++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
> > > @@ -13,9 +13,10 @@ These formats are used for the :ref:`metadata` interface only.
> > >      :maxdepth: 1
> > >  
> > >      metafmt-d4xx
> > > +    metafmt-generic
> > >      metafmt-intel-ipu3
> > >      metafmt-rkisp1
> > >      metafmt-uvc
> > > +    metafmt-vivid
> > >      metafmt-vsp1-hgo
> > >      metafmt-vsp1-hgt
> > > -    metafmt-vivid
> > > diff --git a/Documentation/userspace-api/media/v4l/metafmt-generic.rst b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > > new file mode 100644
> > > index 000000000000..cba34be64dfe
> > > --- /dev/null
> > > +++ b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > > @@ -0,0 +1,328 @@
> > > +.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
> > > +
> > > +********************************************************************************************************************************************************************************************************************************************************************************
> > > +V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O')
> > > +********************************************************************************************************************************************************************************************************************************************************************************
> > > +
> > > +
> > > +Generic line-based metadata formats
> > > +
> > > +
> > > +Description
> > > +===========
> > > +
> > > +These generic line-based metadata formats define the memory layout of the data
> > > +without defining the format or meaning of the metadata itself. These formats may
> > > +only be used with a Media Controller pipeline where the more specific format is
> > > +reported by an :ref:`internal sink pad <MEDIA-PAD-FL-INTERNAL>` of the source
> > > +sub-device. See also :ref:`source routes <subdev-routing>`.
> > > +
> > > +.. _v4l2-meta-fmt-generic-8:
> > > +
> > > +V4L2_META_FMT_GENERIC_8
> > > +-----------------------
> > > +
> > > +The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format.
> > > +
> > > +This format is also used on CSI-2 for both 8 bits per ``Data Unit
> > > +<media-glossary-data-unit>`` as well as for 16 bits per Data Unit when two bytes
> > > +of metadata are packed into one 16-bit Data Unit.
> > > +
> > > +**Byte Order Of V4L2_META_FMT_GENERIC_8.**
> > > +Each cell is one byte. "M" denotes a byte of metadata.
> > > +
> > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|
> > > +
> > > +.. flat-table:: Sample 4x2 Metadata Frame
> > > +    :header-rows:  0
> > > +    :stub-columns: 0
> > > +    :widths: 12 8 8 8 8
> > > +
> > > +    * - start + 0:
> > > +      - M\ :sub:`00`
> > > +      - M\ :sub:`10`
> > > +      - M\ :sub:`20`
> > > +      - M\ :sub:`30`
> > > +    * - start + 4:
> > > +      - M\ :sub:`01`
> > > +      - M\ :sub:`11`
> > > +      - M\ :sub:`21`
> > > +      - M\ :sub:`31`
> > > +
> > > +.. _v4l2-meta-fmt-generic-csi2-10:
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_10
> > > +-----------------------------
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_10 contains 8-bit generic metadata packed in 10-bit
> > > +Data Units, with one padding byte after every four bytes of metadata. This
> > > +format is typically used by CSI-2 receivers with a source that transmits
> > > +MEDIA_BUS_FMT_META_10 and the CSI-2 receiver writes the received data to memory
> > > +as-is.
> > > +
> > > +The packing of the data follows the MIPI CSI-2 specification and the padding of
> > > +the data is defined in the MIPI CCS specification.
> > > +
> > > +This format is also used in conjunction with 20 bits per ``Data Unit
> > 
> > s/in conjunction with/for/
> > 
> > Same below.
> 
> That is shorter but is it better?

I think it's simpler and thus clearer. Up to you.

> > > +<media-glossary-data-unit>`` formats that pack two bytes of metadata into one
> > > +Data Unit.
> > 
> > How about adding a mention of MEDIA_BUS_FMT_META_20 here ? The CCS
> > specification calls this "RAW20 (with more optimal RAW10 packing)",
> > using the same vocabulary could help. Same for
> > V4L2_META_FMT_GENERIC_CSI2_12, and actually V4L2_META_FMT_GENERIC_8 too.
> 
> Is it useful? It's the same as 10-bit packing after all.

I think cross-referencing the two entries that can be used to transmit
metadata for the same image format is useful. 

> RAW20 in CCS refers to image data.

The wording I quoted comes from the CCS specification :-) Using the same
terms as in the spec would help in my opinion.

> > > +
> > > +This format is little endian.
> > > +
> > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
> > > +Each cell is one byte. "M" denotes a byte of metadata and "x" a byte of padding.
> > > +
> > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
> > > +
> > > +.. flat-table:: Sample 4x2 Metadata Frame
> > > +    :header-rows:  0
> > > +    :stub-columns: 0
> > > +    :widths: 12 8 8 8 8 8
> > > +
> > > +    * - start + 0:
> > > +      - M\ :sub:`00`
> > > +      - M\ :sub:`10`
> > > +      - M\ :sub:`20`
> > > +      - M\ :sub:`30`
> > > +      - x
> > > +    * - start + 5:
> > > +      - M\ :sub:`01`
> > > +      - M\ :sub:`11`
> > > +      - M\ :sub:`21`
> > > +      - M\ :sub:`31`
> > > +      - x
> > > +
> > > +.. _v4l2-meta-fmt-generic-csi2-12:
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_12
> > > +-----------------------------
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_12 contains 8-bit generic metadata packed in 12-bit
> > > +Data Units, with two padding bytes after every four bytes of metadata. This
> > 
> > Isn't it one padding byte every two bytes of metadata ?
> 
> Uh, yes. This was mistakenly copied from text for a different bpp.
> 
> > > +format is typically used by CSI-2 receivers with a source that transmits
> > > +MEDIA_BUS_FMT_META_12 and the CSI-2 receiver writes the received data to memory
> > > +as-is.
> > > +
> > > +The packing of the data follows the MIPI CSI-2 specification and the padding of
> > > +the data is defined in the MIPI CCS specification.
> > > +
> > > +This format is also used in conjunction with 24 bits per ``Data Unit
> > > +<media-glossary-data-unit>`` formats that pack two bytes of metadata into one
> > > +Data Unit.
> > > +
> > > +This format is little endian.
> > > +
> > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_12.**
> > > +Each cell is one byte. "M" denotes a byte of metadata and "x" a byte of padding.
> > > +
> > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> > > +
> > > +.. flat-table:: Sample 4x2 Metadata Frame
> > > +    :header-rows:  0
> > > +    :stub-columns: 0
> > > +    :widths: 12 8 8 8 8 8 8
> > > +
> > > +    * - start + 0:
> > > +      - M\ :sub:`00`
> > > +      - M\ :sub:`10`
> > > +      - x
> > > +      - M\ :sub:`20`
> > > +      - M\ :sub:`30`
> > > +      - x
> > > +    * - start + 6:
> > > +      - M\ :sub:`01`
> > > +      - M\ :sub:`11`
> > > +      - x
> > > +      - M\ :sub:`21`
> > > +      - M\ :sub:`31`
> > > +      - x
> > > +
> > > +.. _v4l2-meta-fmt-generic-csi2-14:
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_14
> > > +-----------------------------
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_14 contains 8-bit generic metadata packed in 14-bit
> > > +Data Units, with three padding bytes after every four bytes of metadata. This
> > > +format is typically used by CSI-2 receivers with a source that transmits
> > > +MEDIA_BUS_FMT_META_14 and the CSI-2 receiver writes the received data to memory
> > > +as-is.
> > > +
> > > +The packing of the data follows the MIPI CSI-2 specification and the padding of
> > > +the data is defined in the MIPI CCS specification.
> > > +
> > > +This format is little endian.
> > > +
> > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_14.**
> > > +Each cell is one byte. "M" denotes a byte of metadata and "x" a byte of padding.
> > > +
> > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{.8cm}|
> > > +
> > > +.. flat-table:: Sample 4x2 Metadata Frame
> > > +    :header-rows:  0
> > > +    :stub-columns: 0
> > > +    :widths: 12 8 8 8 8 8 8 8
> > > +
> > > +    * - start + 0:
> > > +      - M\ :sub:`00`
> > > +      - M\ :sub:`10`
> > > +      - M\ :sub:`20`
> > > +      - M\ :sub:`30`
> > > +      - x
> > > +      - x
> > > +      - x
> > > +    * - start + 7:
> > > +      - M\ :sub:`01`
> > > +      - M\ :sub:`11`
> > > +      - M\ :sub:`21`
> > > +      - M\ :sub:`31`
> > > +      - x
> > > +      - x
> > > +      - x
> > > +
> > > +.. _v4l2-meta-fmt-generic-csi2-16:
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_16
> > > +-----------------------------
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_16 contains 8-bit generic metadata packed in 16-bit
> > > +Data Units, with one padding byte after every byte of metadata. This format is
> > > +typically used by CSI-2 receivers with a source that transmits
> > > +MEDIA_BUS_FMT_META_16 and the CSI-2 receiver writes the received data to memory
> > > +as-is.
> > > +
> > > +The packing of the data follows the MIPI CSI-2 specification and the padding of
> > > +the data is defined in the MIPI CCS specification.
> > 
> > Here you could also mention that more optimal packing schemes for
> > MEDIA_BUS_FMT_META_16 use V4L2_META_FMT_GENERIC_8. Same below.
> 
> The "more optimal packing scheme" is really a different format. There's no
> other relation between these, really. I wouldn't mention it here. A driver
> would enumerate both for metadata when they're supported.

It's a different format, but the two are linked to the same image
format. That's why I think it's useful to cross-reference them.

> > > +
> > > +This format is little endian.
> > > +
> > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_16.**
> > > +Each cell is one byte. "M" denotes a byte of metadata and "x" a byte of padding.
> > > +
> > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|
> > > +
> > > +.. flat-table:: Sample 4x2 Metadata Frame
> > > +    :header-rows:  0
> > > +    :stub-columns: 0
> > > +    :widths: 12 8 8 8 8 8 8 8 8
> > > +
> > > +    * - start + 0:
> > > +      - M\ :sub:`00`
> > > +      - x
> > > +      - M\ :sub:`10`
> > > +      - x
> > > +      - M\ :sub:`20`
> > > +      - x
> > > +      - M\ :sub:`30`
> > > +      - x
> > > +    * - start + 8:
> > > +      - M\ :sub:`01`
> > > +      - x
> > > +      - M\ :sub:`11`
> > > +      - x
> > > +      - M\ :sub:`21`
> > > +      - x
> > > +      - M\ :sub:`31`
> > > +      - x
> > > +
> > > +.. _v4l2-meta-fmt-generic-csi2-20:
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_20
> > > +-----------------------------
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_20 contains 8-bit generic metadata packed in 20-bit
> > > +Data Units, with alternating one or two padding bytes after every byte of
> > > +metadata. This format is typically used by CSI-2 receivers with a source that
> > > +transmits MEDIA_BUS_FMT_META_20 and the CSI-2 receiver writes the received data
> > > +to memory as-is.
> > > +
> > > +The packing of the data follows the MIPI CSI-2 specification and the padding of
> > > +the data is defined in the MIPI CCS specification.
> > > +
> > > +This format is little endian.
> > > +
> > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_20.**
> > > +Each cell is one byte. "M" denotes a byte of metadata and "x" a byte of padding.
> > > +
> > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> > > +
> > > +.. flat-table:: Sample 4x2 Metadata Frame
> > > +    :header-rows:  0
> > > +    :stub-columns: 0
> > > +    :widths: 12 8 8 8 8 8 8 8 8 8 8
> > > +
> > > +    * - start + 0:
> > > +      - M\ :sub:`00`
> > > +      - x
> > > +      - M\ :sub:`10`
> > > +      - x
> > > +      - x
> > > +      - M\ :sub:`20`
> > > +      - x
> > > +      - M\ :sub:`30`
> > > +      - x
> > > +      - x
> > > +    * - start + 10:
> > > +      - M\ :sub:`01`
> > > +      - x
> > > +      - M\ :sub:`11`
> > > +      - x
> > > +      - x
> > > +      - M\ :sub:`21`
> > > +      - x
> > > +      - M\ :sub:`31`
> > > +      - x
> > > +      - x
> > > +
> > > +.. _v4l2-meta-fmt-generic-csi2-24:
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_24
> > > +-----------------------------
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_24 contains 8-bit generic metadata packed in 24-bit
> > > +Data Units, with two padding bytes after every byte of metadata. This format is
> > > +typically used by CSI-2 receivers with a source that transmits
> > > +MEDIA_BUS_FMT_META_24 and the CSI-2 receiver writes the received data to memory
> > > +as-is.
> > > +
> > > +The packing of the data follows the MIPI CSI-2 specification and the padding of
> > > +the data is defined in the MIPI CCS specification.
> > > +
> > > +This format is little endian.
> > > +
> > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_24.**
> > > +Each cell is one byte. "M" denotes a byte of metadata and "x" a byte of padding.
> > > +
> > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> > > +
> > > +.. flat-table:: Sample 4x2 Metadata Frame
> > > +    :header-rows:  0
> > > +    :stub-columns: 0
> > > +    :widths: 12 8 8 8 8 8 8 8 8 8 8 8 8
> > > +
> > > +    * - start + 0:
> > > +      - M\ :sub:`00`
> > > +      - x
> > > +      - x
> > > +      - M\ :sub:`10`
> > > +      - x
> > > +      - x
> > > +      - M\ :sub:`20`
> > > +      - x
> > > +      - x
> > > +      - M\ :sub:`30`
> > > +      - x
> > > +      - x
> > > +    * - start + 12:
> > > +      - M\ :sub:`01`
> > > +      - x
> > > +      - x
> > > +      - M\ :sub:`11`
> > > +      - x
> > > +      - x
> > > +      - M\ :sub:`21`
> > > +      - x
> > > +      - x
> > > +      - M\ :sub:`31`
> > > +      - x
> > > +      - x
> > > diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> > > index 5aeff5519407..ae2dca7f2817 100644
> > > --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> > > +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> > > @@ -1460,6 +1460,13 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
> > >  	case V4L2_PIX_FMT_Y210:		descr = "10-bit YUYV Packed"; break;
> > >  	case V4L2_PIX_FMT_Y212:		descr = "12-bit YUYV Packed"; break;
> > >  	case V4L2_PIX_FMT_Y216:		descr = "16-bit YUYV Packed"; break;
> > > +	case V4L2_META_FMT_GENERIC_8:	descr = "8-bit Generic Metadata"; break;
> > > +	case V4L2_META_FMT_GENERIC_CSI2_10:	descr = "8-bit Generic Meta, 10b CSI-2"; break;
> > > +	case V4L2_META_FMT_GENERIC_CSI2_12:	descr = "8-bit Generic Meta, 12b CSI-2"; break;
> > > +	case V4L2_META_FMT_GENERIC_CSI2_14:	descr = "8-bit Generic Meta, 14b CSI-2"; break;
> > > +	case V4L2_META_FMT_GENERIC_CSI2_16:	descr = "8-bit Generic Meta, 16b CSI-2"; break;
> > > +	case V4L2_META_FMT_GENERIC_CSI2_20:	descr = "8-bit Generic Meta, 20b CSI-2"; break;
> > > +	case V4L2_META_FMT_GENERIC_CSI2_24:	descr = "8-bit Generic Meta, 24b CSI-2"; break;
> > >  
> > >  	default:
> > >  		/* Compressed formats */
> > > diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> > > index 2663213b76a4..c7cf20b5da67 100644
> > > --- a/include/uapi/linux/videodev2.h
> > > +++ b/include/uapi/linux/videodev2.h
> > > @@ -839,6 +839,14 @@ struct v4l2_pix_format {
> > >  #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
> > >  #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
> > >  
> > > +#define V4L2_META_FMT_GENERIC_8		v4l2_fourcc('M', 'E', 'T', '8') /* Generic 8-bit metadata */
> > > +#define V4L2_META_FMT_GENERIC_CSI2_10	v4l2_fourcc('M', 'C', '1', 'A') /* 10-bit CSI-2 packed 8-bit metadata */
> > > +#define V4L2_META_FMT_GENERIC_CSI2_12	v4l2_fourcc('M', 'C', '1', 'C') /* 12-bit CSI-2 packed 8-bit metadata */
> > > +#define V4L2_META_FMT_GENERIC_CSI2_14	v4l2_fourcc('M', 'C', '1', 'E') /* 14-bit CSI-2 packed 8-bit metadata */
> > > +#define V4L2_META_FMT_GENERIC_CSI2_16	v4l2_fourcc('M', 'C', '1', 'G') /* 16-bit CSI-2 packed 8-bit metadata */
> > > +#define V4L2_META_FMT_GENERIC_CSI2_20	v4l2_fourcc('M', 'C', '1', 'K') /* 20-bit CSI-2 packed 8-bit metadata */
> > > +#define V4L2_META_FMT_GENERIC_CSI2_24	v4l2_fourcc('M', 'C', '1', 'O') /* 24-bit CSI-2 packed 8-bit metadata */
> > > +
> > >  /* priv field value to indicates that subsequent fields are valid. */
> > >  #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
> > >  

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 06/46] media: uapi: v4l: Add generic 8-bit metadata format definitions

[Sakari Ailus]

Hi Laurent,

On Tue, Apr 23, 2024 at 10:32:37AM +0300, Laurent Pinchart wrote:
> Hi Sakari,
> 
> On Tue, Apr 23, 2024 at 07:04:58AM +0000, Sakari Ailus wrote:
> > On Fri, Apr 19, 2024 at 07:26:19PM +0300, Laurent Pinchart wrote:
> > > On Tue, Apr 16, 2024 at 10:32:39PM +0300, Sakari Ailus wrote:
> > > > Generic 8-bit metadata formats define the in-memory data layout but not
> > > > the format of the data itself. The reasoning for having such formats is to
> > > > allow CSI-2 receiver drivers to receive and DMA drivers to write the data
> > > > to memory without knowing a large number of device-specific formats.
> > > > 
> > > > These formats may be used only in conjunction with a Media controller
> > > 
> > > "Media Controller" or "media controller"
> > 
> > Yes.
> > 
> > > > pipeline where the internal pad of the source sub-device defines the
> > > > specific format of the data (using an mbus code).
> > > > 
> > > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > > ---
> > > >  .../userspace-api/media/v4l/dev-subdev.rst    |   2 +
> > > >  .../userspace-api/media/v4l/meta-formats.rst  |   3 +-
> > > >  .../media/v4l/metafmt-generic.rst             | 328 ++++++++++++++++++
> > > >  drivers/media/v4l2-core/v4l2-ioctl.c          |   7 +
> > > >  include/uapi/linux/videodev2.h                |   8 +
> > > >  5 files changed, 347 insertions(+), 1 deletion(-)
> > > >  create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > > > 
> > > > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > > index 43988516acdd..f375b820ab68 100644
> > > > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > > @@ -506,6 +506,8 @@ source pads.
> > > >  
> > > >      subdev-formats
> > > >  
> > > > +.. _subdev-routing:
> > > > +
> > > >  Streams, multiplexed media pads and internal routing
> > > >  ----------------------------------------------------
> > > >  
> > > > diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
> > > > index 0bb61fc5bc00..c23aac823d2c 100644
> > > > --- a/Documentation/userspace-api/media/v4l/meta-formats.rst
> > > > +++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
> > > > @@ -13,9 +13,10 @@ These formats are used for the :ref:`metadata` interface only.
> > > >      :maxdepth: 1
> > > >  
> > > >      metafmt-d4xx
> > > > +    metafmt-generic
> > > >      metafmt-intel-ipu3
> > > >      metafmt-rkisp1
> > > >      metafmt-uvc
> > > > +    metafmt-vivid
> > > >      metafmt-vsp1-hgo
> > > >      metafmt-vsp1-hgt
> > > > -    metafmt-vivid
> > > > diff --git a/Documentation/userspace-api/media/v4l/metafmt-generic.rst b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > > > new file mode 100644
> > > > index 000000000000..cba34be64dfe
> > > > --- /dev/null
> > > > +++ b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > > > @@ -0,0 +1,328 @@
> > > > +.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
> > > > +
> > > > +********************************************************************************************************************************************************************************************************************************************************************************
> > > > +V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O')
> > > > +********************************************************************************************************************************************************************************************************************************************************************************
> > > > +
> > > > +
> > > > +Generic line-based metadata formats
> > > > +
> > > > +
> > > > +Description
> > > > +===========
> > > > +
> > > > +These generic line-based metadata formats define the memory layout of the data
> > > > +without defining the format or meaning of the metadata itself. These formats may
> > > > +only be used with a Media Controller pipeline where the more specific format is
> > > > +reported by an :ref:`internal sink pad <MEDIA-PAD-FL-INTERNAL>` of the source
> > > > +sub-device. See also :ref:`source routes <subdev-routing>`.
> > > > +
> > > > +.. _v4l2-meta-fmt-generic-8:
> > > > +
> > > > +V4L2_META_FMT_GENERIC_8
> > > > +-----------------------
> > > > +
> > > > +The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format.
> > > > +
> > > > +This format is also used on CSI-2 for both 8 bits per ``Data Unit
> > > > +<media-glossary-data-unit>`` as well as for 16 bits per Data Unit when two bytes
> > > > +of metadata are packed into one 16-bit Data Unit.
> > > > +
> > > > +**Byte Order Of V4L2_META_FMT_GENERIC_8.**
> > > > +Each cell is one byte. "M" denotes a byte of metadata.
> > > > +
> > > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|
> > > > +
> > > > +.. flat-table:: Sample 4x2 Metadata Frame
> > > > +    :header-rows:  0
> > > > +    :stub-columns: 0
> > > > +    :widths: 12 8 8 8 8
> > > > +
> > > > +    * - start + 0:
> > > > +      - M\ :sub:`00`
> > > > +      - M\ :sub:`10`
> > > > +      - M\ :sub:`20`
> > > > +      - M\ :sub:`30`
> > > > +    * - start + 4:
> > > > +      - M\ :sub:`01`
> > > > +      - M\ :sub:`11`
> > > > +      - M\ :sub:`21`
> > > > +      - M\ :sub:`31`
> > > > +
> > > > +.. _v4l2-meta-fmt-generic-csi2-10:
> > > > +
> > > > +V4L2_META_FMT_GENERIC_CSI2_10
> > > > +-----------------------------
> > > > +
> > > > +V4L2_META_FMT_GENERIC_CSI2_10 contains 8-bit generic metadata packed in 10-bit
> > > > +Data Units, with one padding byte after every four bytes of metadata. This
> > > > +format is typically used by CSI-2 receivers with a source that transmits
> > > > +MEDIA_BUS_FMT_META_10 and the CSI-2 receiver writes the received data to memory
> > > > +as-is.
> > > > +
> > > > +The packing of the data follows the MIPI CSI-2 specification and the padding of
> > > > +the data is defined in the MIPI CCS specification.
> > > > +
> > > > +This format is also used in conjunction with 20 bits per ``Data Unit
> > > 
> > > s/in conjunction with/for/
> > > 
> > > Same below.
> > 
> > That is shorter but is it better?
> 
> I think it's simpler and thus clearer. Up to you.

The relation is indirect so I prefer the original text.

> 
> > > > +<media-glossary-data-unit>`` formats that pack two bytes of metadata into one
> > > > +Data Unit.
> > > 
> > > How about adding a mention of MEDIA_BUS_FMT_META_20 here ? The CCS
> > > specification calls this "RAW20 (with more optimal RAW10 packing)",
> > > using the same vocabulary could help. Same for
> > > V4L2_META_FMT_GENERIC_CSI2_12, and actually V4L2_META_FMT_GENERIC_8 too.
> > 
> > Is it useful? It's the same as 10-bit packing after all.
> 
> I think cross-referencing the two entries that can be used to transmit
> metadata for the same image format is useful. 
> 
> > RAW20 in CCS refers to image data.
> 
> The wording I quoted comes from the CCS specification :-) Using the same
> terms as in the spec would help in my opinion.

RAW20 still refers to image data and we don't use that term in V4L2
documentation elsewhere. In the CCS spec the connection between the two is
stronger than in V4L2 as we support a number of different formats.

But I can mention the other format.

> 
> > > > +
> > > > +This format is little endian.
> > > > +
> > > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
> > > > +Each cell is one byte. "M" denotes a byte of metadata and "x" a byte of padding.
> > > > +
> > > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
> > > > +
> > > > +.. flat-table:: Sample 4x2 Metadata Frame
> > > > +    :header-rows:  0
> > > > +    :stub-columns: 0
> > > > +    :widths: 12 8 8 8 8 8
> > > > +
> > > > +    * - start + 0:
> > > > +      - M\ :sub:`00`
> > > > +      - M\ :sub:`10`
> > > > +      - M\ :sub:`20`
> > > > +      - M\ :sub:`30`
> > > > +      - x
> > > > +    * - start + 5:
> > > > +      - M\ :sub:`01`
> > > > +      - M\ :sub:`11`
> > > > +      - M\ :sub:`21`
> > > > +      - M\ :sub:`31`
> > > > +      - x
> > > > +
> > > > +.. _v4l2-meta-fmt-generic-csi2-12:
> > > > +
> > > > +V4L2_META_FMT_GENERIC_CSI2_12
> > > > +-----------------------------
> > > > +
> > > > +V4L2_META_FMT_GENERIC_CSI2_12 contains 8-bit generic metadata packed in 12-bit
> > > > +Data Units, with two padding bytes after every four bytes of metadata. This
> > > 
> > > Isn't it one padding byte every two bytes of metadata ?
> > 
> > Uh, yes. This was mistakenly copied from text for a different bpp.
> > 
> > > > +format is typically used by CSI-2 receivers with a source that transmits
> > > > +MEDIA_BUS_FMT_META_12 and the CSI-2 receiver writes the received data to memory
> > > > +as-is.
> > > > +
> > > > +The packing of the data follows the MIPI CSI-2 specification and the padding of
> > > > +the data is defined in the MIPI CCS specification.
> > > > +
> > > > +This format is also used in conjunction with 24 bits per ``Data Unit
> > > > +<media-glossary-data-unit>`` formats that pack two bytes of metadata into one
> > > > +Data Unit.
> > > > +
> > > > +This format is little endian.
> > > > +
> > > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_12.**
> > > > +Each cell is one byte. "M" denotes a byte of metadata and "x" a byte of padding.
> > > > +
> > > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> > > > +
> > > > +.. flat-table:: Sample 4x2 Metadata Frame
> > > > +    :header-rows:  0
> > > > +    :stub-columns: 0
> > > > +    :widths: 12 8 8 8 8 8 8
> > > > +
> > > > +    * - start + 0:
> > > > +      - M\ :sub:`00`
> > > > +      - M\ :sub:`10`
> > > > +      - x
> > > > +      - M\ :sub:`20`
> > > > +      - M\ :sub:`30`
> > > > +      - x
> > > > +    * - start + 6:
> > > > +      - M\ :sub:`01`
> > > > +      - M\ :sub:`11`
> > > > +      - x
> > > > +      - M\ :sub:`21`
> > > > +      - M\ :sub:`31`
> > > > +      - x
> > > > +
> > > > +.. _v4l2-meta-fmt-generic-csi2-14:
> > > > +
> > > > +V4L2_META_FMT_GENERIC_CSI2_14
> > > > +-----------------------------
> > > > +
> > > > +V4L2_META_FMT_GENERIC_CSI2_14 contains 8-bit generic metadata packed in 14-bit
> > > > +Data Units, with three padding bytes after every four bytes of metadata. This
> > > > +format is typically used by CSI-2 receivers with a source that transmits
> > > > +MEDIA_BUS_FMT_META_14 and the CSI-2 receiver writes the received data to memory
> > > > +as-is.
> > > > +
> > > > +The packing of the data follows the MIPI CSI-2 specification and the padding of
> > > > +the data is defined in the MIPI CCS specification.
> > > > +
> > > > +This format is little endian.
> > > > +
> > > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_14.**
> > > > +Each cell is one byte. "M" denotes a byte of metadata and "x" a byte of padding.
> > > > +
> > > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{.8cm}|
> > > > +
> > > > +.. flat-table:: Sample 4x2 Metadata Frame
> > > > +    :header-rows:  0
> > > > +    :stub-columns: 0
> > > > +    :widths: 12 8 8 8 8 8 8 8
> > > > +
> > > > +    * - start + 0:
> > > > +      - M\ :sub:`00`
> > > > +      - M\ :sub:`10`
> > > > +      - M\ :sub:`20`
> > > > +      - M\ :sub:`30`
> > > > +      - x
> > > > +      - x
> > > > +      - x
> > > > +    * - start + 7:
> > > > +      - M\ :sub:`01`
> > > > +      - M\ :sub:`11`
> > > > +      - M\ :sub:`21`
> > > > +      - M\ :sub:`31`
> > > > +      - x
> > > > +      - x
> > > > +      - x
> > > > +
> > > > +.. _v4l2-meta-fmt-generic-csi2-16:
> > > > +
> > > > +V4L2_META_FMT_GENERIC_CSI2_16
> > > > +-----------------------------
> > > > +
> > > > +V4L2_META_FMT_GENERIC_CSI2_16 contains 8-bit generic metadata packed in 16-bit
> > > > +Data Units, with one padding byte after every byte of metadata. This format is
> > > > +typically used by CSI-2 receivers with a source that transmits
> > > > +MEDIA_BUS_FMT_META_16 and the CSI-2 receiver writes the received data to memory
> > > > +as-is.
> > > > +
> > > > +The packing of the data follows the MIPI CSI-2 specification and the padding of
> > > > +the data is defined in the MIPI CCS specification.
> > > 
> > > Here you could also mention that more optimal packing schemes for
> > > MEDIA_BUS_FMT_META_16 use V4L2_META_FMT_GENERIC_8. Same below.
> > 
> > The "more optimal packing scheme" is really a different format. There's no
> > other relation between these, really. I wouldn't mention it here. A driver
> > would enumerate both for metadata when they're supported.
> 
> It's a different format, but the two are linked to the same image
> format. That's why I think it's useful to cross-reference them.

I can add that.

> 
> > > > +
> > > > +This format is little endian.
> > > > +
> > > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_16.**
> > > > +Each cell is one byte. "M" denotes a byte of metadata and "x" a byte of padding.
> > > > +
> > > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|
> > > > +
> > > > +.. flat-table:: Sample 4x2 Metadata Frame
> > > > +    :header-rows:  0
> > > > +    :stub-columns: 0
> > > > +    :widths: 12 8 8 8 8 8 8 8 8
> > > > +
> > > > +    * - start + 0:
> > > > +      - M\ :sub:`00`
> > > > +      - x
> > > > +      - M\ :sub:`10`
> > > > +      - x
> > > > +      - M\ :sub:`20`
> > > > +      - x
> > > > +      - M\ :sub:`30`
> > > > +      - x
> > > > +    * - start + 8:
> > > > +      - M\ :sub:`01`
> > > > +      - x
> > > > +      - M\ :sub:`11`
> > > > +      - x
> > > > +      - M\ :sub:`21`
> > > > +      - x
> > > > +      - M\ :sub:`31`
> > > > +      - x
> > > > +
> > > > +.. _v4l2-meta-fmt-generic-csi2-20:
> > > > +
> > > > +V4L2_META_FMT_GENERIC_CSI2_20
> > > > +-----------------------------
> > > > +
> > > > +V4L2_META_FMT_GENERIC_CSI2_20 contains 8-bit generic metadata packed in 20-bit
> > > > +Data Units, with alternating one or two padding bytes after every byte of
> > > > +metadata. This format is typically used by CSI-2 receivers with a source that
> > > > +transmits MEDIA_BUS_FMT_META_20 and the CSI-2 receiver writes the received data
> > > > +to memory as-is.
> > > > +
> > > > +The packing of the data follows the MIPI CSI-2 specification and the padding of
> > > > +the data is defined in the MIPI CCS specification.
> > > > +
> > > > +This format is little endian.
> > > > +
> > > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_20.**
> > > > +Each cell is one byte. "M" denotes a byte of metadata and "x" a byte of padding.
> > > > +
> > > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> > > > +
> > > > +.. flat-table:: Sample 4x2 Metadata Frame
> > > > +    :header-rows:  0
> > > > +    :stub-columns: 0
> > > > +    :widths: 12 8 8 8 8 8 8 8 8 8 8
> > > > +
> > > > +    * - start + 0:
> > > > +      - M\ :sub:`00`
> > > > +      - x
> > > > +      - M\ :sub:`10`
> > > > +      - x
> > > > +      - x
> > > > +      - M\ :sub:`20`
> > > > +      - x
> > > > +      - M\ :sub:`30`
> > > > +      - x
> > > > +      - x
> > > > +    * - start + 10:
> > > > +      - M\ :sub:`01`
> > > > +      - x
> > > > +      - M\ :sub:`11`
> > > > +      - x
> > > > +      - x
> > > > +      - M\ :sub:`21`
> > > > +      - x
> > > > +      - M\ :sub:`31`
> > > > +      - x
> > > > +      - x
> > > > +
> > > > +.. _v4l2-meta-fmt-generic-csi2-24:
> > > > +
> > > > +V4L2_META_FMT_GENERIC_CSI2_24
> > > > +-----------------------------
> > > > +
> > > > +V4L2_META_FMT_GENERIC_CSI2_24 contains 8-bit generic metadata packed in 24-bit
> > > > +Data Units, with two padding bytes after every byte of metadata. This format is
> > > > +typically used by CSI-2 receivers with a source that transmits
> > > > +MEDIA_BUS_FMT_META_24 and the CSI-2 receiver writes the received data to memory
> > > > +as-is.
> > > > +
> > > > +The packing of the data follows the MIPI CSI-2 specification and the padding of
> > > > +the data is defined in the MIPI CCS specification.
> > > > +
> > > > +This format is little endian.
> > > > +
> > > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_24.**
> > > > +Each cell is one byte. "M" denotes a byte of metadata and "x" a byte of padding.
> > > > +
> > > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> > > > +
> > > > +.. flat-table:: Sample 4x2 Metadata Frame
> > > > +    :header-rows:  0
> > > > +    :stub-columns: 0
> > > > +    :widths: 12 8 8 8 8 8 8 8 8 8 8 8 8
> > > > +
> > > > +    * - start + 0:
> > > > +      - M\ :sub:`00`
> > > > +      - x
> > > > +      - x
> > > > +      - M\ :sub:`10`
> > > > +      - x
> > > > +      - x
> > > > +      - M\ :sub:`20`
> > > > +      - x
> > > > +      - x
> > > > +      - M\ :sub:`30`
> > > > +      - x
> > > > +      - x
> > > > +    * - start + 12:
> > > > +      - M\ :sub:`01`
> > > > +      - x
> > > > +      - x
> > > > +      - M\ :sub:`11`
> > > > +      - x
> > > > +      - x
> > > > +      - M\ :sub:`21`
> > > > +      - x
> > > > +      - x
> > > > +      - M\ :sub:`31`
> > > > +      - x
> > > > +      - x
> > > > diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> > > > index 5aeff5519407..ae2dca7f2817 100644
> > > > --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> > > > +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> > > > @@ -1460,6 +1460,13 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
> > > >  	case V4L2_PIX_FMT_Y210:		descr = "10-bit YUYV Packed"; break;
> > > >  	case V4L2_PIX_FMT_Y212:		descr = "12-bit YUYV Packed"; break;
> > > >  	case V4L2_PIX_FMT_Y216:		descr = "16-bit YUYV Packed"; break;
> > > > +	case V4L2_META_FMT_GENERIC_8:	descr = "8-bit Generic Metadata"; break;
> > > > +	case V4L2_META_FMT_GENERIC_CSI2_10:	descr = "8-bit Generic Meta, 10b CSI-2"; break;
> > > > +	case V4L2_META_FMT_GENERIC_CSI2_12:	descr = "8-bit Generic Meta, 12b CSI-2"; break;
> > > > +	case V4L2_META_FMT_GENERIC_CSI2_14:	descr = "8-bit Generic Meta, 14b CSI-2"; break;
> > > > +	case V4L2_META_FMT_GENERIC_CSI2_16:	descr = "8-bit Generic Meta, 16b CSI-2"; break;
> > > > +	case V4L2_META_FMT_GENERIC_CSI2_20:	descr = "8-bit Generic Meta, 20b CSI-2"; break;
> > > > +	case V4L2_META_FMT_GENERIC_CSI2_24:	descr = "8-bit Generic Meta, 24b CSI-2"; break;
> > > >  
> > > >  	default:
> > > >  		/* Compressed formats */
> > > > diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> > > > index 2663213b76a4..c7cf20b5da67 100644
> > > > --- a/include/uapi/linux/videodev2.h
> > > > +++ b/include/uapi/linux/videodev2.h
> > > > @@ -839,6 +839,14 @@ struct v4l2_pix_format {
> > > >  #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
> > > >  #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
> > > >  
> > > > +#define V4L2_META_FMT_GENERIC_8		v4l2_fourcc('M', 'E', 'T', '8') /* Generic 8-bit metadata */
> > > > +#define V4L2_META_FMT_GENERIC_CSI2_10	v4l2_fourcc('M', 'C', '1', 'A') /* 10-bit CSI-2 packed 8-bit metadata */
> > > > +#define V4L2_META_FMT_GENERIC_CSI2_12	v4l2_fourcc('M', 'C', '1', 'C') /* 12-bit CSI-2 packed 8-bit metadata */
> > > > +#define V4L2_META_FMT_GENERIC_CSI2_14	v4l2_fourcc('M', 'C', '1', 'E') /* 14-bit CSI-2 packed 8-bit metadata */
> > > > +#define V4L2_META_FMT_GENERIC_CSI2_16	v4l2_fourcc('M', 'C', '1', 'G') /* 16-bit CSI-2 packed 8-bit metadata */
> > > > +#define V4L2_META_FMT_GENERIC_CSI2_20	v4l2_fourcc('M', 'C', '1', 'K') /* 20-bit CSI-2 packed 8-bit metadata */
> > > > +#define V4L2_META_FMT_GENERIC_CSI2_24	v4l2_fourcc('M', 'C', '1', 'O') /* 24-bit CSI-2 packed 8-bit metadata */
> > > > +
> > > >  /* priv field value to indicates that subsequent fields are valid. */
> > > >  #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
> > > >  

-- 
Kind regards,

Sakari Ailus

Re: [PATCH v9 11/46] media: Documentation: Document S_ROUTING behaviour

[Sakari Ailus]

Hi Laurent,

On Fri, Apr 19, 2024 at 08:17:20PM +0300, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.
> 
> On Tue, Apr 16, 2024 at 10:32:44PM +0300, Sakari Ailus wrote:
> > Document S_ROUTING behaviour for different devices.
> > 
> > Generally in devices that produce streams the streams are static and some
> > can be enabled and disabled, whereas in devices that just transport them
> > or write them to memory, more configurability is allowed. Document this.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > Reviewed-by: Julien Massot <julien.massot@collabora.com>
> > ---
> >  .../userspace-api/media/v4l/dev-subdev.rst    | 24 +++++++++++++++++++
> >  1 file changed, 24 insertions(+)
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > index d30dcb9e2537..de8dfd4f11a5 100644
> > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > @@ -593,6 +593,30 @@ Any configurations of a stream within a pad, such as format or selections,
> >  are independent of similar configurations on other streams. This is
> >  subject to change in the future.
> >  
> > +Device types and routing setup
> > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > +
> > +Different kinds of sub-devices have differing behaviour for route activation,
> > +depending on the hardware. In all cases, however, only routes that have the
> > +``V4L2_SUBDEV_STREAM_FL_ACTIVE`` flag set are active.
> > +
> > +Devices generating the streams may allow enabling and disabling some of the
> > +routes or the configuration is fixed. If the routes can be disabled, not
> 
> "... some of the routes, or have a fixed routing configuration."

Seems fine.

> 
> > +declaring the routes (or declaring them without
> > +``VIDIOC_SUBDEV_STREAM_FL_ACTIVE`` flag set) in ``VIDIOC_SUBDEV_S_ROUTING`` will
> > +disable the routes while the sub-device driver retains the streams and their
> > +format and selection configuration.
> 
> I still find the "retains their format and selection configuration"
> quite unclear :-S

Alternatively we could say that the routes are simply not active, without
referring to explicitly to formats and selections. I.e.:

If the routes can be disabled, not declaring the routes (or declaring them
without ``VIDIOC_SUBDEV_STREAM_FL_ACTIVE`` flag set) in
``VIDIOC_SUBDEV_S_ROUTING`` will disable the routes.

> 
> > The ``VIDIOC_SUBDEV_S_ROUTING`` will still
> 
> s/will still/ioctl will still/

Both seem to exist, more common is without "ioctl":

$ git grep '[`<]VIDIOC_SUBDEV' -- Documentation/userspace-api/media/|wc -l
84
$ git grep -i "VIDIOC_SUBDEV.*ioctl" -- Documentation/userspace-api/media/|wc -l
34

> 
> > +return such routes back to the user in the routes array, with the
> > +``V4L2_SUBDEV_STREAM_FL_ACTIVE`` flag unset.
> > +
> > +Devices transporting the streams almost always have more configurability with
> > +respect to routing. Typically any route between the sub-device's sink and source
> > +pads is possible, and multiple routes (usually up to certain limited number) may
> > +be active simultaneously. For such devices, no routes are created by the driver
> > +and user-created routes are fully replaced when ``VIDIOC_SUBDEV_S_ROUTING`` is
> > +called on the sub-device. Such newly created routes have the device's default
> > +configuration for format and selection rectangles.
> > +
> >  Configuring streams
> >  ^^^^^^^^^^^^^^^^^^^
> >  
> 

-- 
Regards,

Sakari Ailus

Re: [PATCH v9 10/46] media: Documentation: v4l: Document internal sink pads

[Sakari Ailus]

Hi Laurent,

Thanks for the review!

On Fri, Apr 19, 2024 at 09:49:26PM +0300, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.
> 
> On Tue, Apr 16, 2024 at 10:32:43PM +0300, Sakari Ailus wrote:
> > Document internal sink pads, pads that have both SINK and INTERNAL flags
> > set. Use the IMX219 camera sensor as an example.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > Reviewed-by Julien Massot <julien.massot@collabora.com>
> > ---
> >  .../userspace-api/media/v4l/dev-subdev.rst    | 145 ++++++++++++++++++
> >  1 file changed, 145 insertions(+)
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > index b76e02e54512..d30dcb9e2537 100644
> > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > @@ -553,6 +553,27 @@ A stream at a specific point in the media pipeline is identified by the
> >  sub-device and a (pad, stream) pair. For sub-devices that do not support
> >  multiplexed streams the 'stream' field is always 0.
> >  
> > +.. _v4l2-subdev-internal-source-pads:
> > +
> > +Internal sink pads and routing
> > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > +
> > +Cases where a single sub-device source pad is traversed by multiple streams, one
> > +or more of which originate from within the sub-device itself, are special as
> > +there is no external sink pad for such routes. In those cases, the sources of
> > +the internally generated streams are represented by internal sink pads, which
> > +are sink pads that have the :ref:`MEDIA_PAD_FL_INTERNAL <MEDIA-PAD-FL-INTERNAL>`
> > +pad flag set.
> > +
> > +Internal pads have all the properties of an external pad, including formats and
> > +selections. The format in this case is the source format of the stream. An
> > +internal pad always has a single stream only (0).
> > +
> > +Routes from an internal sink pad to an external source pad are typically not
> > +modifiable but they can be activated and deactivated using the
> > +:ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE <v4l2-subdev-routing-flags>` flag, depending
> > +on driver capabilities.
> > +
> >  Interaction between routes, streams, formats and selections
> >  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> >  
> > @@ -668,3 +689,127 @@ To configure this pipeline, the userspace must take the following steps:
> >     the configurations along the stream towards the receiver, using
> >     :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each
> >     stream endpoint in each sub-device.
> > +
> > +Internal pads setup example
> > +^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > +
> > +A simple example of a multiplexed stream setup might be as follows:
> > +
> > +- An IMX219 camera sensor source sub-device, with one source pad (0), one
> > +  internal sink pad (1) as the source of the image data and an internal sink
> > +  pad (2) as the source of the embedded data. There are two routes, one from the
> > +  internal sink pad 1 to the source 0 (image data) and another from the internal
> 
> s/source 0/source pad 0/

Yes.

> 
> > +  sink pad 2 to the source pad 0 (embedded data). Both streams are always
> > +  active, i.e. there is no need to separately enable the embedded data
> > +  stream. The sensor uses the CSI-2 interface.
> > +
> > +- A CSI-2 receiver in the SoC. The receiver has a single sink pad (pad 0),
> > +  connected to the sensor, and two source pads (pads 1 and 2), to the DMA
> > +  engine. The receiver demultiplexes the incoming streams to the source pads.
> 
> s/engine/engines/
> 
> maybe better, "to two DMA engines".

"to the two DMA engines".

> 
> > +
> > +- DMA engines in the SoC, one for each stream. Each DMA engine is connected to a
> > +  single source pad of the receiver.
> > +
> > +The sensor and the receiver are modelled as V4L2 sub-devices, exposed to
> > +userspace via /dev/v4l-subdevX device nodes. The DMA engines are modelled as
> > +V4L2 devices, exposed to userspace via /dev/videoX nodes.
> > +
> > +To configure this pipeline, the userspace must take the following steps:
> > +
> > +1) Set up media links between entities: enable the links from the sensor to the
> > +   receiver and from the receiver to the DMA engines. This step does not differ
> > +   from normal non-multiplexed media controller setup.
> > +
> > +2) Configure routing
> > +
> > +.. flat-table:: Camera sensor. There are no configurable routes.
> > +    :header-rows: 1
> > +
> > +    * - Sink Pad/Stream
> > +      - Source Pad/Stream
> > +      - Routing Flags
> > +      - Comments
> > +    * - 1/0
> > +      - 0/0
> > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > +      - Pixel data stream from the sink pad
> 
> "from the internal image sink pad"
> 
> > +    * - 2/0
> > +      - 0/1
> > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > +      - Metadata stream from the internal sink pad
> 
> "internal embedded data sink pad"

Yes...

> 
> > +
> > +.. flat-table:: Receiver routing table. Typically both routes need to be
> > +		explicitly set.
> > +    :header-rows:  1
> > +
> > +    * - Sink Pad/Stream
> > +      - Source Pad/Stream
> > +      - Routing Flags
> > +      - Comments
> > +    * - 0/0
> > +      - 1/0
> > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > +      - Pixel data stream from camera sensor
> > +    * - 0/1
> > +      - 2/0
> > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > +      - Metadata stream from camera sensor
> > +
> > +The options available in sensor's routing configuration are dictated by hardware
> 
> s/in sensor/in the sensor/

Agreed.

> 
> > +capabilities: typically camera sensors always produce an image data stream while
> > +it may be possible to enable and disable the embedded data stream.
> 
> Should this go after the first table ?

I'll move it there.

> 
> > +
> > +3) Configure formats and selections
> > +
> > +   This example assumes that the formats are propagated from sink pad to the
> > +   source pad as-is. The tables contain fields of both struct v4l2_subdev_format
> > +   and struct v4l2_mbus_framefmt.
> > +
> > +.. flat-table:: Formats set on the sub-devices. Bold values are set, others are
> > +                static or propagated. The order is aligned with configured
> > +                routes.
> > +    :header-rows: 1
> > +    :fill-cells:
> > +
> > +    * - Sub-device
> > +      - Pad/Stream
> > +      - Width
> > +      - Height
> > +      - Code
> > +    * - :rspan:`3` IMX219
> > +      - 1/0
> > +      - 3296
> > +      - 2480
> > +      - MEDIA_BUS_FMT_SRGGB10
> > +    * - 0/0
> > +      - **3296**
> > +      - **2480**
> > +      - **MEDIA_BUS_FMT_SRGGB10**
> > +    * - 2/0
> > +      - 3296
> > +      - 2
> > +      - MEDIA_BUS_FMT_IMX219_EMBEDDED
> 
> We need a patch in this series to define this format.

I'm waiting for your imx290 patches. :-)

> 
> > +    * - 1/1
> 
> I think this should be 0/1.

Good catch.

> 
> > +      - 3296
> > +      - 2
> > +      - MEDIA_BUS_FMT_META_10
> > +    * - :rspan:`3` CSI-2 receiver
> > +      - 0/0
> > +      - **3296**
> > +      - **2480**
> > +      - **MEDIA_BUS_FMT_SRGGB10**
> > +    * - 1/0
> > +      - 3296
> > +      - 2480
> > +      - MEDIA_BUS_FMT_SRGGB10
> > +    * - 0/1
> > +      - **3296**
> > +      - **2**
> > +      - **MEDIA_BUS_FMT_META_10**
> > +    * - 2/0
> > +      - 3296
> > +      - 2
> > +      - MEDIA_BUS_FMT_META_10
> > +
> > +The embedded data format does not need to be configured as the format is
> 
> Do you mean the "format on the sensor's pads" ? It's a bit confusing if
> you don't specify it.

That's what was meant, I'll add that.

> 
> > +dictated by the pixel data format in this case.
> 

-- 
Regards,

Sakari Ailus

Re: [PATCH v9 15/46] media: v4l: subdev: Add len_routes field to struct v4l2_subdev_routing

[Sakari Ailus]

Hi Laurent,

On Sat, Apr 20, 2024 at 01:45:19AM +0300, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.
> 
> On Tue, Apr 16, 2024 at 10:32:48PM +0300, Sakari Ailus wrote:
> > The len_routes field is used to tell the size of the routes array in
> > struct v4l2_subdev_routing. This way the number of routes returned from
> > S_ROUTING IOCTL may be larger than the number of routes provided, in case
> > there are more routes returned by the driver.
> > 
> > Note that this uAPI is still disabled in the code, so this change can
> > safely be done. Anyone who manually patched the code to enable this uAPI
> > must update their code.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  .../media/v4l/vidioc-subdev-g-routing.rst     | 50 +++++++++++++------
> >  drivers/media/v4l2-core/v4l2-ioctl.c          |  4 +-
> >  drivers/media/v4l2-core/v4l2-subdev.c         | 12 ++---
> >  include/media/v4l2-subdev.h                   |  2 +
> >  include/uapi/linux/v4l2-subdev.h              |  9 ++--
> >  5 files changed, 52 insertions(+), 25 deletions(-)
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > index 26b5004bfe6d..27eb4c82a0e1 100644
> > --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > @@ -43,23 +43,42 @@ The routing configuration determines the flows of data inside an entity.
> >  Drivers report their current routing tables using the
> >  ``VIDIOC_SUBDEV_G_ROUTING`` ioctl and application may enable or disable routes
> >  with the ``VIDIOC_SUBDEV_S_ROUTING`` ioctl, by adding or removing routes and
> > -setting or clearing flags of the  ``flags`` field of a
> > -struct :c:type:`v4l2_subdev_route`.
> > +setting or clearing flags of the ``flags`` field of a struct
> > +:c:type:`v4l2_subdev_route`.
> >  
> > -All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called. This
> > -means that the userspace must reconfigure all streams after calling the ioctl
> > -with e.g. ``VIDIOC_SUBDEV_S_FMT``.
> > +All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called.
> > +This means that the userspace must reconfigure all stream formats and selections
> > +after calling the ioctl with e.g. ``VIDIOC_SUBDEV_S_FMT``.
> >  
> >  Only subdevices which have both sink and source pads can support routing.
> >  
> > -When inspecting routes through ``VIDIOC_SUBDEV_G_ROUTING`` and the application
> > -provided ``num_routes`` is not big enough to contain all the available routes
> > -the subdevice exposes, drivers return the ENOSPC error code and adjust the
> > -value of the ``num_routes`` field. Application should then reserve enough memory
> > -for all the route entries and call ``VIDIOC_SUBDEV_G_ROUTING`` again.
> > -
> > -On a successful ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the
> > -``num_routes`` field to reflect the actual number of routes returned.
> > +The ``len_routes`` field indicates the number of routes that can fit in the
> > +``routes`` array allocated by userspace. It is set by applications for both
> > +ioctls to indicate how many routes the kernel can return, and is never modified
> > +by the kernel.
> > +
> > +The ``num_routes`` field, when returned from the kernel on both IOCTLs,
> > +indicates the number of routes in the subdevice routing table and when calling
> > +``VIDIOC_SUBDEV_S_ROUTING``, it is set by userspace to the number of routes that
> > +the application stored in the ``routes`` array. The value returned by the kernel
> > +may be smaller or larger than the value of ``num_routes`` set by the application
> > +for ``VIDIOC_SUBDEV_S_ROUTING``, as drivers may adjust the requested routing
> > +table.
> 
> I still think the proposal I made when reviewing the previous version is
> clearer :-)
> 
> ----
> The ``num_routes`` field indicates the number of routes in the subdevice routing
> table. For ``VIDIOC_SUBDEV_S_ROUTING``, it is set by userspace to the number of
> routes that the application stored in the ``routes`` array. For both ioctls, it
> is returned by the kernel and indicates how many routes are stored in the
> subdevice routing table. This may be smaller or larger than the value of
> ``num_routes`` set by the application for ``VIDIOC_SUBDEV_S_ROUTING``, as
> drivers may adjust the requested routing table.
> ----
> 
> You replied that
> 
> > For S_ROUTING this is the number of routes in the IOCTL argument. The
> > routing table may contain more (static routes).
> 
> and that's right, but, even when set by userspace for S_ROUTING, the
> num_routes fields is the number of routes that userspace tries to set in
> the routing table. I think starting with a first sentence that describes
> what the field contains, and then explaining how it's used for the
> different ioctls by userspace and kernel space, is clearer.

The problem with your suggestion is that it's not entirely correct:
num_routes is indeed used for two different purposes. Removing " in the
subdevice routing table" in the first sentence would be a simple fix.

> 
> > +
> > +The kernel can return a ``num_routes`` value larger than ``len_routes`` from
> > +both ioctls. This indicates thare are more routes in the routing table than fits
> > +the ``routes`` array. In this case, the ``routes`` array is filled by the kernel
> > +with the first ``len_routes`` entries of the subdevice routing table. This is
> > +not considered to be an error, and the ioctl call succeeds. If the applications
> > +wants to retrieve the missing routes, it can issue a new
> > +``VIDIOC_SUBDEV_G_ROUTING`` call with a large enough ``routes`` array.
> > +
> > +indicate there are more routes than fits to the ``routes`` array. In this
> > +case first ``len_routes`` were returned back to the userspace in the
> > +``routes`` array. This is not considered as an error.
> 
> I think these 3 lines are a leftover.

Yes, I'll remove them.

> 
> > +
> > +Also ``VIDIOC_SUBDEV_S_ROUTING`` may return more route than the user provided in
> 
> s/Also //
> s/route/routes/

Yes.

> 
> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Thanks!

> 
> > +``num_routes`` field due to e.g. hardware properties.
> >  
> >  .. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
> >  
> > @@ -74,6 +93,9 @@ On a successful ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the
> >        - ``which``
> >        - Routing table to be accessed, from enum
> >          :ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
> > +    * - __u32
> > +      - ``len_routes``
> > +      - The length of the array (as in memory reserved for the array)
> >      * - struct :c:type:`v4l2_subdev_route`
> >        - ``routes[]``
> >        - Array of struct :c:type:`v4l2_subdev_route` entries
> > @@ -81,7 +103,7 @@ On a successful ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the
> >        - ``num_routes``
> >        - Number of entries of the routes array
> >      * - __u32
> > -      - ``reserved``\ [5]
> > +      - ``reserved``\ [11]
> >        - Reserved for future extensions. Applications and drivers must set
> >  	the array to zero.
> >  
> > diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> > index 1863ecae9ec9..f30f61c008c7 100644
> > --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> > +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> > @@ -3185,13 +3185,13 @@ static int check_array_args(unsigned int cmd, void *parg, size_t *array_size,
> >  	case VIDIOC_SUBDEV_S_ROUTING: {
> >  		struct v4l2_subdev_routing *routing = parg;
> >  
> > -		if (routing->num_routes > 256)
> > +		if (routing->len_routes > 256)
> >  			return -E2BIG;
> >  
> >  		*user_ptr = u64_to_user_ptr(routing->routes);
> >  		*kernel_ptr = (void **)&routing->routes;
> >  		*array_size = sizeof(struct v4l2_subdev_route)
> > -			    * routing->num_routes;
> > +			    * routing->len_routes;
> >  		ret = 1;
> >  		break;
> >  	}
> > diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> > index 2ba11e5343f0..904378007a79 100644
> > --- a/drivers/media/v4l2-core/v4l2-subdev.c
> > +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> > @@ -927,6 +927,9 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
> >  		if (routing->which != V4L2_SUBDEV_FORMAT_TRY && ro_subdev)
> >  			return -EPERM;
> >  
> > +		if (routing->num_routes > routing->len_routes)
> > +			return -EINVAL;
> > +
> >  		memset(routing->reserved, 0, sizeof(routing->reserved));
> >  
> >  		for (i = 0; i < routing->num_routes; ++i) {
> > @@ -953,6 +956,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
> >  		}
> >  
> >  		krouting.num_routes = routing->num_routes;
> > +		krouting.len_routes = routing->len_routes;
> >  		krouting.routes = routes;
> >  
> >  		return v4l2_subdev_call(sd, pad, set_routing, state,
> > @@ -973,14 +977,10 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
> >  
> >  		krouting = &state->routing;
> >  
> > -		if (routing->num_routes < krouting->num_routes) {
> > -			routing->num_routes = krouting->num_routes;
> > -			return -ENOSPC;
> > -		}
> > -
> >  		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
> >  		       krouting->routes,
> > -		       krouting->num_routes * sizeof(*krouting->routes));
> > +		       min(krouting->num_routes, routing->len_routes) *
> > +		       sizeof(*krouting->routes));
> >  		routing->num_routes = krouting->num_routes;
> >  
> >  		return 0;
> > diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> > index 9cce48365975..1df6b963a1c9 100644
> > --- a/include/media/v4l2-subdev.h
> > +++ b/include/media/v4l2-subdev.h
> > @@ -728,12 +728,14 @@ struct v4l2_subdev_stream_configs {
> >  /**
> >   * struct v4l2_subdev_krouting - subdev routing table
> >   *
> > + * @len_routes: length of routes array, in routes
> >   * @num_routes: number of routes
> >   * @routes: &struct v4l2_subdev_route
> >   *
> >   * This structure contains the routing table for a subdev.
> >   */
> >  struct v4l2_subdev_krouting {
> > +	unsigned int len_routes;
> >  	unsigned int num_routes;
> >  	struct v4l2_subdev_route *routes;
> >  };
> > diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
> > index 81a24bd38003..6a39128d0606 100644
> > --- a/include/uapi/linux/v4l2-subdev.h
> > +++ b/include/uapi/linux/v4l2-subdev.h
> > @@ -228,15 +228,18 @@ struct v4l2_subdev_route {
> >   * struct v4l2_subdev_routing - Subdev routing information
> >   *
> >   * @which: configuration type (from enum v4l2_subdev_format_whence)
> > - * @num_routes: the total number of routes in the routes array
> > + * @len_routes: the length of the routes array, in routes
> >   * @routes: pointer to the routes array
> > + * @num_routes: the total number of routes, possibly more than fits in the
> > + *		routes array
> >   * @reserved: drivers and applications must zero this array
> >   */
> >  struct v4l2_subdev_routing {
> >  	__u32 which;
> > -	__u32 num_routes;
> > +	__u32 len_routes;
> >  	__u64 routes;
> > -	__u32 reserved[6];
> > +	__u32 num_routes;
> > +	__u32 reserved[11];
> >  };
> >  
> >  /*
> 

-- 
Regards,

Sakari Ailus

Re: [PATCH v9 16/46] media: v4l: subdev: Return routes set using S_ROUTING

[Sakari Ailus]

Hi Laurent,

On Sat, Apr 20, 2024 at 01:55:52AM +0300, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.
> 
> On Tue, Apr 16, 2024 at 10:32:49PM +0300, Sakari Ailus wrote:
> > Return the routes set using S_ROUTING back to the user. Also reflect this
> > in documentation.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  .../media/v4l/vidioc-subdev-g-routing.rst            |  7 ++-----
> >  drivers/media/v4l2-core/v4l2-subdev.c                | 12 +++++++++++-
> >  2 files changed, 13 insertions(+), 6 deletions(-)
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > index 27eb4c82a0e1..88df7bf80b00 100644
> > --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > @@ -44,7 +44,8 @@ Drivers report their current routing tables using the
> >  ``VIDIOC_SUBDEV_G_ROUTING`` ioctl and application may enable or disable routes
> >  with the ``VIDIOC_SUBDEV_S_ROUTING`` ioctl, by adding or removing routes and
> >  setting or clearing flags of the ``flags`` field of a struct
> > -:c:type:`v4l2_subdev_route`.
> > +:c:type:`v4l2_subdev_route`. Similarly to ``VIDIOC_SUBDEV_G_ROUTING``, also
> > +``VIDIOC_SUBDEV_S_ROUTING`` returns the routes back to the user.
> 
> :c:type:`v4l2_subdev_route`. Similarly to ``VIDIOC_SUBDEV_G_ROUTING``,
> ``VIDIOC_SUBDEV_S_ROUTING`` also returns the routes back to the user.

The intention here is to say that S_ROUTING returns the routes back to the
user, just as G_ROUTING does. Moving "also" after S_ROUTING makes the
meaning of the sentence unclear.

> 
> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Thanks!

> 
> >  
> >  All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called.
> >  This means that the userspace must reconfigure all stream formats and selections
> > @@ -157,10 +158,6 @@ On success 0 is returned, on error -1 and the ``errno`` variable is set
> >  appropriately. The generic error codes are described at the
> >  :ref:`Generic Error Codes <gen-errors>` chapter.
> >  
> > -ENOSPC
> > -   The application provided ``num_routes`` is not big enough to contain
> > -   all the available routes the subdevice exposes.
> > -
> >  EINVAL
> >     The sink or source pad identifiers reference a non-existing pad or reference
> >     pads of different types (ie. the sink_pad identifiers refers to a source
> > diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> > index 904378007a79..36b2f78cd551 100644
> > --- a/drivers/media/v4l2-core/v4l2-subdev.c
> > +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> > @@ -959,8 +959,18 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
> >  		krouting.len_routes = routing->len_routes;
> >  		krouting.routes = routes;
> >  
> > -		return v4l2_subdev_call(sd, pad, set_routing, state,
> > +		rval = v4l2_subdev_call(sd, pad, set_routing, state,
> >  					routing->which, &krouting);
> > +		if (rval < 0)
> > +			return rval;
> > +
> > +		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
> > +		       state->routing.routes,
> > +		       min(state->routing.num_routes, routing->len_routes) *
> > +		       sizeof(*state->routing.routes));
> > +		routing->num_routes = state->routing.num_routes;
> > +
> > +		return 0;
> >  	}
> >  
> >  	case VIDIOC_SUBDEV_G_ROUTING: {
> 

-- 
Regards,

Sakari Ailus

Re: [PATCH v9 29/46] media: Documentation: ccs: Document routing

[Sakari Ailus]

Hi Laurent,

On Sat, Apr 20, 2024 at 11:31:47AM +0300, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.
> 
> On Tue, Apr 16, 2024 at 10:33:02PM +0300, Sakari Ailus wrote:
> > Document which routes are available for the CCS driver (source) sub-device
> > and what configuration are possible.
> > 
> > Also update copyright.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  .../userspace-api/media/drivers/ccs.rst       | 38 ++++++++++++++++++-
> >  .../media/v4l/subdev-formats.rst              |  2 +
> >  2 files changed, 39 insertions(+), 1 deletion(-)
> > 
> > diff --git a/Documentation/userspace-api/media/drivers/ccs.rst b/Documentation/userspace-api/media/drivers/ccs.rst
> > index 03015b33d5ab..53890ac54dab 100644
> > --- a/Documentation/userspace-api/media/drivers/ccs.rst
> > +++ b/Documentation/userspace-api/media/drivers/ccs.rst
> > @@ -111,4 +111,40 @@ than in the centre.
> >  Shading correction needs to be enabled for luminance correction level to have an
> >  effect.
> >  
> > -**Copyright** |copy| 2020 Intel Corporation
> > +.. _media-ccs-routes:
> > +
> > +Routes
> > +------
> > +
> > +The CCS driver implements one or two :ref:`routes <subdev-routing>` in
> > +its source sub-device (scaler sub-device if exists for the device, otherwise
> 
> s/if exists/if it exists/

Sounds good.

> 
> > +binner) depending on whether the sensor supports embedded data. (All CCS
> > +compliant sensors do but the CCS driver supports preceding standards that did
> > +not require embedded data support, too.)
> > +
> > +The first route of the CCS source sub-device is for pixel data (sink pad
> > +0/stream 0 -> source pad 1/stream 0) and the second one is for embedded data
> > +(internal sink pad 2/stream 0 -> source pad 1/stream 1).
> > +
> > +Embedded data
> > +~~~~~~~~~~~~~
> > +
> > +MIPI CCS supports generation of camera sensor embedded data. The media bus code
> > +used for this format on the internal sink pad is
> > +:ref:`MEDIA_BUS_FMT_CCS_EMBEDDDED <MEDIA-BUS-FMT-CCS-EMBEDDED>`.
> > +
> > +The bit depth of the CCS pixel data affects how the sensor will output the
> > +embedded data, adding padding to align with CSI-2 bus :ref:`Data Units
> > +<media-glossary-data-unit>` for that particular bit depth. This is indicated by
> > +the generic metadata format on the sensor's source sub-device's source pad.
> 
> That's a bit hard to parse, maybe "on the source pad of the sensor's
> source sub-device" ?

Sounds good.

> 
> > +
> > +Devices supporting embedded data for bit depths greater than or equal to 16 may
> > +support more dense packing or legacy single metadata byte per data unit, or both
> > +of these. The supported embedded data formats can be enumerated and configured
> > +on stream 1 of the source pad (1) of the CCS source sub-device.
> > +
> > +The use of the denser packing results in embedded data lines being longer than
> > +the pixel data in data units since the data units are smaller. In bytes the
> > +embedded data lines are still not longer than the image data lines.
> > +
> > +**Copyright** |copy| 2020, 2023 Intel Corporation
> 
> 2024 ?

Yes.

> 
> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Thanks!

> 
> > diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > index fa181ce8f48c..a04756092238 100644
> > --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > @@ -8594,3 +8594,5 @@ levels above.
> >  This mbus code are only used for "2-byte simplified tagged data format" (code
> >  0xa) but their use may be extended further in the future, to cover other CCS
> >  embedded data format codes.
> > +
> > +Also see :ref:`CCS driver documentation <media-ccs-routes>`.
> 

-- 
Regards,

Sakari Ailus

Re: [PATCH v9 16/46] media: v4l: subdev: Return routes set using S_ROUTING

[Laurent Pinchart]

On Tue, Apr 23, 2024 at 10:49:11AM +0000, Sakari Ailus wrote:
> On Sat, Apr 20, 2024 at 01:55:52AM +0300, Laurent Pinchart wrote:
> > On Tue, Apr 16, 2024 at 10:32:49PM +0300, Sakari Ailus wrote:
> > > Return the routes set using S_ROUTING back to the user. Also reflect this
> > > in documentation.
> > > 
> > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > ---
> > >  .../media/v4l/vidioc-subdev-g-routing.rst            |  7 ++-----
> > >  drivers/media/v4l2-core/v4l2-subdev.c                | 12 +++++++++++-
> > >  2 files changed, 13 insertions(+), 6 deletions(-)
> > > 
> > > diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > > index 27eb4c82a0e1..88df7bf80b00 100644
> > > --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > > +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > > @@ -44,7 +44,8 @@ Drivers report their current routing tables using the
> > >  ``VIDIOC_SUBDEV_G_ROUTING`` ioctl and application may enable or disable routes
> > >  with the ``VIDIOC_SUBDEV_S_ROUTING`` ioctl, by adding or removing routes and
> > >  setting or clearing flags of the ``flags`` field of a struct
> > > -:c:type:`v4l2_subdev_route`.
> > > +:c:type:`v4l2_subdev_route`. Similarly to ``VIDIOC_SUBDEV_G_ROUTING``, also
> > > +``VIDIOC_SUBDEV_S_ROUTING`` returns the routes back to the user.
> > 
> > :c:type:`v4l2_subdev_route`. Similarly to ``VIDIOC_SUBDEV_G_ROUTING``,
> > ``VIDIOC_SUBDEV_S_ROUTING`` also returns the routes back to the user.
> 
> The intention here is to say that S_ROUTING returns the routes back to the
> user, just as G_ROUTING does. Moving "also" after S_ROUTING makes the
> meaning of the sentence unclear.

Keeping it before makes it sound non-native English to me :-) But I'm
not a native speaker.

> > Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> 
> Thanks!
> 
> > >  
> > >  All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called.
> > >  This means that the userspace must reconfigure all stream formats and selections
> > > @@ -157,10 +158,6 @@ On success 0 is returned, on error -1 and the ``errno`` variable is set
> > >  appropriately. The generic error codes are described at the
> > >  :ref:`Generic Error Codes <gen-errors>` chapter.
> > >  
> > > -ENOSPC
> > > -   The application provided ``num_routes`` is not big enough to contain
> > > -   all the available routes the subdevice exposes.
> > > -
> > >  EINVAL
> > >     The sink or source pad identifiers reference a non-existing pad or reference
> > >     pads of different types (ie. the sink_pad identifiers refers to a source
> > > diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> > > index 904378007a79..36b2f78cd551 100644
> > > --- a/drivers/media/v4l2-core/v4l2-subdev.c
> > > +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> > > @@ -959,8 +959,18 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
> > >  		krouting.len_routes = routing->len_routes;
> > >  		krouting.routes = routes;
> > >  
> > > -		return v4l2_subdev_call(sd, pad, set_routing, state,
> > > +		rval = v4l2_subdev_call(sd, pad, set_routing, state,
> > >  					routing->which, &krouting);
> > > +		if (rval < 0)
> > > +			return rval;
> > > +
> > > +		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
> > > +		       state->routing.routes,
> > > +		       min(state->routing.num_routes, routing->len_routes) *
> > > +		       sizeof(*state->routing.routes));
> > > +		routing->num_routes = state->routing.num_routes;
> > > +
> > > +		return 0;
> > >  	}
> > >  
> > >  	case VIDIOC_SUBDEV_G_ROUTING: {

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 30/46] media: ccs: Add support for embedded data stream

[Sakari Ailus]

Hi Laurent,

On Sat, Apr 20, 2024 at 11:59:49AM +0300, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.

Thanks for the review!

> 
> On Tue, Apr 16, 2024 at 10:33:03PM +0300, Sakari Ailus wrote:
> > Add support for embedded data stream, in UAPI and frame descriptor.
> > 
> > This patch adds also a new embedded data pad (2) to the source sub-device.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  drivers/media/i2c/ccs/ccs-core.c | 372 +++++++++++++++++++++++++++++--
> >  drivers/media/i2c/ccs/ccs.h      |  18 +-
> >  2 files changed, 362 insertions(+), 28 deletions(-)
> > 
> > diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
> > index 3ca2415fca3b..ba629eafec43 100644
> > --- a/drivers/media/i2c/ccs/ccs-core.c
> > +++ b/drivers/media/i2c/ccs/ccs-core.c
> > @@ -1903,6 +1903,13 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
> >  	if (rval < 0)
> >  		goto err_pm_put;
> >  
> > +	/* Configure embedded data */
> > +	if (sensor->csi_format->compressed >= 16) {
> > +		rval = ccs_write(sensor, EMB_DATA_CTRL, sensor->emb_data_ctrl);
> > +		if (rval < 0)
> > +			goto err_pm_put;
> > +	}
> > +
> >  	if (CCS_LIM(sensor, FLASH_MODE_CAPABILITY) &
> >  	    (CCS_FLASH_MODE_CAPABILITY_SINGLE_STROBE |
> >  	     SMIAPP_FLASH_MODE_CAPABILITY_MULTIPLE_STROBE) &&
> > @@ -2022,6 +2029,57 @@ static const struct ccs_csi_data_format
> >  	return sensor->csi_format;
> >  }
> >  
> > +#define CCS_EMBEDDED_CODE_DEPTH(depth, half_depth)			\
> > +	depth,								\
> > +	CCS_EMB_DATA_CAPABILITY_TWO_BYTES_PER_RAW##depth,		\
> > +	CCS_EMB_DATA_CAPABILITY_NO_ONE_BYTE_PER_RAW##depth,		\
> > +	CCS_EMB_DATA_CTRL_RAW##half_depth##_PACKING_FOR_RAW##depth,	\
> > +	MEDIA_BUS_FMT_META_##half_depth,				\
> > +	MEDIA_BUS_FMT_META_##depth,					\
> > +
> > +static const struct ccs_embedded_code {
> > +	u8 depth;
> > +	u8 cap_two_bytes_per_sample;
> > +	u8 cap_no_legacy;
> > +	u8 ctrl;
> > +	u32 code_two_bytes;
> > +	u32 code_legacy;
> > +} ccs_embedded_codes[] = {
> > +	{ CCS_EMBEDDED_CODE_DEPTH(16, 8) },
> > +	{ CCS_EMBEDDED_CODE_DEPTH(20, 10) },
> > +	{ CCS_EMBEDDED_CODE_DEPTH(24, 12) },
> > +};
> > +
> > +static const struct ccs_embedded_code *ccs_embedded_code(unsigned int bpp)
> > +{
> > +	unsigned int i;
> > +
> > +	for (i = 0; i < ARRAY_SIZE(ccs_embedded_codes); i++)
> > +		if (ccs_embedded_codes[i].depth == bpp)
> > +			return ccs_embedded_codes + i;
> > +
> > +	WARN_ON(1);
> > +
> > +	return ccs_embedded_codes;
> > +}
> > +
> > +static u32
> > +ccs_default_embedded_code(struct ccs_sensor *sensor,
> > +			  const struct ccs_embedded_code *embedded_code)
> > +{
> > +	if (CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
> > +	    BIT(embedded_code->cap_two_bytes_per_sample))
> > +		return embedded_code->code_two_bytes;
> > +
> > +	if (!(CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
> > +	      BIT(embedded_code->cap_no_legacy)))
> > +		return embedded_code->code_legacy;
> > +
> > +	WARN_ON(1);
> > +
> > +	return embedded_code->code_legacy;
> > +}
> > +
> >  static int ccs_enum_mbus_code(struct v4l2_subdev *subdev,
> >  			      struct v4l2_subdev_state *sd_state,
> >  			      struct v4l2_subdev_mbus_code_enum *code)
> > @@ -2037,6 +2095,69 @@ static int ccs_enum_mbus_code(struct v4l2_subdev *subdev,
> >  	dev_err(&client->dev, "subdev %s, pad %u, index %u\n",
> >  		subdev->name, code->pad, code->index);
> >  
> > +	if (subdev == &sensor->src->sd) {
> > +		if (code->pad == CCS_PAD_META) {
> > +			if (code->index)
> > +				goto out;
> > +
> > +			code->code = MEDIA_BUS_FMT_CCS_EMBEDDED;
> > +
> > +			rval = 0;
> > +			goto out;
> > +		}
> > +		if (code->stream == CCS_STREAM_META) {
> > +			struct v4l2_mbus_framefmt *pix_fmt =
> > +				v4l2_subdev_state_get_format(sd_state,
> > +							     CCS_PAD_SRC,
> > +							     CCS_STREAM_PIXEL);
> > +			const struct ccs_csi_data_format *csi_format =
> > +				ccs_validate_csi_data_format(sensor,
> > +							     pix_fmt->code);
> > +			unsigned int i = 0;
> > +			u32 codes[2];
> > +
> > +			switch (csi_format->compressed) {
> > +			case 8:
> > +				codes[i++] = MEDIA_BUS_FMT_META_8;
> > +				break;
> > +			case 10:
> > +				codes[i++] = MEDIA_BUS_FMT_META_10;
> > +				break;
> > +			case 12:
> > +				codes[i++] = MEDIA_BUS_FMT_META_12;
> > +				break;
> > +			case 14:
> > +				codes[i++] = MEDIA_BUS_FMT_META_14;
> > +				break;
> > +			case 16:
> > +			case 20:
> > +			case 24: {
> > +				const struct ccs_embedded_code *embedded_code =
> > +					ccs_embedded_code(csi_format->compressed);
> > +
> > +				if (CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
> > +				    BIT(embedded_code->cap_two_bytes_per_sample))
> > +					codes[i++] =
> > +						embedded_code->code_two_bytes;
> > +
> > +				if (!(CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
> > +				      BIT(embedded_code->cap_no_legacy)))
> > +					codes[i++] = embedded_code->code_legacy;
> > +				break;
> > +			}
> > +			default:
> > +				WARN_ON(1);
> > +			}
> > +
> > +			if (WARN_ON(i > ARRAY_SIZE(codes)) || code->index >= i)
> > +				goto out;
> > +
> > +			code->code = codes[code->index];
> > +			rval = 0;
> > +			goto out;
> > +		}
> > +	}
> > +
> >  	if (subdev != &sensor->src->sd || code->pad != CCS_PAD_SRC) {
> >  		if (code->index)
> >  			goto out;
> > @@ -2079,8 +2200,11 @@ static int __ccs_get_format(struct v4l2_subdev *subdev,
> >  			    struct v4l2_subdev_state *sd_state,
> >  			    struct v4l2_subdev_format *fmt)
> >  {
> > -	fmt->format = *v4l2_subdev_state_get_format(sd_state, fmt->pad);
> > -	fmt->format.code = __ccs_get_mbus_code(subdev, fmt->pad);
> > +	fmt->format = *v4l2_subdev_state_get_format(sd_state, fmt->pad,
> > +						    fmt->stream);
> > +
> > +	if (fmt->pad != CCS_PAD_META && fmt->stream != CCS_STREAM_META)
> > +		fmt->format.code = __ccs_get_mbus_code(subdev, fmt->pad);
> >  
> >  	return 0;
> >  }
> > @@ -2110,10 +2234,11 @@ static void ccs_get_crop_compose(struct v4l2_subdev *subdev,
> >  	if (crops)
> >  		for (i = 0; i < subdev->entity.num_pads; i++)
> >  			crops[i] =
> > -				v4l2_subdev_state_get_crop(sd_state, i);
> > +				v4l2_subdev_state_get_crop(sd_state, i,
> > +							   CCS_STREAM_PIXEL);
> >  	if (comps)
> > -		*comps = v4l2_subdev_state_get_compose(sd_state,
> > -						       ssd->sink_pad);
> > +		*comps = v4l2_subdev_state_get_compose(sd_state, ssd->sink_pad,
> > +						       CCS_STREAM_PIXEL);
> 
> This hunk and the next one could have been moved to the patch that
> introduced CCS_STREAM_PIXEL. Same for the change in __ccs_init_state()
> below.

Sounds good.

> 
> >  }
> >  
> >  /* Changes require propagation only on sink pad. */
> > @@ -2146,7 +2271,8 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
> >  		fallthrough;
> >  	case V4L2_SEL_TGT_COMPOSE:
> >  		*crops[CCS_PAD_SRC] = *comp;
> > -		fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC);
> > +		fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC,
> > +						   CCS_STREAM_PIXEL);
> >  		fmt->width = comp->width;
> >  		fmt->height = comp->height;
> >  		if (which == V4L2_SUBDEV_FORMAT_ACTIVE && ssd == sensor->src)
> > @@ -2210,6 +2336,83 @@ static int ccs_set_format_source(struct v4l2_subdev *subdev,
> >  	return ccs_pll_update(sensor);
> >  }
> >  
> > +static int ccs_set_format_meta(struct v4l2_subdev *subdev,
> > +			       struct v4l2_subdev_state *sd_state,
> > +			       struct v4l2_mbus_framefmt *fmt)
> > +{
> > +	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
> > +	const struct ccs_csi_data_format *csi_format;
> > +	struct v4l2_mbus_framefmt *pix_fmt;
> > +	struct v4l2_mbus_framefmt *meta_fmt;
> > +	struct v4l2_mbus_framefmt *meta_out_fmt;
> > +	u32 code;
> > +
> > +	pix_fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC,
> > +					       CCS_STREAM_PIXEL);
> > +	meta_fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_META, 0);
> > +	meta_out_fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC,
> > +						    CCS_STREAM_META);
> > +
> > +	code = fmt ? fmt->code : meta_out_fmt->code;
> 
> When this function is called from __ccs_init_state(), fmt will be NULL,
> and meta_out_fmt will be uninitialized. Is this intended ?

Not uninitialised but zero. I'll use zero explicitly instead.

> 
> > +
> > +	meta_out_fmt->width = meta_fmt->width = pix_fmt->width;
> > +	meta_out_fmt->height = meta_fmt->height =
> > +		sensor->embedded_end - sensor->embedded_start;
> > +	meta_fmt->code = MEDIA_BUS_FMT_CCS_EMBEDDED;
> > +
> > +	csi_format = ccs_validate_csi_data_format(sensor, pix_fmt->code);
> > +
> > +	switch (csi_format->compressed) {
> > +	case 8:
> > +		meta_out_fmt->code = MEDIA_BUS_FMT_META_8;
> > +		break;
> > +	case 10:
> > +		meta_out_fmt->code = MEDIA_BUS_FMT_META_10;
> > +		break;
> > +	case 12:
> > +		meta_out_fmt->code = MEDIA_BUS_FMT_META_12;
> > +		break;
> > +	case 14:
> > +		meta_out_fmt->code = MEDIA_BUS_FMT_META_14;
> > +		break;
> > +	case 16:
> > +	case 20:
> > +	case 24: {
> > +		const struct ccs_embedded_code *embedded_code;
> > +
> > +		embedded_code = ccs_embedded_code(csi_format->compressed);
> > +		meta_out_fmt->code =
> > +			ccs_default_embedded_code(sensor, embedded_code);
> > +
> > +		if (!(CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
> > +		      BIT(embedded_code->cap_no_legacy)) &&
> > +		    code == embedded_code->code_legacy) {
> > +			meta_out_fmt->code = embedded_code->code_legacy;
> > +			sensor->emb_data_ctrl = 0;
> > +		}
> > +
> > +		if (CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
> > +		    BIT(embedded_code->cap_two_bytes_per_sample) &&
> > +		    code == embedded_code->code_two_bytes) {
> > +			meta_out_fmt->code = embedded_code->code_two_bytes;
> > +			sensor->emb_data_ctrl = embedded_code->ctrl;
> > +			meta_fmt->width <<= 1;
> > +			meta_out_fmt->width <<= 1;
> > +		}
> > +
> > +		break;
> > +	}
> > +	default:
> > +		WARN_ON(1);
> > +		return 0;
> > +	}
> > +
> > +	if (fmt)
> > +		*fmt = *meta_out_fmt;
> > +
> > +	return 0;
> > +}
> > +
> >  static int ccs_set_format(struct v4l2_subdev *subdev,
> >  			  struct v4l2_subdev_state *sd_state,
> >  			  struct v4l2_subdev_format *fmt)
> > @@ -2218,12 +2421,25 @@ static int ccs_set_format(struct v4l2_subdev *subdev,
> >  	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
> >  	struct v4l2_rect *crops[CCS_PADS];
> >  
> > +	if (subdev == &sensor->src->sd && fmt->pad == CCS_PAD_META)
> 
> You could also write
> 
> 	if (ssd == sensor->src && fmt->pad == CCS_PAD_META)

Sounds good.

> 
> Same below.
> 
> > +		return ccs_get_format(subdev, sd_state, fmt);
> > +
> >  	mutex_lock(&sensor->mutex);
> 
> Is this needed, shouldn't the state lock be enough ?

Not while the access to the device's state is serialised using the driver's
own mutex. This changes two patches later though.

> 
> >  
> > +	if (subdev == &sensor->src->sd && fmt->stream == CCS_STREAM_META) {
> > +		ccs_set_format_meta(subdev, sd_state, &fmt->format);
> > +
> > +		mutex_unlock(&sensor->mutex);
> > +
> > +		return 0;
> > +	}
> > +
> >  	if (fmt->pad == ssd->source_pad) {
> >  		int rval;
> >  
> >  		rval = ccs_set_format_source(subdev, sd_state, fmt);
> > +		if (sensor->embedded_start != sensor->embedded_end)
> 
> A ccs_sensor_has_embedded_data() (name bikeshedding allowed) inline
> helper could be nice to replace this manual check could be nice, as you
> do the same in many locations below.

Sounds good.

> 
> > +			ccs_set_format_meta(subdev, sd_state, NULL);
> 
> This doesn't seem correct, you shouldn't set the metadata format on
> subdevs that are not the source subdev.

Good point. I'll add a check.

> 
> A comment to explain how the metadata format is propagated would also be
> useful.

I'll add this to the documentation patch which actually could be better
after this patch, not before.

> 
> >  
> >  		mutex_unlock(&sensor->mutex);
> >  
> > @@ -2498,6 +2714,12 @@ static int ccs_sel_supported(struct v4l2_subdev *subdev,
> >  	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
> >  	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
> >  
> > +	if (sel->stream != CCS_STREAM_PIXEL)
> > +		return -EINVAL;
> > +
> > +	if (sel->pad == CCS_PAD_META)
> > +		return -EINVAL;
> > +
> >  	/* We only implement crop in three places. */
> >  	switch (sel->target) {
> >  	case V4L2_SEL_TGT_CROP:
> > @@ -2542,7 +2764,8 @@ static int ccs_set_crop(struct v4l2_subdev *subdev,
> >  
> >  	if (sel->pad == ssd->sink_pad) {
> >  		struct v4l2_mbus_framefmt *mfmt =
> > -			v4l2_subdev_state_get_format(sd_state, sel->pad);
> > +			v4l2_subdev_state_get_format(sd_state, sel->pad,
> > +						     CCS_STREAM_PIXEL);
> >  
> >  		src_size.width = mfmt->width;
> >  		src_size.height = mfmt->height;
> > @@ -2603,7 +2826,9 @@ static int ccs_get_selection(struct v4l2_subdev *subdev,
> >  		} else if (sel->pad == ssd->sink_pad) {
> >  			struct v4l2_mbus_framefmt *sink_fmt =
> >  				v4l2_subdev_state_get_format(sd_state,
> > -							     ssd->sink_pad);
> > +							     ssd->sink_pad,
> > +							     CCS_STREAM_PIXEL);
> > +
> >  			sel->r.top = sel->r.left = 0;
> >  			sel->r.width = sink_fmt->width;
> >  			sel->r.height = sink_fmt->height;
> > @@ -2686,6 +2911,14 @@ static int ccs_get_frame_desc(struct v4l2_subdev *subdev, unsigned int pad,
> >  	entry++;
> >  	desc->num_entries++;
> >  
> > +	if (sensor->embedded_start != sensor->embedded_end) {
> > +		entry->pixelcode = MEDIA_BUS_FMT_CCS_EMBEDDED;
> 
> I think you need to report the generic pixel code here.

I'll fix that for v10.

> 
> > +		entry->stream = CCS_STREAM_META;
> > +		entry->bus.csi2.dt = MIPI_CSI2_DT_EMBEDDED_8B;
> > +		entry++;
> > +		desc->num_entries++;
> > +	}
> > +
> >  	return 0;
> >  }
> >  
> > @@ -3004,6 +3237,8 @@ static void ccs_cleanup(struct ccs_sensor *sensor)
> >  	ccs_free_controls(sensor);
> >  }
> >  
> > +static const struct v4l2_subdev_internal_ops ccs_internal_ops;
> > +
> >  static int ccs_init_subdev(struct ccs_sensor *sensor,
> >  			   struct ccs_subdev *ssd, const char *name,
> >  			   unsigned short num_pads, u32 function,
> > @@ -3016,15 +3251,18 @@ static int ccs_init_subdev(struct ccs_sensor *sensor,
> >  	if (!ssd)
> >  		return 0;
> >  
> > -	if (ssd != sensor->src)
> > +	if (ssd != sensor->src) {
> >  		v4l2_subdev_init(&ssd->sd, &ccs_ops);
> > +		ssd->sd.internal_ops = &ccs_internal_ops;
> > +	}
> >  
> >  	ssd->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
> >  	ssd->sd.entity.function = function;
> >  	ssd->sensor = sensor;
> >  
> >  	ssd->npads = num_pads;
> > -	ssd->source_pad = num_pads - 1;
> > +	ssd->source_pad =
> > +		ssd == sensor->pixel_array ? CCS_PA_PAD_SRC : CCS_PAD_SRC;
> >  
> >  	v4l2_i2c_subdev_set_name(&ssd->sd, client, sensor->minfo.name, name);
> >  
> > @@ -3038,6 +3276,10 @@ static int ccs_init_subdev(struct ccs_sensor *sensor,
> >  		ssd->sd.owner = THIS_MODULE;
> >  		ssd->sd.dev = &client->dev;
> >  		v4l2_set_subdevdata(&ssd->sd, client);
> > +	} else {
> > +		ssd->sd.flags |= V4L2_SUBDEV_FL_STREAMS;
> > +		ssd->pads[CCS_PAD_META].flags =
> > +			MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL;
> >  	}
> >  
> >  	rval = media_entity_pads_init(&ssd->sd.entity, ssd->npads, ssd->pads);
> > @@ -3055,21 +3297,19 @@ static int ccs_init_subdev(struct ccs_sensor *sensor,
> >  	return 0;
> >  }
> >  
> > -static int ccs_init_state(struct v4l2_subdev *sd,
> > -			  struct v4l2_subdev_state *sd_state)
> > +static int __ccs_init_state(struct v4l2_subdev *sd,
> > +			    struct v4l2_subdev_state *sd_state)
> >  {
> >  	struct ccs_subdev *ssd = to_ccs_subdev(sd);
> >  	struct ccs_sensor *sensor = ssd->sensor;
> >  	unsigned int pad = ssd == sensor->pixel_array ?
> >  		CCS_PA_PAD_SRC : CCS_PAD_SINK;
> >  	struct v4l2_mbus_framefmt *fmt =
> > -		v4l2_subdev_state_get_format(sd_state, pad);
> > +		v4l2_subdev_state_get_format(sd_state, pad, CCS_STREAM_PIXEL);
> >  	struct v4l2_rect *crop =
> > -		v4l2_subdev_state_get_crop(sd_state, pad);
> > +		v4l2_subdev_state_get_crop(sd_state, pad, CCS_STREAM_PIXEL);
> >  	bool is_active = !sd->active_state || sd->active_state == sd_state;
> >  
> > -	mutex_lock(&sensor->mutex);
> > -
> >  	ccs_get_native_size(ssd, crop);
> >  
> >  	fmt->width = crop->width;
> > @@ -3081,20 +3321,78 @@ static int ccs_init_state(struct v4l2_subdev *sd,
> >  		if (is_active)
> >  			sensor->pa_src = *crop;
> >  
> > -		mutex_unlock(&sensor->mutex);
> >  		return 0;
> >  	}
> >  
> > -	fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC);
> > +	fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC,
> > +					   CCS_STREAM_PIXEL);
> >  	fmt->code = ssd == sensor->src ?
> >  		sensor->csi_format->code : sensor->internal_csi_format->code;
> >  	fmt->field = V4L2_FIELD_NONE;
> >  
> >  	ccs_propagate(sd, sd_state, is_active, V4L2_SEL_TGT_CROP);
> >  
> > +	return 0;
> > +}
> > +
> > +static int ccs_init_state(struct v4l2_subdev *sd,
> > +			  struct v4l2_subdev_state *sd_state)
> > +{
> > +	struct ccs_subdev *ssd = to_ccs_subdev(sd);
> > +	struct ccs_sensor *sensor = ssd->sensor;
> > +	int rval;
> > +
> > +	mutex_lock(&sensor->mutex);
> > +	rval = __ccs_init_state(sd, sd_state);
> >  	mutex_unlock(&sensor->mutex);
> >  
> > -	return 0;
> > +	return rval;
> > +}
> > +
> > +static int ccs_src_init_state(struct v4l2_subdev *sd,
> > +			      struct v4l2_subdev_state *sd_state)
> > +{
> > +	struct v4l2_subdev_route routes[] = {
> > +		{
> > +			.sink_pad = CCS_PAD_SINK,
> > +			.source_pad = CCS_PAD_SRC,
> > +			.source_stream = CCS_STREAM_PIXEL,
> > +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> > +		}, {
> > +			.sink_pad = CCS_PAD_META,
> > +			.source_pad = CCS_PAD_SRC,
> > +			.source_stream = CCS_STREAM_META,
> > +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> > +		}
> > +	};
> > +	struct v4l2_subdev_krouting routing = {
> > +		.routes = routes,
> > +		.num_routes = 1,
> > +	};
> > +	struct ccs_subdev *ssd = to_ccs_subdev(sd);
> > +	struct ccs_sensor *sensor = ssd->sensor;
> > +	int rval;
> > +
> > +	mutex_lock(&sensor->mutex);
> 
> Is this needed, shouldn't the state lock be enough ?

Same as for the other mutex: this will no longer be needed after relying on
the state locks for locking, two patches later.

> 
> > +
> > +	if (sensor->embedded_start != sensor->embedded_end)
> > +		routing.num_routes++;
> > +
> > +	rval = v4l2_subdev_set_routing(sd, sd_state, &routing);
> > +	if (rval)
> > +		goto out;
> > +
> > +	rval = __ccs_init_state(sd, sd_state);
> > +	if (rval)
> > +		goto out;
> > +
> > +	if (sensor->embedded_start != sensor->embedded_end)
> > +		ccs_set_format_meta(sd, sd_state, NULL);
> > +
> > +out:
> > +	mutex_unlock(&sensor->mutex);
> > +
> > +	return rval;
> >  }
> >  
> >  static const struct v4l2_subdev_video_ops ccs_video_ops = {
> > @@ -3109,6 +3407,14 @@ static const struct v4l2_subdev_pad_ops ccs_pad_ops = {
> >  	.set_fmt = ccs_set_format,
> >  	.get_selection = ccs_get_selection,
> >  	.set_selection = ccs_set_selection,
> > +};
> > +
> > +static const struct v4l2_subdev_pad_ops ccs_src_pad_ops = {
> > +	.enum_mbus_code = ccs_enum_mbus_code,
> > +	.get_fmt = ccs_get_format,
> 
> I'm surprised you need to implement .get_fmt(). The
> v4l2_subdev_get_fmt() helper should have been enough.

It should be possible to get rid of that now, too. I'll add a new patch for
this.

> 
> > +	.set_fmt = ccs_set_format,
> > +	.get_selection = ccs_get_selection,
> > +	.set_selection = ccs_set_selection,
> >  	.enable_streams = ccs_enable_streams,
> >  	.disable_streams = ccs_disable_streams,
> >  	.get_frame_desc = ccs_get_frame_desc,
> > @@ -3125,12 +3431,22 @@ static const struct v4l2_subdev_ops ccs_ops = {
> >  	.sensor = &ccs_sensor_ops,
> >  };
> >  
> > +static const struct v4l2_subdev_ops ccs_src_ops = {
> > +	.video = &ccs_video_ops,
> > +	.pad = &ccs_src_pad_ops,
> > +	.sensor = &ccs_sensor_ops,
> > +};
> > +
> >  static const struct media_entity_operations ccs_entity_ops = {
> >  	.link_validate = v4l2_subdev_link_validate,
> >  };
> >  
> > -static const struct v4l2_subdev_internal_ops ccs_internal_src_ops = {
> > +static const struct v4l2_subdev_internal_ops ccs_internal_ops = {
> >  	.init_state = ccs_init_state,
> > +};
> > +
> > +static const struct v4l2_subdev_internal_ops ccs_src_internal_ops = {
> > +	.init_state = ccs_src_init_state,
> >  	.registered = ccs_registered,
> >  	.unregistered = ccs_unregistered,
> >  };
> > @@ -3281,8 +3597,8 @@ static int ccs_probe(struct i2c_client *client)
> >  
> >  	sensor->src = &sensor->ssds[sensor->ssds_used];
> >  
> > -	v4l2_i2c_subdev_init(&sensor->src->sd, client, &ccs_ops);
> > -	sensor->src->sd.internal_ops = &ccs_internal_src_ops;
> > +	v4l2_i2c_subdev_init(&sensor->src->sd, client, &ccs_src_ops);
> > +	sensor->src->sd.internal_ops = &ccs_src_internal_ops;
> >  
> >  	sensor->regulators = devm_kcalloc(&client->dev,
> >  					  ARRAY_SIZE(ccs_regulators),
> > @@ -3553,12 +3869,20 @@ static int ccs_probe(struct i2c_client *client)
> >  		goto out_cleanup;
> >  	}
> >  
> > -	rval = ccs_init_subdev(sensor, sensor->scaler, " scaler", 2,
> > +	rval = ccs_init_subdev(sensor, sensor->scaler, " scaler",
> > +			       sensor->ssds_used != CCS_SUBDEVS ?
> > +			       CCS_PADS_NOMETA :
> > +			       sensor->embedded_start == sensor->embedded_end ?
> > +			       CCS_PADS_NOMETA : CCS_PADS,
> >  			       MEDIA_ENT_F_PROC_VIDEO_SCALER,
> >  			       "ccs scaler mutex", &scaler_lock_key);
> >  	if (rval)
> >  		goto out_cleanup;
> > -	rval = ccs_init_subdev(sensor, sensor->binner, " binner", 2,
> > +	rval = ccs_init_subdev(sensor, sensor->binner, " binner",
> > +			       sensor->ssds_used == CCS_SUBDEVS ?
> > +			       CCS_PADS_NOMETA :
> > +			       sensor->embedded_start == sensor->embedded_end ?
> > +			       CCS_PADS_NOMETA : CCS_PADS,
> >  			       MEDIA_ENT_F_PROC_VIDEO_SCALER,
> >  			       "ccs binner mutex", &binner_lock_key);
> >  	if (rval)
> > diff --git a/drivers/media/i2c/ccs/ccs.h b/drivers/media/i2c/ccs/ccs.h
> > index 90b442a3d53e..477b2fb99aa0 100644
> > --- a/drivers/media/i2c/ccs/ccs.h
> > +++ b/drivers/media/i2c/ccs/ccs.h
> > @@ -173,11 +173,18 @@ struct ccs_csi_data_format {
> >  #define CCS_SUBDEVS			3
> >  
> >  #define CCS_PA_PAD_SRC			0
> > -#define CCS_PAD_SINK			0
> > -#define CCS_PAD_SRC			1
> > -#define CCS_PADS			2
> > +enum {
> > +	CCS_PAD_SINK,
> > +	CCS_PAD_SRC,
> > +	CCS_PAD_META,
> > +	CCS_PADS_NOMETA = CCS_PAD_META,
> 
> This doesn't seem to improve readability of the code above :-S

The number of pads is different on sub-devices, with the introduction of
one more internal pad in the source. I don't think another enum for the
purpose would be better as the same code is dealing with all of the
driver's sub-devices.

> 
> > +	CCS_PADS,
> > +};
> >  
> > -#define CCS_STREAM_PIXEL		0
> > +enum {
> > +	CCS_STREAM_PIXEL,
> > +	CCS_STREAM_META,
> > +};
> >  
> >  struct ccs_binning_subtype {
> >  	u8 horizontal:4;
> > @@ -228,6 +235,9 @@ struct ccs_sensor {
> >  	int default_pixel_order;
> >  	struct ccs_data_container sdata, mdata;
> >  
> > +	u32 embedded_mbus_code;
> 
> Not used.

I'll remove it.

> 
> > +	u8 emb_data_ctrl;
> 
> The general direction I'd like to take with v4l2_subdev_state is to
> avoid storing state information in the device private structure. Could
> this be dropped and computed in ccs_enable_streams() instead ?

Deriving this information from mbus code is possible. I'll add a new patch
for this.

> 
> > +
> >  	u8 binning_horizontal;
> >  	u8 binning_vertical;
> >  
> 

-- 
Regards,

Sakari Ailus

Re: [PATCH v9 30/46] media: ccs: Add support for embedded data stream

[Laurent Pinchart]

Hi Sakari,

On Tue, Apr 23, 2024 at 12:33:23PM +0000, Sakari Ailus wrote:
> On Sat, Apr 20, 2024 at 11:59:49AM +0300, Laurent Pinchart wrote:
> > On Tue, Apr 16, 2024 at 10:33:03PM +0300, Sakari Ailus wrote:
> > > Add support for embedded data stream, in UAPI and frame descriptor.
> > > 
> > > This patch adds also a new embedded data pad (2) to the source sub-device.
> > > 
> > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > ---
> > >  drivers/media/i2c/ccs/ccs-core.c | 372 +++++++++++++++++++++++++++++--
> > >  drivers/media/i2c/ccs/ccs.h      |  18 +-
> > >  2 files changed, 362 insertions(+), 28 deletions(-)
> > > 
> > > diff --git a/drivers/media/i2c/ccs/ccs-core.c b/drivers/media/i2c/ccs/ccs-core.c
> > > index 3ca2415fca3b..ba629eafec43 100644
> > > --- a/drivers/media/i2c/ccs/ccs-core.c
> > > +++ b/drivers/media/i2c/ccs/ccs-core.c
> > > @@ -1903,6 +1903,13 @@ static int ccs_enable_streams(struct v4l2_subdev *subdev,
> > >  	if (rval < 0)
> > >  		goto err_pm_put;
> > >  
> > > +	/* Configure embedded data */
> > > +	if (sensor->csi_format->compressed >= 16) {
> > > +		rval = ccs_write(sensor, EMB_DATA_CTRL, sensor->emb_data_ctrl);
> > > +		if (rval < 0)
> > > +			goto err_pm_put;
> > > +	}
> > > +
> > >  	if (CCS_LIM(sensor, FLASH_MODE_CAPABILITY) &
> > >  	    (CCS_FLASH_MODE_CAPABILITY_SINGLE_STROBE |
> > >  	     SMIAPP_FLASH_MODE_CAPABILITY_MULTIPLE_STROBE) &&
> > > @@ -2022,6 +2029,57 @@ static const struct ccs_csi_data_format
> > >  	return sensor->csi_format;
> > >  }
> > >  
> > > +#define CCS_EMBEDDED_CODE_DEPTH(depth, half_depth)			\
> > > +	depth,								\
> > > +	CCS_EMB_DATA_CAPABILITY_TWO_BYTES_PER_RAW##depth,		\
> > > +	CCS_EMB_DATA_CAPABILITY_NO_ONE_BYTE_PER_RAW##depth,		\
> > > +	CCS_EMB_DATA_CTRL_RAW##half_depth##_PACKING_FOR_RAW##depth,	\
> > > +	MEDIA_BUS_FMT_META_##half_depth,				\
> > > +	MEDIA_BUS_FMT_META_##depth,					\
> > > +
> > > +static const struct ccs_embedded_code {
> > > +	u8 depth;
> > > +	u8 cap_two_bytes_per_sample;
> > > +	u8 cap_no_legacy;
> > > +	u8 ctrl;
> > > +	u32 code_two_bytes;
> > > +	u32 code_legacy;
> > > +} ccs_embedded_codes[] = {
> > > +	{ CCS_EMBEDDED_CODE_DEPTH(16, 8) },
> > > +	{ CCS_EMBEDDED_CODE_DEPTH(20, 10) },
> > > +	{ CCS_EMBEDDED_CODE_DEPTH(24, 12) },
> > > +};
> > > +
> > > +static const struct ccs_embedded_code *ccs_embedded_code(unsigned int bpp)
> > > +{
> > > +	unsigned int i;
> > > +
> > > +	for (i = 0; i < ARRAY_SIZE(ccs_embedded_codes); i++)
> > > +		if (ccs_embedded_codes[i].depth == bpp)
> > > +			return ccs_embedded_codes + i;
> > > +
> > > +	WARN_ON(1);
> > > +
> > > +	return ccs_embedded_codes;
> > > +}
> > > +
> > > +static u32
> > > +ccs_default_embedded_code(struct ccs_sensor *sensor,
> > > +			  const struct ccs_embedded_code *embedded_code)
> > > +{
> > > +	if (CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
> > > +	    BIT(embedded_code->cap_two_bytes_per_sample))
> > > +		return embedded_code->code_two_bytes;
> > > +
> > > +	if (!(CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
> > > +	      BIT(embedded_code->cap_no_legacy)))
> > > +		return embedded_code->code_legacy;
> > > +
> > > +	WARN_ON(1);
> > > +
> > > +	return embedded_code->code_legacy;
> > > +}
> > > +
> > >  static int ccs_enum_mbus_code(struct v4l2_subdev *subdev,
> > >  			      struct v4l2_subdev_state *sd_state,
> > >  			      struct v4l2_subdev_mbus_code_enum *code)
> > > @@ -2037,6 +2095,69 @@ static int ccs_enum_mbus_code(struct v4l2_subdev *subdev,
> > >  	dev_err(&client->dev, "subdev %s, pad %u, index %u\n",
> > >  		subdev->name, code->pad, code->index);
> > >  
> > > +	if (subdev == &sensor->src->sd) {
> > > +		if (code->pad == CCS_PAD_META) {
> > > +			if (code->index)
> > > +				goto out;
> > > +
> > > +			code->code = MEDIA_BUS_FMT_CCS_EMBEDDED;
> > > +
> > > +			rval = 0;
> > > +			goto out;
> > > +		}
> > > +		if (code->stream == CCS_STREAM_META) {
> > > +			struct v4l2_mbus_framefmt *pix_fmt =
> > > +				v4l2_subdev_state_get_format(sd_state,
> > > +							     CCS_PAD_SRC,
> > > +							     CCS_STREAM_PIXEL);
> > > +			const struct ccs_csi_data_format *csi_format =
> > > +				ccs_validate_csi_data_format(sensor,
> > > +							     pix_fmt->code);
> > > +			unsigned int i = 0;
> > > +			u32 codes[2];
> > > +
> > > +			switch (csi_format->compressed) {
> > > +			case 8:
> > > +				codes[i++] = MEDIA_BUS_FMT_META_8;
> > > +				break;
> > > +			case 10:
> > > +				codes[i++] = MEDIA_BUS_FMT_META_10;
> > > +				break;
> > > +			case 12:
> > > +				codes[i++] = MEDIA_BUS_FMT_META_12;
> > > +				break;
> > > +			case 14:
> > > +				codes[i++] = MEDIA_BUS_FMT_META_14;
> > > +				break;
> > > +			case 16:
> > > +			case 20:
> > > +			case 24: {
> > > +				const struct ccs_embedded_code *embedded_code =
> > > +					ccs_embedded_code(csi_format->compressed);
> > > +
> > > +				if (CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
> > > +				    BIT(embedded_code->cap_two_bytes_per_sample))
> > > +					codes[i++] =
> > > +						embedded_code->code_two_bytes;
> > > +
> > > +				if (!(CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
> > > +				      BIT(embedded_code->cap_no_legacy)))
> > > +					codes[i++] = embedded_code->code_legacy;
> > > +				break;
> > > +			}
> > > +			default:
> > > +				WARN_ON(1);
> > > +			}
> > > +
> > > +			if (WARN_ON(i > ARRAY_SIZE(codes)) || code->index >= i)
> > > +				goto out;
> > > +
> > > +			code->code = codes[code->index];
> > > +			rval = 0;
> > > +			goto out;
> > > +		}
> > > +	}
> > > +
> > >  	if (subdev != &sensor->src->sd || code->pad != CCS_PAD_SRC) {
> > >  		if (code->index)
> > >  			goto out;
> > > @@ -2079,8 +2200,11 @@ static int __ccs_get_format(struct v4l2_subdev *subdev,
> > >  			    struct v4l2_subdev_state *sd_state,
> > >  			    struct v4l2_subdev_format *fmt)
> > >  {
> > > -	fmt->format = *v4l2_subdev_state_get_format(sd_state, fmt->pad);
> > > -	fmt->format.code = __ccs_get_mbus_code(subdev, fmt->pad);
> > > +	fmt->format = *v4l2_subdev_state_get_format(sd_state, fmt->pad,
> > > +						    fmt->stream);
> > > +
> > > +	if (fmt->pad != CCS_PAD_META && fmt->stream != CCS_STREAM_META)
> > > +		fmt->format.code = __ccs_get_mbus_code(subdev, fmt->pad);
> > >  
> > >  	return 0;
> > >  }
> > > @@ -2110,10 +2234,11 @@ static void ccs_get_crop_compose(struct v4l2_subdev *subdev,
> > >  	if (crops)
> > >  		for (i = 0; i < subdev->entity.num_pads; i++)
> > >  			crops[i] =
> > > -				v4l2_subdev_state_get_crop(sd_state, i);
> > > +				v4l2_subdev_state_get_crop(sd_state, i,
> > > +							   CCS_STREAM_PIXEL);
> > >  	if (comps)
> > > -		*comps = v4l2_subdev_state_get_compose(sd_state,
> > > -						       ssd->sink_pad);
> > > +		*comps = v4l2_subdev_state_get_compose(sd_state, ssd->sink_pad,
> > > +						       CCS_STREAM_PIXEL);
> > 
> > This hunk and the next one could have been moved to the patch that
> > introduced CCS_STREAM_PIXEL. Same for the change in __ccs_init_state()
> > below.
> 
> Sounds good.
> 
> > >  }
> > >  
> > >  /* Changes require propagation only on sink pad. */
> > > @@ -2146,7 +2271,8 @@ static void ccs_propagate(struct v4l2_subdev *subdev,
> > >  		fallthrough;
> > >  	case V4L2_SEL_TGT_COMPOSE:
> > >  		*crops[CCS_PAD_SRC] = *comp;
> > > -		fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC);
> > > +		fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC,
> > > +						   CCS_STREAM_PIXEL);
> > >  		fmt->width = comp->width;
> > >  		fmt->height = comp->height;
> > >  		if (which == V4L2_SUBDEV_FORMAT_ACTIVE && ssd == sensor->src)
> > > @@ -2210,6 +2336,83 @@ static int ccs_set_format_source(struct v4l2_subdev *subdev,
> > >  	return ccs_pll_update(sensor);
> > >  }
> > >  
> > > +static int ccs_set_format_meta(struct v4l2_subdev *subdev,
> > > +			       struct v4l2_subdev_state *sd_state,
> > > +			       struct v4l2_mbus_framefmt *fmt)
> > > +{
> > > +	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
> > > +	const struct ccs_csi_data_format *csi_format;
> > > +	struct v4l2_mbus_framefmt *pix_fmt;
> > > +	struct v4l2_mbus_framefmt *meta_fmt;
> > > +	struct v4l2_mbus_framefmt *meta_out_fmt;
> > > +	u32 code;
> > > +
> > > +	pix_fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC,
> > > +					       CCS_STREAM_PIXEL);
> > > +	meta_fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_META, 0);
> > > +	meta_out_fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC,
> > > +						    CCS_STREAM_META);
> > > +
> > > +	code = fmt ? fmt->code : meta_out_fmt->code;
> > 
> > When this function is called from __ccs_init_state(), fmt will be NULL,
> > and meta_out_fmt will be uninitialized. Is this intended ?
> 
> Not uninitialised but zero. I'll use zero explicitly instead.
> 
> > > +
> > > +	meta_out_fmt->width = meta_fmt->width = pix_fmt->width;
> > > +	meta_out_fmt->height = meta_fmt->height =
> > > +		sensor->embedded_end - sensor->embedded_start;
> > > +	meta_fmt->code = MEDIA_BUS_FMT_CCS_EMBEDDED;
> > > +
> > > +	csi_format = ccs_validate_csi_data_format(sensor, pix_fmt->code);
> > > +
> > > +	switch (csi_format->compressed) {
> > > +	case 8:
> > > +		meta_out_fmt->code = MEDIA_BUS_FMT_META_8;
> > > +		break;
> > > +	case 10:
> > > +		meta_out_fmt->code = MEDIA_BUS_FMT_META_10;
> > > +		break;
> > > +	case 12:
> > > +		meta_out_fmt->code = MEDIA_BUS_FMT_META_12;
> > > +		break;
> > > +	case 14:
> > > +		meta_out_fmt->code = MEDIA_BUS_FMT_META_14;
> > > +		break;
> > > +	case 16:
> > > +	case 20:
> > > +	case 24: {
> > > +		const struct ccs_embedded_code *embedded_code;
> > > +
> > > +		embedded_code = ccs_embedded_code(csi_format->compressed);
> > > +		meta_out_fmt->code =
> > > +			ccs_default_embedded_code(sensor, embedded_code);
> > > +
> > > +		if (!(CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
> > > +		      BIT(embedded_code->cap_no_legacy)) &&
> > > +		    code == embedded_code->code_legacy) {
> > > +			meta_out_fmt->code = embedded_code->code_legacy;
> > > +			sensor->emb_data_ctrl = 0;
> > > +		}
> > > +
> > > +		if (CCS_LIM(sensor, EMB_DATA_CAPABILITY) &
> > > +		    BIT(embedded_code->cap_two_bytes_per_sample) &&
> > > +		    code == embedded_code->code_two_bytes) {
> > > +			meta_out_fmt->code = embedded_code->code_two_bytes;
> > > +			sensor->emb_data_ctrl = embedded_code->ctrl;
> > > +			meta_fmt->width <<= 1;
> > > +			meta_out_fmt->width <<= 1;
> > > +		}
> > > +
> > > +		break;
> > > +	}
> > > +	default:
> > > +		WARN_ON(1);
> > > +		return 0;
> > > +	}
> > > +
> > > +	if (fmt)
> > > +		*fmt = *meta_out_fmt;
> > > +
> > > +	return 0;
> > > +}
> > > +
> > >  static int ccs_set_format(struct v4l2_subdev *subdev,
> > >  			  struct v4l2_subdev_state *sd_state,
> > >  			  struct v4l2_subdev_format *fmt)
> > > @@ -2218,12 +2421,25 @@ static int ccs_set_format(struct v4l2_subdev *subdev,
> > >  	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
> > >  	struct v4l2_rect *crops[CCS_PADS];
> > >  
> > > +	if (subdev == &sensor->src->sd && fmt->pad == CCS_PAD_META)
> > 
> > You could also write
> > 
> > 	if (ssd == sensor->src && fmt->pad == CCS_PAD_META)
> 
> Sounds good.
> 
> > Same below.
> > 
> > > +		return ccs_get_format(subdev, sd_state, fmt);
> > > +
> > >  	mutex_lock(&sensor->mutex);
> > 
> > Is this needed, shouldn't the state lock be enough ?
> 
> Not while the access to the device's state is serialised using the driver's
> own mutex. This changes two patches later though.

I realized that during the review, two patches later :-)

> > >  
> > > +	if (subdev == &sensor->src->sd && fmt->stream == CCS_STREAM_META) {
> > > +		ccs_set_format_meta(subdev, sd_state, &fmt->format);
> > > +
> > > +		mutex_unlock(&sensor->mutex);
> > > +
> > > +		return 0;
> > > +	}
> > > +
> > >  	if (fmt->pad == ssd->source_pad) {
> > >  		int rval;
> > >  
> > >  		rval = ccs_set_format_source(subdev, sd_state, fmt);
> > > +		if (sensor->embedded_start != sensor->embedded_end)
> > 
> > A ccs_sensor_has_embedded_data() (name bikeshedding allowed) inline
> > helper could be nice to replace this manual check could be nice, as you
> > do the same in many locations below.
> 
> Sounds good.
> 
> > > +			ccs_set_format_meta(subdev, sd_state, NULL);
> > 
> > This doesn't seem correct, you shouldn't set the metadata format on
> > subdevs that are not the source subdev.
> 
> Good point. I'll add a check.
> 
> > A comment to explain how the metadata format is propagated would also be
> > useful.
> 
> I'll add this to the documentation patch which actually could be better
> after this patch, not before.

I meant comments in the source code, to make it easier to follow the
code flow. Format propagation is error-prone, having comments explaining
what the code does next to the code helps during review, and should also
help during futher developments.

> > >  
> > >  		mutex_unlock(&sensor->mutex);
> > >  
> > > @@ -2498,6 +2714,12 @@ static int ccs_sel_supported(struct v4l2_subdev *subdev,
> > >  	struct ccs_sensor *sensor = to_ccs_sensor(subdev);
> > >  	struct ccs_subdev *ssd = to_ccs_subdev(subdev);
> > >  
> > > +	if (sel->stream != CCS_STREAM_PIXEL)
> > > +		return -EINVAL;
> > > +
> > > +	if (sel->pad == CCS_PAD_META)
> > > +		return -EINVAL;
> > > +
> > >  	/* We only implement crop in three places. */
> > >  	switch (sel->target) {
> > >  	case V4L2_SEL_TGT_CROP:
> > > @@ -2542,7 +2764,8 @@ static int ccs_set_crop(struct v4l2_subdev *subdev,
> > >  
> > >  	if (sel->pad == ssd->sink_pad) {
> > >  		struct v4l2_mbus_framefmt *mfmt =
> > > -			v4l2_subdev_state_get_format(sd_state, sel->pad);
> > > +			v4l2_subdev_state_get_format(sd_state, sel->pad,
> > > +						     CCS_STREAM_PIXEL);
> > >  
> > >  		src_size.width = mfmt->width;
> > >  		src_size.height = mfmt->height;
> > > @@ -2603,7 +2826,9 @@ static int ccs_get_selection(struct v4l2_subdev *subdev,
> > >  		} else if (sel->pad == ssd->sink_pad) {
> > >  			struct v4l2_mbus_framefmt *sink_fmt =
> > >  				v4l2_subdev_state_get_format(sd_state,
> > > -							     ssd->sink_pad);
> > > +							     ssd->sink_pad,
> > > +							     CCS_STREAM_PIXEL);
> > > +
> > >  			sel->r.top = sel->r.left = 0;
> > >  			sel->r.width = sink_fmt->width;
> > >  			sel->r.height = sink_fmt->height;
> > > @@ -2686,6 +2911,14 @@ static int ccs_get_frame_desc(struct v4l2_subdev *subdev, unsigned int pad,
> > >  	entry++;
> > >  	desc->num_entries++;
> > >  
> > > +	if (sensor->embedded_start != sensor->embedded_end) {
> > > +		entry->pixelcode = MEDIA_BUS_FMT_CCS_EMBEDDED;
> > 
> > I think you need to report the generic pixel code here.
> 
> I'll fix that for v10.
> 
> > > +		entry->stream = CCS_STREAM_META;
> > > +		entry->bus.csi2.dt = MIPI_CSI2_DT_EMBEDDED_8B;
> > > +		entry++;
> > > +		desc->num_entries++;
> > > +	}
> > > +
> > >  	return 0;
> > >  }
> > >  
> > > @@ -3004,6 +3237,8 @@ static void ccs_cleanup(struct ccs_sensor *sensor)
> > >  	ccs_free_controls(sensor);
> > >  }
> > >  
> > > +static const struct v4l2_subdev_internal_ops ccs_internal_ops;
> > > +
> > >  static int ccs_init_subdev(struct ccs_sensor *sensor,
> > >  			   struct ccs_subdev *ssd, const char *name,
> > >  			   unsigned short num_pads, u32 function,
> > > @@ -3016,15 +3251,18 @@ static int ccs_init_subdev(struct ccs_sensor *sensor,
> > >  	if (!ssd)
> > >  		return 0;
> > >  
> > > -	if (ssd != sensor->src)
> > > +	if (ssd != sensor->src) {
> > >  		v4l2_subdev_init(&ssd->sd, &ccs_ops);
> > > +		ssd->sd.internal_ops = &ccs_internal_ops;
> > > +	}
> > >  
> > >  	ssd->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
> > >  	ssd->sd.entity.function = function;
> > >  	ssd->sensor = sensor;
> > >  
> > >  	ssd->npads = num_pads;
> > > -	ssd->source_pad = num_pads - 1;
> > > +	ssd->source_pad =
> > > +		ssd == sensor->pixel_array ? CCS_PA_PAD_SRC : CCS_PAD_SRC;
> > >  
> > >  	v4l2_i2c_subdev_set_name(&ssd->sd, client, sensor->minfo.name, name);
> > >  
> > > @@ -3038,6 +3276,10 @@ static int ccs_init_subdev(struct ccs_sensor *sensor,
> > >  		ssd->sd.owner = THIS_MODULE;
> > >  		ssd->sd.dev = &client->dev;
> > >  		v4l2_set_subdevdata(&ssd->sd, client);
> > > +	} else {
> > > +		ssd->sd.flags |= V4L2_SUBDEV_FL_STREAMS;
> > > +		ssd->pads[CCS_PAD_META].flags =
> > > +			MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL;
> > >  	}
> > >  
> > >  	rval = media_entity_pads_init(&ssd->sd.entity, ssd->npads, ssd->pads);
> > > @@ -3055,21 +3297,19 @@ static int ccs_init_subdev(struct ccs_sensor *sensor,
> > >  	return 0;
> > >  }
> > >  
> > > -static int ccs_init_state(struct v4l2_subdev *sd,
> > > -			  struct v4l2_subdev_state *sd_state)
> > > +static int __ccs_init_state(struct v4l2_subdev *sd,
> > > +			    struct v4l2_subdev_state *sd_state)
> > >  {
> > >  	struct ccs_subdev *ssd = to_ccs_subdev(sd);
> > >  	struct ccs_sensor *sensor = ssd->sensor;
> > >  	unsigned int pad = ssd == sensor->pixel_array ?
> > >  		CCS_PA_PAD_SRC : CCS_PAD_SINK;
> > >  	struct v4l2_mbus_framefmt *fmt =
> > > -		v4l2_subdev_state_get_format(sd_state, pad);
> > > +		v4l2_subdev_state_get_format(sd_state, pad, CCS_STREAM_PIXEL);
> > >  	struct v4l2_rect *crop =
> > > -		v4l2_subdev_state_get_crop(sd_state, pad);
> > > +		v4l2_subdev_state_get_crop(sd_state, pad, CCS_STREAM_PIXEL);
> > >  	bool is_active = !sd->active_state || sd->active_state == sd_state;
> > >  
> > > -	mutex_lock(&sensor->mutex);
> > > -
> > >  	ccs_get_native_size(ssd, crop);
> > >  
> > >  	fmt->width = crop->width;
> > > @@ -3081,20 +3321,78 @@ static int ccs_init_state(struct v4l2_subdev *sd,
> > >  		if (is_active)
> > >  			sensor->pa_src = *crop;
> > >  
> > > -		mutex_unlock(&sensor->mutex);
> > >  		return 0;
> > >  	}
> > >  
> > > -	fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC);
> > > +	fmt = v4l2_subdev_state_get_format(sd_state, CCS_PAD_SRC,
> > > +					   CCS_STREAM_PIXEL);
> > >  	fmt->code = ssd == sensor->src ?
> > >  		sensor->csi_format->code : sensor->internal_csi_format->code;
> > >  	fmt->field = V4L2_FIELD_NONE;
> > >  
> > >  	ccs_propagate(sd, sd_state, is_active, V4L2_SEL_TGT_CROP);
> > >  
> > > +	return 0;
> > > +}
> > > +
> > > +static int ccs_init_state(struct v4l2_subdev *sd,
> > > +			  struct v4l2_subdev_state *sd_state)
> > > +{
> > > +	struct ccs_subdev *ssd = to_ccs_subdev(sd);
> > > +	struct ccs_sensor *sensor = ssd->sensor;
> > > +	int rval;
> > > +
> > > +	mutex_lock(&sensor->mutex);
> > > +	rval = __ccs_init_state(sd, sd_state);
> > >  	mutex_unlock(&sensor->mutex);
> > >  
> > > -	return 0;
> > > +	return rval;
> > > +}
> > > +
> > > +static int ccs_src_init_state(struct v4l2_subdev *sd,
> > > +			      struct v4l2_subdev_state *sd_state)
> > > +{
> > > +	struct v4l2_subdev_route routes[] = {
> > > +		{
> > > +			.sink_pad = CCS_PAD_SINK,
> > > +			.source_pad = CCS_PAD_SRC,
> > > +			.source_stream = CCS_STREAM_PIXEL,
> > > +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> > > +		}, {
> > > +			.sink_pad = CCS_PAD_META,
> > > +			.source_pad = CCS_PAD_SRC,
> > > +			.source_stream = CCS_STREAM_META,
> > > +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> > > +		}
> > > +	};
> > > +	struct v4l2_subdev_krouting routing = {
> > > +		.routes = routes,
> > > +		.num_routes = 1,
> > > +	};
> > > +	struct ccs_subdev *ssd = to_ccs_subdev(sd);
> > > +	struct ccs_sensor *sensor = ssd->sensor;
> > > +	int rval;
> > > +
> > > +	mutex_lock(&sensor->mutex);
> > 
> > Is this needed, shouldn't the state lock be enough ?
> 
> Same as for the other mutex: this will no longer be needed after relying on
> the state locks for locking, two patches later.
> 
> > > +
> > > +	if (sensor->embedded_start != sensor->embedded_end)
> > > +		routing.num_routes++;
> > > +
> > > +	rval = v4l2_subdev_set_routing(sd, sd_state, &routing);
> > > +	if (rval)
> > > +		goto out;
> > > +
> > > +	rval = __ccs_init_state(sd, sd_state);
> > > +	if (rval)
> > > +		goto out;
> > > +
> > > +	if (sensor->embedded_start != sensor->embedded_end)
> > > +		ccs_set_format_meta(sd, sd_state, NULL);
> > > +
> > > +out:
> > > +	mutex_unlock(&sensor->mutex);
> > > +
> > > +	return rval;
> > >  }
> > >  
> > >  static const struct v4l2_subdev_video_ops ccs_video_ops = {
> > > @@ -3109,6 +3407,14 @@ static const struct v4l2_subdev_pad_ops ccs_pad_ops = {
> > >  	.set_fmt = ccs_set_format,
> > >  	.get_selection = ccs_get_selection,
> > >  	.set_selection = ccs_set_selection,
> > > +};
> > > +
> > > +static const struct v4l2_subdev_pad_ops ccs_src_pad_ops = {
> > > +	.enum_mbus_code = ccs_enum_mbus_code,
> > > +	.get_fmt = ccs_get_format,
> > 
> > I'm surprised you need to implement .get_fmt(). The
> > v4l2_subdev_get_fmt() helper should have been enough.
> 
> It should be possible to get rid of that now, too. I'll add a new patch for
> this.
> 
> > > +	.set_fmt = ccs_set_format,
> > > +	.get_selection = ccs_get_selection,
> > > +	.set_selection = ccs_set_selection,
> > >  	.enable_streams = ccs_enable_streams,
> > >  	.disable_streams = ccs_disable_streams,
> > >  	.get_frame_desc = ccs_get_frame_desc,
> > > @@ -3125,12 +3431,22 @@ static const struct v4l2_subdev_ops ccs_ops = {
> > >  	.sensor = &ccs_sensor_ops,
> > >  };
> > >  
> > > +static const struct v4l2_subdev_ops ccs_src_ops = {
> > > +	.video = &ccs_video_ops,
> > > +	.pad = &ccs_src_pad_ops,
> > > +	.sensor = &ccs_sensor_ops,
> > > +};
> > > +
> > >  static const struct media_entity_operations ccs_entity_ops = {
> > >  	.link_validate = v4l2_subdev_link_validate,
> > >  };
> > >  
> > > -static const struct v4l2_subdev_internal_ops ccs_internal_src_ops = {
> > > +static const struct v4l2_subdev_internal_ops ccs_internal_ops = {
> > >  	.init_state = ccs_init_state,
> > > +};
> > > +
> > > +static const struct v4l2_subdev_internal_ops ccs_src_internal_ops = {
> > > +	.init_state = ccs_src_init_state,
> > >  	.registered = ccs_registered,
> > >  	.unregistered = ccs_unregistered,
> > >  };
> > > @@ -3281,8 +3597,8 @@ static int ccs_probe(struct i2c_client *client)
> > >  
> > >  	sensor->src = &sensor->ssds[sensor->ssds_used];
> > >  
> > > -	v4l2_i2c_subdev_init(&sensor->src->sd, client, &ccs_ops);
> > > -	sensor->src->sd.internal_ops = &ccs_internal_src_ops;
> > > +	v4l2_i2c_subdev_init(&sensor->src->sd, client, &ccs_src_ops);
> > > +	sensor->src->sd.internal_ops = &ccs_src_internal_ops;
> > >  
> > >  	sensor->regulators = devm_kcalloc(&client->dev,
> > >  					  ARRAY_SIZE(ccs_regulators),
> > > @@ -3553,12 +3869,20 @@ static int ccs_probe(struct i2c_client *client)
> > >  		goto out_cleanup;
> > >  	}
> > >  
> > > -	rval = ccs_init_subdev(sensor, sensor->scaler, " scaler", 2,
> > > +	rval = ccs_init_subdev(sensor, sensor->scaler, " scaler",
> > > +			       sensor->ssds_used != CCS_SUBDEVS ?
> > > +			       CCS_PADS_NOMETA :
> > > +			       sensor->embedded_start == sensor->embedded_end ?
> > > +			       CCS_PADS_NOMETA : CCS_PADS,
> > >  			       MEDIA_ENT_F_PROC_VIDEO_SCALER,
> > >  			       "ccs scaler mutex", &scaler_lock_key);
> > >  	if (rval)
> > >  		goto out_cleanup;
> > > -	rval = ccs_init_subdev(sensor, sensor->binner, " binner", 2,
> > > +	rval = ccs_init_subdev(sensor, sensor->binner, " binner",
> > > +			       sensor->ssds_used == CCS_SUBDEVS ?
> > > +			       CCS_PADS_NOMETA :
> > > +			       sensor->embedded_start == sensor->embedded_end ?
> > > +			       CCS_PADS_NOMETA : CCS_PADS,
> > >  			       MEDIA_ENT_F_PROC_VIDEO_SCALER,
> > >  			       "ccs binner mutex", &binner_lock_key);
> > >  	if (rval)
> > > diff --git a/drivers/media/i2c/ccs/ccs.h b/drivers/media/i2c/ccs/ccs.h
> > > index 90b442a3d53e..477b2fb99aa0 100644
> > > --- a/drivers/media/i2c/ccs/ccs.h
> > > +++ b/drivers/media/i2c/ccs/ccs.h
> > > @@ -173,11 +173,18 @@ struct ccs_csi_data_format {
> > >  #define CCS_SUBDEVS			3
> > >  
> > >  #define CCS_PA_PAD_SRC			0
> > > -#define CCS_PAD_SINK			0
> > > -#define CCS_PAD_SRC			1
> > > -#define CCS_PADS			2
> > > +enum {
> > > +	CCS_PAD_SINK,
> > > +	CCS_PAD_SRC,
> > > +	CCS_PAD_META,
> > > +	CCS_PADS_NOMETA = CCS_PAD_META,
> > 
> > This doesn't seem to improve readability of the code above :-S
> 
> The number of pads is different on sub-devices, with the introduction of
> one more internal pad in the source. I don't think another enum for the
> purpose would be better as the same code is dealing with all of the
> driver's sub-devices.
> 
> > > +	CCS_PADS,
> > > +};
> > >  
> > > -#define CCS_STREAM_PIXEL		0
> > > +enum {
> > > +	CCS_STREAM_PIXEL,
> > > +	CCS_STREAM_META,
> > > +};
> > >  
> > >  struct ccs_binning_subtype {
> > >  	u8 horizontal:4;
> > > @@ -228,6 +235,9 @@ struct ccs_sensor {
> > >  	int default_pixel_order;
> > >  	struct ccs_data_container sdata, mdata;
> > >  
> > > +	u32 embedded_mbus_code;
> > 
> > Not used.
> 
> I'll remove it.
> 
> > > +	u8 emb_data_ctrl;
> > 
> > The general direction I'd like to take with v4l2_subdev_state is to
> > avoid storing state information in the device private structure. Could
> > this be dropped and computed in ccs_enable_streams() instead ?
> 
> Deriving this information from mbus code is possible. I'll add a new patch
> for this.
> 
> > > +
> > >  	u8 binning_horizontal;
> > >  	u8 binning_vertical;
> > >  

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 15/46] media: v4l: subdev: Add len_routes field to struct v4l2_subdev_routing

[Laurent Pinchart]

Hi Sakari,

On Tue, Apr 23, 2024 at 10:45:54AM +0000, Sakari Ailus wrote:
> On Sat, Apr 20, 2024 at 01:45:19AM +0300, Laurent Pinchart wrote:
> > On Tue, Apr 16, 2024 at 10:32:48PM +0300, Sakari Ailus wrote:
> > > The len_routes field is used to tell the size of the routes array in
> > > struct v4l2_subdev_routing. This way the number of routes returned from
> > > S_ROUTING IOCTL may be larger than the number of routes provided, in case
> > > there are more routes returned by the driver.
> > > 
> > > Note that this uAPI is still disabled in the code, so this change can
> > > safely be done. Anyone who manually patched the code to enable this uAPI
> > > must update their code.
> > > 
> > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > ---
> > >  .../media/v4l/vidioc-subdev-g-routing.rst     | 50 +++++++++++++------
> > >  drivers/media/v4l2-core/v4l2-ioctl.c          |  4 +-
> > >  drivers/media/v4l2-core/v4l2-subdev.c         | 12 ++---
> > >  include/media/v4l2-subdev.h                   |  2 +
> > >  include/uapi/linux/v4l2-subdev.h              |  9 ++--
> > >  5 files changed, 52 insertions(+), 25 deletions(-)
> > > 
> > > diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > > index 26b5004bfe6d..27eb4c82a0e1 100644
> > > --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > > +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > > @@ -43,23 +43,42 @@ The routing configuration determines the flows of data inside an entity.
> > >  Drivers report their current routing tables using the
> > >  ``VIDIOC_SUBDEV_G_ROUTING`` ioctl and application may enable or disable routes
> > >  with the ``VIDIOC_SUBDEV_S_ROUTING`` ioctl, by adding or removing routes and
> > > -setting or clearing flags of the  ``flags`` field of a
> > > -struct :c:type:`v4l2_subdev_route`.
> > > +setting or clearing flags of the ``flags`` field of a struct
> > > +:c:type:`v4l2_subdev_route`.
> > >  
> > > -All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called. This
> > > -means that the userspace must reconfigure all streams after calling the ioctl
> > > -with e.g. ``VIDIOC_SUBDEV_S_FMT``.
> > > +All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called.
> > > +This means that the userspace must reconfigure all stream formats and selections
> > > +after calling the ioctl with e.g. ``VIDIOC_SUBDEV_S_FMT``.
> > >  
> > >  Only subdevices which have both sink and source pads can support routing.
> > >  
> > > -When inspecting routes through ``VIDIOC_SUBDEV_G_ROUTING`` and the application
> > > -provided ``num_routes`` is not big enough to contain all the available routes
> > > -the subdevice exposes, drivers return the ENOSPC error code and adjust the
> > > -value of the ``num_routes`` field. Application should then reserve enough memory
> > > -for all the route entries and call ``VIDIOC_SUBDEV_G_ROUTING`` again.
> > > -
> > > -On a successful ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the
> > > -``num_routes`` field to reflect the actual number of routes returned.
> > > +The ``len_routes`` field indicates the number of routes that can fit in the
> > > +``routes`` array allocated by userspace. It is set by applications for both
> > > +ioctls to indicate how many routes the kernel can return, and is never modified
> > > +by the kernel.
> > > +
> > > +The ``num_routes`` field, when returned from the kernel on both IOCTLs,
> > > +indicates the number of routes in the subdevice routing table and when calling
> > > +``VIDIOC_SUBDEV_S_ROUTING``, it is set by userspace to the number of routes that
> > > +the application stored in the ``routes`` array. The value returned by the kernel
> > > +may be smaller or larger than the value of ``num_routes`` set by the application
> > > +for ``VIDIOC_SUBDEV_S_ROUTING``, as drivers may adjust the requested routing
> > > +table.
> > 
> > I still think the proposal I made when reviewing the previous version is
> > clearer :-)
> > 
> > ----
> > The ``num_routes`` field indicates the number of routes in the subdevice routing
> > table. For ``VIDIOC_SUBDEV_S_ROUTING``, it is set by userspace to the number of
> > routes that the application stored in the ``routes`` array. For both ioctls, it
> > is returned by the kernel and indicates how many routes are stored in the
> > subdevice routing table. This may be smaller or larger than the value of
> > ``num_routes`` set by the application for ``VIDIOC_SUBDEV_S_ROUTING``, as
> > drivers may adjust the requested routing table.
> > ----
> > 
> > You replied that
> > 
> > > For S_ROUTING this is the number of routes in the IOCTL argument. The
> > > routing table may contain more (static routes).
> > 
> > and that's right, but, even when set by userspace for S_ROUTING, the
> > num_routes fields is the number of routes that userspace tries to set in
> > the routing table. I think starting with a first sentence that describes
> > what the field contains, and then explaining how it's used for the
> > different ioctls by userspace and kernel space, is clearer.
> 
> The problem with your suggestion is that it's not entirely correct:
> num_routes is indeed used for two different purposes. Removing " in the
> subdevice routing table" in the first sentence would be a simple fix.

How about dropping just "subdevice", and keeping "in the routing table"
? That should cover both cases.

> > > +
> > > +The kernel can return a ``num_routes`` value larger than ``len_routes`` from
> > > +both ioctls. This indicates thare are more routes in the routing table than fits
> > > +the ``routes`` array. In this case, the ``routes`` array is filled by the kernel
> > > +with the first ``len_routes`` entries of the subdevice routing table. This is
> > > +not considered to be an error, and the ioctl call succeeds. If the applications
> > > +wants to retrieve the missing routes, it can issue a new
> > > +``VIDIOC_SUBDEV_G_ROUTING`` call with a large enough ``routes`` array.
> > > +
> > > +indicate there are more routes than fits to the ``routes`` array. In this
> > > +case first ``len_routes`` were returned back to the userspace in the
> > > +``routes`` array. This is not considered as an error.
> > 
> > I think these 3 lines are a leftover.
> 
> Yes, I'll remove them.
> 
> > > +
> > > +Also ``VIDIOC_SUBDEV_S_ROUTING`` may return more route than the user provided in
> > 
> > s/Also //
> > s/route/routes/
> 
> Yes.
> 
> > Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> 
> Thanks!
> 
> > 
> > > +``num_routes`` field due to e.g. hardware properties.
> > >  
> > >  .. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
> > >  
> > > @@ -74,6 +93,9 @@ On a successful ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the
> > >        - ``which``
> > >        - Routing table to be accessed, from enum
> > >          :ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
> > > +    * - __u32
> > > +      - ``len_routes``
> > > +      - The length of the array (as in memory reserved for the array)
> > >      * - struct :c:type:`v4l2_subdev_route`
> > >        - ``routes[]``
> > >        - Array of struct :c:type:`v4l2_subdev_route` entries
> > > @@ -81,7 +103,7 @@ On a successful ``VIDIOC_SUBDEV_G_ROUTING`` call the driver updates the
> > >        - ``num_routes``
> > >        - Number of entries of the routes array
> > >      * - __u32
> > > -      - ``reserved``\ [5]
> > > +      - ``reserved``\ [11]
> > >        - Reserved for future extensions. Applications and drivers must set
> > >  	the array to zero.
> > >  
> > > diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> > > index 1863ecae9ec9..f30f61c008c7 100644
> > > --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> > > +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> > > @@ -3185,13 +3185,13 @@ static int check_array_args(unsigned int cmd, void *parg, size_t *array_size,
> > >  	case VIDIOC_SUBDEV_S_ROUTING: {
> > >  		struct v4l2_subdev_routing *routing = parg;
> > >  
> > > -		if (routing->num_routes > 256)
> > > +		if (routing->len_routes > 256)
> > >  			return -E2BIG;
> > >  
> > >  		*user_ptr = u64_to_user_ptr(routing->routes);
> > >  		*kernel_ptr = (void **)&routing->routes;
> > >  		*array_size = sizeof(struct v4l2_subdev_route)
> > > -			    * routing->num_routes;
> > > +			    * routing->len_routes;
> > >  		ret = 1;
> > >  		break;
> > >  	}
> > > diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> > > index 2ba11e5343f0..904378007a79 100644
> > > --- a/drivers/media/v4l2-core/v4l2-subdev.c
> > > +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> > > @@ -927,6 +927,9 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
> > >  		if (routing->which != V4L2_SUBDEV_FORMAT_TRY && ro_subdev)
> > >  			return -EPERM;
> > >  
> > > +		if (routing->num_routes > routing->len_routes)
> > > +			return -EINVAL;
> > > +
> > >  		memset(routing->reserved, 0, sizeof(routing->reserved));
> > >  
> > >  		for (i = 0; i < routing->num_routes; ++i) {
> > > @@ -953,6 +956,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
> > >  		}
> > >  
> > >  		krouting.num_routes = routing->num_routes;
> > > +		krouting.len_routes = routing->len_routes;
> > >  		krouting.routes = routes;
> > >  
> > >  		return v4l2_subdev_call(sd, pad, set_routing, state,
> > > @@ -973,14 +977,10 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
> > >  
> > >  		krouting = &state->routing;
> > >  
> > > -		if (routing->num_routes < krouting->num_routes) {
> > > -			routing->num_routes = krouting->num_routes;
> > > -			return -ENOSPC;
> > > -		}
> > > -
> > >  		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
> > >  		       krouting->routes,
> > > -		       krouting->num_routes * sizeof(*krouting->routes));
> > > +		       min(krouting->num_routes, routing->len_routes) *
> > > +		       sizeof(*krouting->routes));
> > >  		routing->num_routes = krouting->num_routes;
> > >  
> > >  		return 0;
> > > diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> > > index 9cce48365975..1df6b963a1c9 100644
> > > --- a/include/media/v4l2-subdev.h
> > > +++ b/include/media/v4l2-subdev.h
> > > @@ -728,12 +728,14 @@ struct v4l2_subdev_stream_configs {
> > >  /**
> > >   * struct v4l2_subdev_krouting - subdev routing table
> > >   *
> > > + * @len_routes: length of routes array, in routes
> > >   * @num_routes: number of routes
> > >   * @routes: &struct v4l2_subdev_route
> > >   *
> > >   * This structure contains the routing table for a subdev.
> > >   */
> > >  struct v4l2_subdev_krouting {
> > > +	unsigned int len_routes;
> > >  	unsigned int num_routes;
> > >  	struct v4l2_subdev_route *routes;
> > >  };
> > > diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
> > > index 81a24bd38003..6a39128d0606 100644
> > > --- a/include/uapi/linux/v4l2-subdev.h
> > > +++ b/include/uapi/linux/v4l2-subdev.h
> > > @@ -228,15 +228,18 @@ struct v4l2_subdev_route {
> > >   * struct v4l2_subdev_routing - Subdev routing information
> > >   *
> > >   * @which: configuration type (from enum v4l2_subdev_format_whence)
> > > - * @num_routes: the total number of routes in the routes array
> > > + * @len_routes: the length of the routes array, in routes
> > >   * @routes: pointer to the routes array
> > > + * @num_routes: the total number of routes, possibly more than fits in the
> > > + *		routes array
> > >   * @reserved: drivers and applications must zero this array
> > >   */
> > >  struct v4l2_subdev_routing {
> > >  	__u32 which;
> > > -	__u32 num_routes;
> > > +	__u32 len_routes;
> > >  	__u64 routes;
> > > -	__u32 reserved[6];
> > > +	__u32 num_routes;
> > > +	__u32 reserved[11];
> > >  };
> > >  
> > >  /*

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 10/46] media: Documentation: v4l: Document internal sink pads

[Laurent Pinchart]

Hi Sakari,

On Tue, Apr 23, 2024 at 10:27:11AM +0000, Sakari Ailus wrote:
> On Fri, Apr 19, 2024 at 09:49:26PM +0300, Laurent Pinchart wrote:
> > On Tue, Apr 16, 2024 at 10:32:43PM +0300, Sakari Ailus wrote:
> > > Document internal sink pads, pads that have both SINK and INTERNAL flags
> > > set. Use the IMX219 camera sensor as an example.
> > > 
> > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > Reviewed-by Julien Massot <julien.massot@collabora.com>
> > > ---
> > >  .../userspace-api/media/v4l/dev-subdev.rst    | 145 ++++++++++++++++++
> > >  1 file changed, 145 insertions(+)
> > > 
> > > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > index b76e02e54512..d30dcb9e2537 100644
> > > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > @@ -553,6 +553,27 @@ A stream at a specific point in the media pipeline is identified by the
> > >  sub-device and a (pad, stream) pair. For sub-devices that do not support
> > >  multiplexed streams the 'stream' field is always 0.
> > >  
> > > +.. _v4l2-subdev-internal-source-pads:
> > > +
> > > +Internal sink pads and routing
> > > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > +
> > > +Cases where a single sub-device source pad is traversed by multiple streams, one
> > > +or more of which originate from within the sub-device itself, are special as
> > > +there is no external sink pad for such routes. In those cases, the sources of
> > > +the internally generated streams are represented by internal sink pads, which
> > > +are sink pads that have the :ref:`MEDIA_PAD_FL_INTERNAL <MEDIA-PAD-FL-INTERNAL>`
> > > +pad flag set.
> > > +
> > > +Internal pads have all the properties of an external pad, including formats and
> > > +selections. The format in this case is the source format of the stream. An
> > > +internal pad always has a single stream only (0).
> > > +
> > > +Routes from an internal sink pad to an external source pad are typically not
> > > +modifiable but they can be activated and deactivated using the
> > > +:ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE <v4l2-subdev-routing-flags>` flag, depending
> > > +on driver capabilities.
> > > +
> > >  Interaction between routes, streams, formats and selections
> > >  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > >  
> > > @@ -668,3 +689,127 @@ To configure this pipeline, the userspace must take the following steps:
> > >     the configurations along the stream towards the receiver, using
> > >     :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each
> > >     stream endpoint in each sub-device.
> > > +
> > > +Internal pads setup example
> > > +^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > +
> > > +A simple example of a multiplexed stream setup might be as follows:
> > > +
> > > +- An IMX219 camera sensor source sub-device, with one source pad (0), one
> > > +  internal sink pad (1) as the source of the image data and an internal sink
> > > +  pad (2) as the source of the embedded data. There are two routes, one from the
> > > +  internal sink pad 1 to the source 0 (image data) and another from the internal
> > 
> > s/source 0/source pad 0/
> 
> Yes.
> 
> > > +  sink pad 2 to the source pad 0 (embedded data). Both streams are always
> > > +  active, i.e. there is no need to separately enable the embedded data
> > > +  stream. The sensor uses the CSI-2 interface.
> > > +
> > > +- A CSI-2 receiver in the SoC. The receiver has a single sink pad (pad 0),
> > > +  connected to the sensor, and two source pads (pads 1 and 2), to the DMA
> > > +  engine. The receiver demultiplexes the incoming streams to the source pads.
> > 
> > s/engine/engines/
> > 
> > maybe better, "to two DMA engines".
> 
> "to the two DMA engines".
> 
> > > +
> > > +- DMA engines in the SoC, one for each stream. Each DMA engine is connected to a
> > > +  single source pad of the receiver.
> > > +
> > > +The sensor and the receiver are modelled as V4L2 sub-devices, exposed to
> > > +userspace via /dev/v4l-subdevX device nodes. The DMA engines are modelled as
> > > +V4L2 devices, exposed to userspace via /dev/videoX nodes.
> > > +
> > > +To configure this pipeline, the userspace must take the following steps:
> > > +
> > > +1) Set up media links between entities: enable the links from the sensor to the
> > > +   receiver and from the receiver to the DMA engines. This step does not differ
> > > +   from normal non-multiplexed media controller setup.
> > > +
> > > +2) Configure routing
> > > +
> > > +.. flat-table:: Camera sensor. There are no configurable routes.
> > > +    :header-rows: 1
> > > +
> > > +    * - Sink Pad/Stream
> > > +      - Source Pad/Stream
> > > +      - Routing Flags
> > > +      - Comments
> > > +    * - 1/0
> > > +      - 0/0
> > > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > > +      - Pixel data stream from the sink pad
> > 
> > "from the internal image sink pad"
> > 
> > > +    * - 2/0
> > > +      - 0/1
> > > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > > +      - Metadata stream from the internal sink pad
> > 
> > "internal embedded data sink pad"
> 
> Yes...
> 
> > > +
> > > +.. flat-table:: Receiver routing table. Typically both routes need to be
> > > +		explicitly set.
> > > +    :header-rows:  1
> > > +
> > > +    * - Sink Pad/Stream
> > > +      - Source Pad/Stream
> > > +      - Routing Flags
> > > +      - Comments
> > > +    * - 0/0
> > > +      - 1/0
> > > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > > +      - Pixel data stream from camera sensor
> > > +    * - 0/1
> > > +      - 2/0
> > > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > > +      - Metadata stream from camera sensor
> > > +
> > > +The options available in sensor's routing configuration are dictated by hardware
> > 
> > s/in sensor/in the sensor/
> 
> Agreed.
> 
> > > +capabilities: typically camera sensors always produce an image data stream while
> > > +it may be possible to enable and disable the embedded data stream.
> > 
> > Should this go after the first table ?
> 
> I'll move it there.
> 
> > > +
> > > +3) Configure formats and selections
> > > +
> > > +   This example assumes that the formats are propagated from sink pad to the
> > > +   source pad as-is. The tables contain fields of both struct v4l2_subdev_format
> > > +   and struct v4l2_mbus_framefmt.
> > > +
> > > +.. flat-table:: Formats set on the sub-devices. Bold values are set, others are
> > > +                static or propagated. The order is aligned with configured
> > > +                routes.
> > > +    :header-rows: 1
> > > +    :fill-cells:
> > > +
> > > +    * - Sub-device
> > > +      - Pad/Stream
> > > +      - Width
> > > +      - Height
> > > +      - Code
> > > +    * - :rspan:`3` IMX219
> > > +      - 1/0
> > > +      - 3296
> > > +      - 2480
> > > +      - MEDIA_BUS_FMT_SRGGB10
> > > +    * - 0/0
> > > +      - **3296**
> > > +      - **2480**
> > > +      - **MEDIA_BUS_FMT_SRGGB10**
> > > +    * - 2/0
> > > +      - 3296
> > > +      - 2
> > > +      - MEDIA_BUS_FMT_IMX219_EMBEDDED
> > 
> > We need a patch in this series to define this format.
> 
> I'm waiting for your imx290 patches. :-)

I assume you mean imx219 :-) They will come, but we need to discuss
usage of internal pads for raw sensors first. That won't make it in time
for v6.10, I hope we'll be in time for v6.11.

> > > +    * - 1/1
> > 
> > I think this should be 0/1.
> 
> Good catch.
> 
> > > +      - 3296
> > > +      - 2
> > > +      - MEDIA_BUS_FMT_META_10
> > > +    * - :rspan:`3` CSI-2 receiver
> > > +      - 0/0
> > > +      - **3296**
> > > +      - **2480**
> > > +      - **MEDIA_BUS_FMT_SRGGB10**
> > > +    * - 1/0
> > > +      - 3296
> > > +      - 2480
> > > +      - MEDIA_BUS_FMT_SRGGB10
> > > +    * - 0/1
> > > +      - **3296**
> > > +      - **2**
> > > +      - **MEDIA_BUS_FMT_META_10**
> > > +    * - 2/0
> > > +      - 3296
> > > +      - 2
> > > +      - MEDIA_BUS_FMT_META_10
> > > +
> > > +The embedded data format does not need to be configured as the format is
> > 
> > Do you mean the "format on the sensor's pads" ? It's a bit confusing if
> > you don't specify it.
> 
> That's what was meant, I'll add that.
> 
> > > +dictated by the pixel data format in this case.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 11/46] media: Documentation: Document S_ROUTING behaviour

[Laurent Pinchart]

Hi Sakari,

On Tue, Apr 23, 2024 at 10:08:05AM +0000, Sakari Ailus wrote:
> On Fri, Apr 19, 2024 at 08:17:20PM +0300, Laurent Pinchart wrote:
> > On Tue, Apr 16, 2024 at 10:32:44PM +0300, Sakari Ailus wrote:
> > > Document S_ROUTING behaviour for different devices.
> > > 
> > > Generally in devices that produce streams the streams are static and some
> > > can be enabled and disabled, whereas in devices that just transport them
> > > or write them to memory, more configurability is allowed. Document this.
> > > 
> > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > Reviewed-by: Julien Massot <julien.massot@collabora.com>
> > > ---
> > >  .../userspace-api/media/v4l/dev-subdev.rst    | 24 +++++++++++++++++++
> > >  1 file changed, 24 insertions(+)
> > > 
> > > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > index d30dcb9e2537..de8dfd4f11a5 100644
> > > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > @@ -593,6 +593,30 @@ Any configurations of a stream within a pad, such as format or selections,
> > >  are independent of similar configurations on other streams. This is
> > >  subject to change in the future.
> > >  
> > > +Device types and routing setup
> > > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > +
> > > +Different kinds of sub-devices have differing behaviour for route activation,
> > > +depending on the hardware. In all cases, however, only routes that have the
> > > +``V4L2_SUBDEV_STREAM_FL_ACTIVE`` flag set are active.
> > > +
> > > +Devices generating the streams may allow enabling and disabling some of the
> > > +routes or the configuration is fixed. If the routes can be disabled, not
> > 
> > "... some of the routes, or have a fixed routing configuration."
> 
> Seems fine.
> 
> > > +declaring the routes (or declaring them without
> > > +``VIDIOC_SUBDEV_STREAM_FL_ACTIVE`` flag set) in ``VIDIOC_SUBDEV_S_ROUTING`` will
> > > +disable the routes while the sub-device driver retains the streams and their
> > > +format and selection configuration.
> > 
> > I still find the "retains their format and selection configuration"
> > quite unclear :-S
> 
> Alternatively we could say that the routes are simply not active, without
> referring to explicitly to formats and selections. I.e.:
> 
> If the routes can be disabled, not declaring the routes (or declaring them
> without ``VIDIOC_SUBDEV_STREAM_FL_ACTIVE`` flag set) in
> ``VIDIOC_SUBDEV_S_ROUTING`` will disable the routes.

I'm fine with that.

> > > The ``VIDIOC_SUBDEV_S_ROUTING`` will still
> > 
> > s/will still/ioctl will still/
> 
> Both seem to exist, more common is without "ioctl":
> 
> $ git grep '[`<]VIDIOC_SUBDEV' -- Documentation/userspace-api/media/|wc -l
> 84
> $ git grep -i "VIDIOC_SUBDEV.*ioctl" -- Documentation/userspace-api/media/|wc -l
> 34

You'll often find "ioctl" at the beginning of the next line :-) If you
would like to avoid it, you should drop "The" at the beginning of the
sentence.

> > > +return such routes back to the user in the routes array, with the
> > > +``V4L2_SUBDEV_STREAM_FL_ACTIVE`` flag unset.
> > > +
> > > +Devices transporting the streams almost always have more configurability with
> > > +respect to routing. Typically any route between the sub-device's sink and source
> > > +pads is possible, and multiple routes (usually up to certain limited number) may
> > > +be active simultaneously. For such devices, no routes are created by the driver
> > > +and user-created routes are fully replaced when ``VIDIOC_SUBDEV_S_ROUTING`` is
> > > +called on the sub-device. Such newly created routes have the device's default
> > > +configuration for format and selection rectangles.
> > > +
> > >  Configuring streams
> > >  ^^^^^^^^^^^^^^^^^^^
> > >  

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 30/46] media: ccs: Add support for embedded data stream

[Sakari Ailus]

Hi Laurent,

On Tue, Apr 23, 2024 at 03:50:36PM +0300, Laurent Pinchart wrote:
> > > > +		return ccs_get_format(subdev, sd_state, fmt);
> > > > +
> > > >  	mutex_lock(&sensor->mutex);
> > > 
> > > Is this needed, shouldn't the state lock be enough ?
> > 
> > Not while the access to the device's state is serialised using the driver's
> > own mutex. This changes two patches later though.
> 
> I realized that during the review, two patches later :-)

Yes, this was my expectation, too. ;)

> 
> > > >  
> > > > +	if (subdev == &sensor->src->sd && fmt->stream == CCS_STREAM_META) {
> > > > +		ccs_set_format_meta(subdev, sd_state, &fmt->format);
> > > > +
> > > > +		mutex_unlock(&sensor->mutex);
> > > > +
> > > > +		return 0;
> > > > +	}
> > > > +
> > > >  	if (fmt->pad == ssd->source_pad) {
> > > >  		int rval;
> > > >  
> > > >  		rval = ccs_set_format_source(subdev, sd_state, fmt);
> > > > +		if (sensor->embedded_start != sensor->embedded_end)
> > > 
> > > A ccs_sensor_has_embedded_data() (name bikeshedding allowed) inline
> > > helper could be nice to replace this manual check could be nice, as you
> > > do the same in many locations below.
> > 
> > Sounds good.
> > 
> > > > +			ccs_set_format_meta(subdev, sd_state, NULL);
> > > 
> > > This doesn't seem correct, you shouldn't set the metadata format on
> > > subdevs that are not the source subdev.
> > 
> > Good point. I'll add a check.
> > 
> > > A comment to explain how the metadata format is propagated would also be
> > > useful.
> > 
> > I'll add this to the documentation patch which actually could be better
> > after this patch, not before.
> 
> I meant comments in the source code, to make it easier to follow the
> code flow. Format propagation is error-prone, having comments explaining
> what the code does next to the code helps during review, and should also
> help during futher developments.

I'll add some here, too.

...

> > > > @@ -3109,6 +3407,14 @@ static const struct v4l2_subdev_pad_ops ccs_pad_ops = {
> > > >  	.set_fmt = ccs_set_format,
> > > >  	.get_selection = ccs_get_selection,
> > > >  	.set_selection = ccs_set_selection,
> > > > +};
> > > > +
> > > > +static const struct v4l2_subdev_pad_ops ccs_src_pad_ops = {
> > > > +	.enum_mbus_code = ccs_enum_mbus_code,
> > > > +	.get_fmt = ccs_get_format,
> > > 
> > > I'm surprised you need to implement .get_fmt(). The
> > > v4l2_subdev_get_fmt() helper should have been enough.
> > 
> > It should be possible to get rid of that now, too. I'll add a new patch for
> > this.

Now I remember why it's here. The controls affect the mbus code and
changing this is outside the scope for now (I'm not sure if someone would
complain about this changing).

Presumably such changes could be merged with the sensor API changes in
order to avoid several separate changes, so to be determined later on. The
same goes for this patch (post 6.10).

-- 
Regards,

Sakari Ailus

Re: [PATCH v9 11/46] media: Documentation: Document S_ROUTING behaviour

[Sakari Ailus]

Hi Laurent,

On Tue, Apr 23, 2024 at 03:59:44PM +0300, Laurent Pinchart wrote:
> Hi Sakari,
> 
> On Tue, Apr 23, 2024 at 10:08:05AM +0000, Sakari Ailus wrote:
> > On Fri, Apr 19, 2024 at 08:17:20PM +0300, Laurent Pinchart wrote:
> > > On Tue, Apr 16, 2024 at 10:32:44PM +0300, Sakari Ailus wrote:
> > > > Document S_ROUTING behaviour for different devices.
> > > > 
> > > > Generally in devices that produce streams the streams are static and some
> > > > can be enabled and disabled, whereas in devices that just transport them
> > > > or write them to memory, more configurability is allowed. Document this.
> > > > 
> > > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > > Reviewed-by: Julien Massot <julien.massot@collabora.com>
> > > > ---
> > > >  .../userspace-api/media/v4l/dev-subdev.rst    | 24 +++++++++++++++++++
> > > >  1 file changed, 24 insertions(+)
> > > > 
> > > > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > > index d30dcb9e2537..de8dfd4f11a5 100644
> > > > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > > @@ -593,6 +593,30 @@ Any configurations of a stream within a pad, such as format or selections,
> > > >  are independent of similar configurations on other streams. This is
> > > >  subject to change in the future.
> > > >  
> > > > +Device types and routing setup
> > > > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > > +
> > > > +Different kinds of sub-devices have differing behaviour for route activation,
> > > > +depending on the hardware. In all cases, however, only routes that have the
> > > > +``V4L2_SUBDEV_STREAM_FL_ACTIVE`` flag set are active.
> > > > +
> > > > +Devices generating the streams may allow enabling and disabling some of the
> > > > +routes or the configuration is fixed. If the routes can be disabled, not
> > > 
> > > "... some of the routes, or have a fixed routing configuration."
> > 
> > Seems fine.
> > 
> > > > +declaring the routes (or declaring them without
> > > > +``VIDIOC_SUBDEV_STREAM_FL_ACTIVE`` flag set) in ``VIDIOC_SUBDEV_S_ROUTING`` will
> > > > +disable the routes while the sub-device driver retains the streams and their
> > > > +format and selection configuration.
> > > 
> > > I still find the "retains their format and selection configuration"
> > > quite unclear :-S
> > 
> > Alternatively we could say that the routes are simply not active, without
> > referring to explicitly to formats and selections. I.e.:
> > 
> > If the routes can be disabled, not declaring the routes (or declaring them
> > without ``VIDIOC_SUBDEV_STREAM_FL_ACTIVE`` flag set) in
> > ``VIDIOC_SUBDEV_S_ROUTING`` will disable the routes.
> 
> I'm fine with that.
> 
> > > > The ``VIDIOC_SUBDEV_S_ROUTING`` will still
> > > 
> > > s/will still/ioctl will still/
> > 
> > Both seem to exist, more common is without "ioctl":
> > 
> > $ git grep '[`<]VIDIOC_SUBDEV' -- Documentation/userspace-api/media/|wc -l
> > 84
> > $ git grep -i "VIDIOC_SUBDEV.*ioctl" -- Documentation/userspace-api/media/|wc -l
> > 34
> 
> You'll often find "ioctl" at the beginning of the next line :-) If you
> would like to avoid it, you should drop "The" at the beginning of the
> sentence.

Sounds good.

> 
> > > > +return such routes back to the user in the routes array, with the
> > > > +``V4L2_SUBDEV_STREAM_FL_ACTIVE`` flag unset.
> > > > +
> > > > +Devices transporting the streams almost always have more configurability with
> > > > +respect to routing. Typically any route between the sub-device's sink and source
> > > > +pads is possible, and multiple routes (usually up to certain limited number) may
> > > > +be active simultaneously. For such devices, no routes are created by the driver
> > > > +and user-created routes are fully replaced when ``VIDIOC_SUBDEV_S_ROUTING`` is
> > > > +called on the sub-device. Such newly created routes have the device's default
> > > > +configuration for format and selection rectangles.
> > > > +
> > > >  Configuring streams
> > > >  ^^^^^^^^^^^^^^^^^^^
> > > >  
> 

-- 
Regards,

Sakari Ailus

Re: [PATCH v9 15/46] media: v4l: subdev: Add len_routes field to struct v4l2_subdev_routing

[Sakari Ailus]

Hi Laurent,

On Tue, Apr 23, 2024 at 03:54:50PM +0300, Laurent Pinchart wrote:
> How about dropping just "subdevice", and keeping "in the routing table"
> ? That should cover both cases.

Sounds good!

-- 
Sakari Ailus

Re: [PATCH v9 10/46] media: Documentation: v4l: Document internal sink pads

[Sakari Ailus]

Hi Laurent,

On Tue, Apr 23, 2024 at 03:56:18PM +0300, Laurent Pinchart wrote:
> > > > +      - MEDIA_BUS_FMT_IMX219_EMBEDDED
> > > 
> > > We need a patch in this series to define this format.
> > 
> > I'm waiting for your imx290 patches. :-)
> 
> I assume you mean imx219 :-) They will come, but we need to discuss
> usage of internal pads for raw sensors first. That won't make it in time
> for v6.10, I hope we'll be in time for v6.11.

Sure. This patch should actually be located later in the set, after the
INTERNAL pad flag patch. I'll move it there for v10.

-- 
Regards,

Sakari Ailus

Re: [PATCH v9 42/46] media: ov2740: Add generic sensor fwnode properties as controls

[Sakari Ailus]

Hi Laurent,

On Sat, Apr 20, 2024 at 12:40:16PM +0300, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.
> 
> On Tue, Apr 16, 2024 at 10:33:15PM +0300, Sakari Ailus wrote:
> > Add generic sensor property information as controĺs by using
> > v4l2_fwnode_device_parse() and v4l2_ctrl_new_fwnode_properties().
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  drivers/media/i2c/ov2740.c | 6 ++++++
> >  1 file changed, 6 insertions(+)
> > 
> > diff --git a/drivers/media/i2c/ov2740.c b/drivers/media/i2c/ov2740.c
> > index dc0931308053..e37d824291fe 100644
> > --- a/drivers/media/i2c/ov2740.c
> > +++ b/drivers/media/i2c/ov2740.c
> > @@ -779,6 +779,8 @@ static const struct v4l2_ctrl_ops ov2740_ctrl_ops = {
> >  
> >  static int ov2740_init_controls(struct ov2740 *ov2740)
> >  {
> > +	struct i2c_client *client = v4l2_get_subdevdata(&ov2740->sd);
> > +	struct v4l2_fwnode_device_properties props;
> >  	struct v4l2_ctrl_handler *ctrl_hdlr;
> >  	s64 exposure_max, h_blank, pixel_rate;
> >  	u32 vblank_min, vblank_max, vblank_default;
> > @@ -789,6 +791,10 @@ static int ov2740_init_controls(struct ov2740 *ov2740)
> >  	if (ret)
> >  		return ret;
> >  
> > +	if (!v4l2_fwnode_device_parse(&client->dev, &props))
> 
> If you moved the parsing earlier, you could set the right number of
> controls when initializing the handler. This being said, maybe we should
> instead try to get rid of the controls count hint to the handler
> initialization function.

I'm not quite sure how that's related.

But I'll update the number.

> 
> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Thanks!

> 
> > +		v4l2_ctrl_new_fwnode_properties(ctrl_hdlr, &ov2740_ctrl_ops,
> > +						&props);
> > +
> >  	ov2740->link_freq =
> >  		v4l2_ctrl_new_int_menu(ctrl_hdlr, &ov2740_ctrl_ops,
> >  				       V4L2_CID_LINK_FREQ,
> 

-- 
Regards,

Sakari Ailus

Re: [PATCH v9 42/46] media: ov2740: Add generic sensor fwnode properties as controls

[Laurent Pinchart]

On Tue, Apr 23, 2024 at 04:17:23PM +0000, Sakari Ailus wrote:
> On Sat, Apr 20, 2024 at 12:40:16PM +0300, Laurent Pinchart wrote:
> > On Tue, Apr 16, 2024 at 10:33:15PM +0300, Sakari Ailus wrote:
> > > Add generic sensor property information as controĺs by using
> > > v4l2_fwnode_device_parse() and v4l2_ctrl_new_fwnode_properties().
> > > 
> > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > ---
> > >  drivers/media/i2c/ov2740.c | 6 ++++++
> > >  1 file changed, 6 insertions(+)
> > > 
> > > diff --git a/drivers/media/i2c/ov2740.c b/drivers/media/i2c/ov2740.c
> > > index dc0931308053..e37d824291fe 100644
> > > --- a/drivers/media/i2c/ov2740.c
> > > +++ b/drivers/media/i2c/ov2740.c
> > > @@ -779,6 +779,8 @@ static const struct v4l2_ctrl_ops ov2740_ctrl_ops = {
> > >  
> > >  static int ov2740_init_controls(struct ov2740 *ov2740)
> > >  {
> > > +	struct i2c_client *client = v4l2_get_subdevdata(&ov2740->sd);
> > > +	struct v4l2_fwnode_device_properties props;
> > >  	struct v4l2_ctrl_handler *ctrl_hdlr;
> > >  	s64 exposure_max, h_blank, pixel_rate;
> > >  	u32 vblank_min, vblank_max, vblank_default;
> > > @@ -789,6 +791,10 @@ static int ov2740_init_controls(struct ov2740 *ov2740)
> > >  	if (ret)
> > >  		return ret;
> > >  
> > > +	if (!v4l2_fwnode_device_parse(&client->dev, &props))
> > 
> > If you moved the parsing earlier, you could set the right number of
> > controls when initializing the handler. This being said, maybe we should
> > instead try to get rid of the controls count hint to the handler
> > initialization function.
> 
> I'm not quite sure how that's related.

The move is related because you need to know if
v4l2_fwnode_device_parse() succeeded to know how many controls
v4l2_ctrl_new_fwnode_properties() will add.

> But I'll update the number.
> 
> > Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> 
> Thanks!
> 
> > > +		v4l2_ctrl_new_fwnode_properties(ctrl_hdlr, &ov2740_ctrl_ops,
> > > +						&props);
> > > +
> > >  	ov2740->link_freq =
> > >  		v4l2_ctrl_new_int_menu(ctrl_hdlr, &ov2740_ctrl_ops,
> > >  				       V4L2_CID_LINK_FREQ,

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 37/46] media: ov2740: Fix LINK_FREQ and PIXEL_RATE control value reporting

[Sakari Ailus]

Hi Laurent,

On Sat, Apr 20, 2024 at 12:42:48PM +0300, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.
> 
> On Tue, Apr 16, 2024 at 10:33:10PM +0300, Sakari Ailus wrote:
> > The driver dug the supported link frequency up from the V4L2 fwnode
> 
> s/dug/digs/
> 
> > endpoint and used it internally, but failed to report this in the
> 
> s/used/uses/
> s/failed/fails/
> 
> > LINK_FREQ and PIXEL_RATE controls. Fix this.
> > 
> > Fixes: 0677a2d9b735 ("media: ov2740: Add support for 180 MHz link frequency")
> > Cc: stable@vger.kernel.org # for v6.8 and later
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> 
> You're missing the tags given by Hans and Bingbu. As this patch is
> unrelated to the rest of the series, it should be split off and merged
> in v6.10.
> 
> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Thanks. I think I forgot the patch to this branch as well. It's been merged
already so all is well.

-- 
Regards,

Sakari Ailus