[PATCH v3 00/10] Generic line based metadata support, internal pads

[PATCH v3 00/10] Generic line based metadata support, internal pads

[Sakari Ailus]

Hi folks,

Here are a few patches to add support generic, line based metadata as well
as internal pads. While the amount of code is not very large, to the
contrary it is quite small actually IMO, I presume what this is about and
why it is being proposed requires some explaining.

Metadata mbus codes and formats have existed for some time in V4L2. They
however have been only used by drivers that produce the data itself and
effectively this metadata has always been statistics of some sort (at
least when it comes to ISPs). What is different here is that we intend to
add support for metadata originating from camera sensors.

Camera sensors produce different kinds of metadata, embedded data (usually
register address--value pairs used to capture the frame, in a more or less
sensor specific format), histograms (in a very sensor specific format),
dark pixels etc. The number of these formats is probably going to be about
as large as image data formats if not larger, as the image data formats
are much better standardised but a smaller subset of them will be
supported by V4L2, at least initially but possibly much more in the long
run.

Having this many device specific formats would be a major problem for all
the other drivers along that pipeline (not to mention the users of those
drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2
receiver drivers that have DMA: the poor driver developer would not only
need to know camera sensor specific formats but to choose the specific
packing of that format suitable for the DMA used by the hardware. It is
unlikely many of these would ever get tested while being present on the
driver API. Also adding new sensors with new embedded data formats would
involve updating all bridge and CSI-2 receiver drivers. I don't expect
this to be a workable approach.

Instead what I'm proposing is to use specific metadata formats on the
sensor devices only, on internal pads (more about those soon) of the
sensors, only visible in the UAPI, and then generic mbus formats along the
pipeline and finally generic V4L2 metadata formats on the DMAs (specific
to bit depth and packing). This would unsnarl the two, defining what data
there is (specific mbus code) and how that is transported and packed
(generic mbus codes and V4L2 formats).

The user space would be required to "know" the path of that data from the
sensor's internal pad to the V4L2 video node. I do not see this as these
devices require at least some knowledge of the pipeline, i.e. hardware at
hand. Separating what the data means and how it is packed may even be
beneficial: it allows separating code that interprets the data (sensor
internal mbus code) from the code that accesses it (packing).

These formats are in practice line based, meaning that there may be
padding at the end of the line, depending on the bus as well as the DMA.
If non-line based formats are needed, it is always possible to set the
"height" field to 1.

The internal (source) pads are an alternative to source routes [1]. The
source routes were not universally liked and I do have to say I like
re-using existing interface concepts (pads and everything you can do with
pads, including access format, selections etc.) wherever it makes sense,
instead of duplicating functionality.

Effectively internal source pads behave mostly just like sink pads, but
they describe a flow of data that originates from a sub-device instead of
arriving to a sub-device. The SUBDEV_S_ROUTING IOCTLs are used to enable
and disable routes from internal source pads to sub-device's source pads.
The subdev format IOCTLs are usable, too, so one can find which subdev
format is available on given internal source pad.

This set depends on these patches:

<URL:https://lore.kernel.org/linux-media/20230505205416.55002-1-sakari.ailus@linux.intel.com/T/#t>

I've also pushed these here and I'll keep updating the branch:

<URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=metadata>

Questions and comments are most welcome.

Driver support is to be added, as well as an example.

[1] <URL:https://lore.kernel.org/linux-media/20220831141357.1396081-20-tomi.valkeinen@ideasonboard.com/>

since v3:

- Add a better example, with formats.

- Add CCS static data media bus codes.

since v2:

- Added an example demonstrating the use of internal pads. --- Is the
  level of detail enough for the purpose?

- Improved documentation.

- Added a macro to tell whether a format is a metadata format.
  (Documentation could be added.)

- A small ReST syntax fix in the same section.

- Drop leftovers of a patch checking for the INTERNAL_SOURCE flag.

since v1:

- Make the new pad flag just "INTERNAL", requiring either SINK or SOURCE
  pad flag to accompany it. Removed the union in struct v4l2_subdev_route.

- Add the term "stream" to MC glossary.

- Improved and fixed documentation (according to comments).

- Note these formats are little endian.

- Remove 1X8 from the names of the mbus codes. These formats have generally
  8 bits per pixel.

- Fix mbus code numbering (had holes in RFC).

- Add new metadata fields to debug prints.

- Fix a minor documentation build issue.

Sakari Ailus (10):
  media: Documentation: Align numbered list
  media: mc: Check pad flag validity
  media: mc: Add INTERNAL pad flag
  media: uapi: Add generic serial metadata mbus formats
  media: uapi: Document which mbus format fields are valid for metadata
  media: uapi: Add a macro to tell whether an mbus code is metadata
  media: uapi: Add generic 8-bit metadata format definitions
  media: v4l: Support line-based metadata capture
  media: Add media bus codes for MIPI CCS embedded data
  media: uapi: v4l: Document source routes

 .../userspace-api/media/glossary.rst          |   6 +
 .../media/mediactl/media-types.rst            |   6 +
 .../userspace-api/media/v4l/dev-meta.rst      |  15 +
 .../userspace-api/media/v4l/dev-subdev.rst    | 206 ++++++++++-
 .../userspace-api/media/v4l/meta-formats.rst  |   1 +
 .../media/v4l/metafmt-generic.rst             | 331 ++++++++++++++++++
 .../media/v4l/subdev-formats.rst              | 289 +++++++++++++++
 .../media/v4l/vidioc-enum-fmt.rst             |   7 +
 .../media/videodev2.h.rst.exceptions          |   1 +
 drivers/media/mc/mc-entity.c                  |  17 +-
 drivers/media/v4l2-core/v4l2-ioctl.c          |  13 +-
 include/uapi/linux/media-bus-format.h         |  20 ++
 include/uapi/linux/media.h                    |   1 +
 include/uapi/linux/v4l2-mediabus.h            |  18 +-
 include/uapi/linux/videodev2.h                |  19 +
 15 files changed, 926 insertions(+), 24 deletions(-)
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst

-- 
2.39.2

[PATCH v3 01/10] media: Documentation: Align numbered list

[Sakari Ailus]

Align lines for numbered list so that Sphinx produces an uniform output
for all list entries.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../userspace-api/media/v4l/dev-subdev.rst    | 27 +++++++++----------
 1 file changed, 13 insertions(+), 14 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index a4f1df7093e8..b023918177b5 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -579,20 +579,19 @@ is started.
 
 There are three steps in configuring the streams:
 
-1) Set up links. Connect the pads between sub-devices using the :ref:`Media
-Controller API <media_controller>`
+1) Set up links. Connect the pads between sub-devices using the
+   :ref:`Media Controller API <media_controller>`
 
-2) Streams. Streams are declared and their routing is configured by
-setting the routing table for the sub-device using
-:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that
-setting the routing table will reset formats and selections in the
-sub-device to default values.
+2) Streams. Streams are declared and their routing is configured by setting the
+   routing table for the sub-device using :ref:`VIDIOC_SUBDEV_S_ROUTING
+   <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that setting the routing table will
+   reset formats and selections in the sub-device to default values.
 
-3) Configure formats and selections. Formats and selections of each stream
-are configured separately as documented for plain sub-devices in
-:ref:`format-propagation`. The stream ID is set to the same stream ID
-associated with either sink or source pads of routes configured using the
-:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl.
+3) Configure formats and selections. Formats and selections of each stream are
+   configured separately as documented for plain sub-devices in
+   :ref:`format-propagation`. The stream ID is set to the same stream ID
+   associated with either sink or source pads of routes configured using the
+   :ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl.
 
 Multiplexed streams setup example
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -619,8 +618,8 @@ modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
 To configure this pipeline, the userspace must take the following steps:
 
 1) Set up media links between entities: connect the sensors to the bridge,
-bridge to the receiver, and the receiver to the DMA engines. This step does
-not differ from normal non-multiplexed media controller setup.
+   bridge to the receiver, and the receiver to the DMA engines. This step does
+   not differ from normal non-multiplexed media controller setup.
 
 2) Configure routing
 
-- 
2.39.2

[PATCH v3 02/10] media: mc: Check pad flag validity

[Sakari Ailus]

Check the validity of pad flags on entity init. Exactly one of the flags
must be set.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/mc/mc-entity.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 83468d4a440b..4991281dcccc 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -197,6 +197,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 	struct media_device *mdev = entity->graph_obj.mdev;
 	struct media_pad *iter;
 	unsigned int i = 0;
+	int ret = 0;
 
 	if (num_pads >= MEDIA_ENTITY_MAX_PADS)
 		return -E2BIG;
@@ -210,6 +211,14 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 	media_entity_for_each_pad(entity, iter) {
 		iter->entity = entity;
 		iter->index = i++;
+
+		if (hweight32(iter->flags & (MEDIA_PAD_FL_SINK |
+					     MEDIA_PAD_FL_SOURCE))
+		    != 1) {
+			ret = -EINVAL;
+			break;
+		}
+
 		if (mdev)
 			media_gobj_create(mdev, MEDIA_GRAPH_PAD,
 					  &iter->graph_obj);
@@ -218,7 +227,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 	if (mdev)
 		mutex_unlock(&mdev->graph_mutex);
 
-	return 0;
+	return ret;
 }
 EXPORT_SYMBOL_GPL(media_entity_pads_init);
 
-- 
2.39.2

[PATCH v3 03/10] media: mc: Add INTERNAL pad flag

[Sakari Ailus]

Internal source pads will be used as routing endpoints in V4L2
[GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.

Also prevent creating links to pads that have been flagged as internal.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 Documentation/userspace-api/media/glossary.rst             | 6 ++++++
 Documentation/userspace-api/media/mediactl/media-types.rst | 6 ++++++
 drivers/media/mc/mc-entity.c                               | 6 +++++-
 include/uapi/linux/media.h                                 | 1 +
 4 files changed, 18 insertions(+), 1 deletion(-)

diff --git a/Documentation/userspace-api/media/glossary.rst b/Documentation/userspace-api/media/glossary.rst
index 96a360edbf3b..f7b99a4527c7 100644
--- a/Documentation/userspace-api/media/glossary.rst
+++ b/Documentation/userspace-api/media/glossary.rst
@@ -173,6 +173,12 @@ Glossary
 	An integrated circuit that integrates all components of a computer
 	or other electronic systems.
 
+_media-glossary-stream:
+    Stream
+	A distinct flow of data (image data or metadata) over a media pipeline
+	from source to sink. A source may be e.g. an image sensor and a sink
+	e.g. a memory buffer.
+
     V4L2 API
 	**V4L2 userspace API**
 
diff --git a/Documentation/userspace-api/media/mediactl/media-types.rst b/Documentation/userspace-api/media/mediactl/media-types.rst
index 0ffeece1e0c8..28941da27790 100644
--- a/Documentation/userspace-api/media/mediactl/media-types.rst
+++ b/Documentation/userspace-api/media/mediactl/media-types.rst
@@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
 .. _MEDIA-PAD-FL-SINK:
 .. _MEDIA-PAD-FL-SOURCE:
 .. _MEDIA-PAD-FL-MUST-CONNECT:
+.. _MEDIA-PAD-FL-INTERNAL:
 
 .. flat-table:: Media pad flags
     :header-rows:  0
@@ -382,6 +383,11 @@ Types and flags used to represent the media graph elements
 	  when this flag isn't set; the absence of the flag doesn't imply
 	  there is none.
 
+    *  -  ``MEDIA_PAD_FL_INTERNAL``
+       -  The internal flag indicates an internal pad that has no external
+	  connections. Such a pad shall not be connected with a link. The
+	  internal flag indicates that the :ref:``stream
+	  <media-glossary-stream>`` either starts or ends in the entity.
 
 One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
 must be set for every pad.
diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 4991281dcccc..03a9e0b8ebab 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -1071,7 +1071,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
 
 	for (i = 0; i < entity->num_pads; i++) {
 		if ((entity->pads[i].flags &
-		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
+		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
+		      MEDIA_PAD_FL_INTERNAL)) != pad_type)
 			continue;
 
 		if (entity->pads[i].sig_type == sig_type)
@@ -1094,6 +1095,9 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
 		return -EINVAL;
 	if (WARN_ON(!(source->pads[source_pad].flags & MEDIA_PAD_FL_SOURCE)))
 		return -EINVAL;
+	if (WARN_ON(source->pads[source_pad].flags & MEDIA_PAD_FL_SOURCE &&
+		    source->pads[source_pad].flags & MEDIA_PAD_FL_INTERNAL))
+		return -EINVAL;
 	if (WARN_ON(!(sink->pads[sink_pad].flags & MEDIA_PAD_FL_SINK)))
 		return -EINVAL;
 
diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
index 1c80b1d6bbaf..80cfd12a43fc 100644
--- a/include/uapi/linux/media.h
+++ b/include/uapi/linux/media.h
@@ -208,6 +208,7 @@ struct media_entity_desc {
 #define MEDIA_PAD_FL_SINK			(1U << 0)
 #define MEDIA_PAD_FL_SOURCE			(1U << 1)
 #define MEDIA_PAD_FL_MUST_CONNECT		(1U << 2)
+#define MEDIA_PAD_FL_INTERNAL			(1U << 3)
 
 struct media_pad_desc {
 	__u32 entity;		/* entity ID */
-- 
2.39.2

[PATCH v3 04/10] media: uapi: Add generic serial metadata mbus formats

[Sakari Ailus]

Add generic serial metadata mbus formats. These formats describe data
width and packing but not the content itself. The reason for specifying
such formats is that the formats as such are fairly device specific but
they are still handled by CSI-2 receiver drivers that should not be aware
of device specific formats. What makes generic metadata formats possible
is that these formats are parsed by software only, after capturing the
data to system memory.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../media/v4l/subdev-formats.rst              | 257 ++++++++++++++++++
 include/uapi/linux/media-bus-format.h         |   9 +
 2 files changed, 266 insertions(+)

diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
index a3a35eeed708..c615da08502d 100644
--- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
+++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
@@ -8234,3 +8234,260 @@ The following table lists the existing metadata formats.
 	both sides of the link and the bus format is a fixed
 	metadata format that is not configurable from userspace.
 	Width and height will be set to 0 for this format.
+
+Generic Serial Metadata Formats
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Generic serial metadata formats are used on serial busses where the actual data
+content is more or less device specific but the data is transmitted and received
+by multiple devices that do not process the data in any way, simply writing
+it to system memory for processing in software at the end of the pipeline.
+
+The more specific variant describing the actual data is used on the internal
+source pad of the originating sub-device.
+
+"b" in an array cell signifies a byte of data, followed by the number of byte
+and finally the bit number in subscript. "p" indicates a padding bit.
+
+.. _media-bus-format-generic-meta:
+
+.. cssclass: longtable
+
+.. flat-table:: Generic Serial Metadata Formats
+    :header-rows:  2
+    :stub-columns: 0
+
+    * - Identifier
+      - Code
+      -
+      - :cspan:`23` Data organization
+    * -
+      -
+      - Bit
+      - 23
+      - 22
+      - 21
+      - 20
+      - 19
+      - 18
+      - 17
+      - 16
+      - 15
+      - 14
+      - 13
+      - 12
+      - 11
+      - 10
+      - 9
+      - 8
+      - 7
+      - 6
+      - 5
+      - 4
+      - 3
+      - 2
+      - 1
+      - 0
+    * .. _MEDIA-BUS-FMT-META-8:
+
+      - MEDIA_BUS_FMT_META_8
+      - 0x8001
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+    * .. _MEDIA-BUS-FMT-META-10:
+
+      - MEDIA_BUS_FMT_META_10
+      - 0x8002
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-12:
+
+      - MEDIA_BUS_FMT_META_12
+      - 0x8003
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-14:
+
+      - MEDIA_BUS_FMT_META_14
+      - 0x8004
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-16:
+
+      - MEDIA_BUS_FMT_META_16
+      - 0x8005
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-20:
+
+      - MEDIA_BUS_FMT_META_20
+      - 0x8007
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-24:
+
+      - MEDIA_BUS_FMT_META_24
+      - 0x8009
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
index a03c543cb072..9ee031397372 100644
--- a/include/uapi/linux/media-bus-format.h
+++ b/include/uapi/linux/media-bus-format.h
@@ -173,4 +173,13 @@
  */
 #define MEDIA_BUS_FMT_METADATA_FIXED		0x7001
 
+/* Generic line based metadata formats for serial buses. Next is 0x8008. */
+#define MEDIA_BUS_FMT_META_8			0x8001
+#define MEDIA_BUS_FMT_META_10			0x8002
+#define MEDIA_BUS_FMT_META_12			0x8003
+#define MEDIA_BUS_FMT_META_14			0x8004
+#define MEDIA_BUS_FMT_META_16			0x8005
+#define MEDIA_BUS_FMT_META_20			0x8006
+#define MEDIA_BUS_FMT_META_24			0x8007
+
 #endif /* __LINUX_MEDIA_BUS_FORMAT_H */
-- 
2.39.2

[PATCH v3 05/10] media: uapi: Document which mbus format fields are valid for metadata

[Sakari Ailus]

Now that metadata mbus formats have been added, it is necessary to define
which fields in struct v4l2_mbus_format are applicable to them (not many).

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 include/uapi/linux/v4l2-mediabus.h | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)

diff --git a/include/uapi/linux/v4l2-mediabus.h b/include/uapi/linux/v4l2-mediabus.h
index 6b07b73473b5..3cadb3b58b85 100644
--- a/include/uapi/linux/v4l2-mediabus.h
+++ b/include/uapi/linux/v4l2-mediabus.h
@@ -19,12 +19,18 @@
  * @width:	image width
  * @height:	image height
  * @code:	data format code (from enum v4l2_mbus_pixelcode)
- * @field:	used interlacing type (from enum v4l2_field)
- * @colorspace:	colorspace of the data (from enum v4l2_colorspace)
- * @ycbcr_enc:	YCbCr encoding of the data (from enum v4l2_ycbcr_encoding)
- * @hsv_enc:	HSV encoding of the data (from enum v4l2_hsv_encoding)
- * @quantization: quantization of the data (from enum v4l2_quantization)
- * @xfer_func:  transfer function of the data (from enum v4l2_xfer_func)
+ * @field:	used interlacing type (from enum v4l2_field), not applicable
+ *		to metadata mbus codes
+ * @colorspace:	colorspace of the data (from enum v4l2_colorspace), zero on
+ *		metadata mbus codes
+ * @ycbcr_enc:	YCbCr encoding of the data (from enum v4l2_ycbcr_encoding), zero
+ *		on metadata mbus codes
+ * @hsv_enc:	HSV encoding of the data (from enum v4l2_hsv_encoding), zero on
+ *		metadata mbus codes
+ * @quantization: quantization of the data (from enum v4l2_quantization), zero
+ *		on metadata mbus codes
+ * @xfer_func:  transfer function of the data (from enum v4l2_xfer_func), zero
+ *		on metadata mbus codes
  * @flags:	flags (V4L2_MBUS_FRAMEFMT_*)
  * @reserved:  reserved bytes that can be later used
  */
-- 
2.39.2

[PATCH v3 06/10] media: uapi: Add a macro to tell whether an mbus code is metadata

[Sakari Ailus]

Add a macro to tell whether a given mbus code is metadata.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 include/uapi/linux/media-bus-format.h | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
index 9ee031397372..2486b4178c5f 100644
--- a/include/uapi/linux/media-bus-format.h
+++ b/include/uapi/linux/media-bus-format.h
@@ -182,4 +182,7 @@
 #define MEDIA_BUS_FMT_META_20			0x8006
 #define MEDIA_BUS_FMT_META_24			0x8007
 
+#define MEDIA_BUS_FMT_IS_META(code)		\
+	((code) & 0xf000 == 0x7000 || (code) & 0xf000 == 0x8000)
+
 #endif /* __LINUX_MEDIA_BUS_FORMAT_H */
-- 
2.39.2

[PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

[Sakari Ailus]

Generic 8-bit metadata formats define the in-memory data layout but not
the format of the data itself. The reasoning for having such formats is to
allow CSI-2 receiver drivers to receive and DMA drivers to write the data
to memory without knowing a large number of device specific formats.

These formats may be used only in conjunction of a Media controller
pipeline where the internal pad of the source sub-device defines the
specific format of the data (using an mbus code).

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../userspace-api/media/v4l/meta-formats.rst  |   1 +
 .../media/v4l/metafmt-generic.rst             | 331 ++++++++++++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
 include/uapi/linux/videodev2.h                |   9 +
 4 files changed, 349 insertions(+)
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst

diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
index 0bb61fc5bc00..919f595576b9 100644
--- a/Documentation/userspace-api/media/v4l/meta-formats.rst
+++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
@@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only.
     metafmt-vsp1-hgo
     metafmt-vsp1-hgt
     metafmt-vivid
+    metafmt-generic
diff --git a/Documentation/userspace-api/media/v4l/metafmt-generic.rst b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
new file mode 100644
index 000000000000..a27bfc721edf
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
@@ -0,0 +1,331 @@
+.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
+
+**************************************************************************************************************************************************************************************************************************************************************************************************************************
+V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O'), V4L2_META_FMT_GENERIC_CSI2_2_24 ('MC2O')
+**************************************************************************************************************************************************************************************************************************************************************************************************************************
+
+
+Generic line-based metadata formats
+
+
+Description
+===========
+
+These generic line-based metadata formats define the memory layout of the data
+without defining the format or meaning of the metadata itself. These formats may
+only be used with a Media controller pipeline where the more specific format is
+defined in an :ref:`internal source pad <MEDIA-PAD-FL-INTERNAL>` of the source
+sub-device. See also :ref:`source routes <v4l2-subdev-source-routes>`.
+
+.. _v4l2-meta-fmt-generic-8:
+
+V4L2_META_FMT_GENERIC_8
+-----------------------
+
+The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format.
+
+This format is also used on CSI-2 on both 8 bits per sample as well as on
+16 bits per sample when two bytes of metadata are packed into one sample.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_8.**
+Each cell is one byte. "M" denotes a byte of metadata.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+    * - start + 4:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+
+.. _v4l2-meta-fmt-generic-csi2-10:
+
+V4L2_META_FMT_GENERIC_CSI2_10
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
+for each 8 bits of data. Every four bytes of metadata is followed by a single
+byte of padding. The way the data is stored follows the CSI-2 specification.
+
+This format is also used on CSI-2 on 20 bits per sample format that packs two
+bytes of metadata into one sample.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+    * - start + 5:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-12:
+
+V4L2_META_FMT_GENERIC_CSI2_12
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_12 contains packed 8-bit generic metadata, 12 bits
+for each 8 bits of data. Every four bytes of metadata is followed by two bytes
+of padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_12.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+      - p
+    * - start + 6:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-14:
+
+V4L2_META_FMT_GENERIC_CSI2_14
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_14 contains packed 8-bit generic metadata, 14 bits
+for each 8 bits of data. Every four bytes of metadata is followed by three
+bytes of padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_14.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+      - p
+      - p
+    * - start + 7:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-16:
+
+V4L2_META_FMT_GENERIC_CSI2_16
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_16 contains packed 8-bit generic metadata, 16 bits
+for each 8 bits of data. Every byte of metadata is followed by one byte of
+padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_16.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - p
+      - M\ :sub:`10`
+      - p
+      - M\ :sub:`20`
+      - p
+      - M\ :sub:`30`
+      - p
+    * - start + 8:
+      - M\ :sub:`01`
+      - p
+      - M\ :sub:`11`
+      - p
+      - M\ :sub:`21`
+      - p
+      - M\ :sub:`31`
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-20:
+
+V4L2_META_FMT_GENERIC_CSI2_20
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_20 contains packed 8-bit generic metadata, 20 bits
+for each 8 bits of data. Every byte of metadata is followed by alternating one
+and two bytes of padding. The way the data is stored follows the CSI-2
+specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_20.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - p
+      - M\ :sub:`10`
+      - p
+      - p
+      - M\ :sub:`20`
+      - p
+      - M\ :sub:`30`
+      - p
+      - p
+    * - start + 10:
+      - M\ :sub:`01`
+      - p
+      - M\ :sub:`11`
+      - p
+      - p
+      - M\ :sub:`21`
+      - p
+      - M\ :sub:`31`
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-24:
+
+V4L2_META_FMT_GENERIC_CSI2_24
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_24 contains packed 8-bit generic metadata, 24 bits
+for each 8 bits of data. Every byte of metadata is followed by two bytes of
+padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_24.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - p
+      - p
+      - M\ :sub:`10`
+      - p
+      - p
+      - M\ :sub:`20`
+      - p
+      - p
+      - M\ :sub:`30`
+      - p
+      - p
+    * - start + 12:
+      - M\ :sub:`01`
+      - p
+      - p
+      - M\ :sub:`11`
+      - p
+      - p
+      - M\ :sub:`21`
+      - p
+      - p
+      - M\ :sub:`31`
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-2-24:
+
+V4L2_META_FMT_GENERIC_CSI2_2_24
+-------------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_2_24 contains packed 8-bit generic metadata, 24 bits
+for each two times 8 bits of data. Every two bytes of metadata are followed by
+one byte of padding. The way the data is stored follows the CSI-2
+specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_2_24.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - p
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+    * - start + 6:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - p
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index f4d9d6279094..fbbddc333a30 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -1452,6 +1452,14 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
 	case V4L2_PIX_FMT_Y210:		descr = "10-bit YUYV Packed"; break;
 	case V4L2_PIX_FMT_Y212:		descr = "12-bit YUYV Packed"; break;
 	case V4L2_PIX_FMT_Y216:		descr = "16-bit YUYV Packed"; break;
+	case V4L2_META_FMT_GENERIC_8:	descr = "8-bit Generic Metadata"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_10:	descr = "8b Generic Meta, 10b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_12:	descr = "8b Generic Meta, 12b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_14:	descr = "8b Generic Meta, 14b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_16:	descr = "8b Generic Meta, 16b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_20:	descr = "8b Generic Meta, 20b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_24:	descr = "8b Generic Meta, 24b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_2_24:	descr = "2x8b Generic Meta, 24b CSI-2"; break;
 
 	default:
 		/* Compressed formats */
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 78260e5d9985..b4284a564025 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -838,6 +838,15 @@ struct v4l2_pix_format {
 #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
 #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
 
+#define V4L2_META_FMT_GENERIC_8		v4l2_fourcc('M', 'E', 'T', '8') /* Generic 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_10	v4l2_fourcc('M', 'C', '1', 'A') /* 10-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_12	v4l2_fourcc('M', 'C', '1', 'C') /* 12-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_14	v4l2_fourcc('M', 'C', '1', 'E') /* 14-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_16	v4l2_fourcc('M', 'C', '1', 'G') /* 16-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_20	v4l2_fourcc('M', 'C', '1', 'K') /* 20-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_24	v4l2_fourcc('M', 'C', '1', 'O') /* 24-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_2_24	v4l2_fourcc('M', 'C', '2', 'O') /* 2 bytes of 8-bit metadata, 24-bit CSI-2 packed */
+
 /* priv field value to indicates that subsequent fields are valid. */
 #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
 
-- 
2.39.2

[PATCH v3 08/10] media: v4l: Support line-based metadata capture

[Sakari Ailus]

many camera sensors, among other devices, transmit embedded data and image
data for each CSI-2 frame. This embedded data typically contains register
configuration of the sensor that has been used to capture the image data
of the same frame.

The embedded data is received by the CSI-2 receiver and has the same
properties as the image data, including that it is line based: it has
width, height and bytesperline (stride).

Add these fields to struct v4l2_meta_format and document them.

Also add V4L2_FMT_FLAG_META_LINE_BASED to tell a given format is
line-based i.e. these fields of struct v4l2_meta_format are valid for it.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../userspace-api/media/v4l/dev-meta.rst          | 15 +++++++++++++++
 .../userspace-api/media/v4l/vidioc-enum-fmt.rst   |  7 +++++++
 .../media/videodev2.h.rst.exceptions              |  1 +
 drivers/media/v4l2-core/v4l2-ioctl.c              |  5 +++--
 include/uapi/linux/videodev2.h                    | 10 ++++++++++
 5 files changed, 36 insertions(+), 2 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/dev-meta.rst b/Documentation/userspace-api/media/v4l/dev-meta.rst
index 0e7e1ee1471a..4b24bae6e171 100644
--- a/Documentation/userspace-api/media/v4l/dev-meta.rst
+++ b/Documentation/userspace-api/media/v4l/dev-meta.rst
@@ -65,3 +65,18 @@ to 0.
       - ``buffersize``
       - Maximum buffer size in bytes required for data. The value is set by the
         driver.
+    * - __u32
+      - ``width``
+      - Width of a line of metadata in samples. Valid when :c:type`v4l2_fmtdesc`
+	flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
+	:c:func:`VIDIOC_ENUM_FMT`.
+    * - __u32
+      - ``height``
+      - Number of rows of metadata. Valid when :c:type`v4l2_fmtdesc` flag
+	``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
+	:c:func:`VIDIOC_ENUM_FMT`.
+    * - __u32
+      - ``bytesperline``
+      - Offset in bytes between the beginning of two consecutive lines. Valid
+	when :c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is
+	set, otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
diff --git a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
index 000c154b0f98..6d7664345a4e 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
@@ -227,6 +227,13 @@ the ``mbus_code`` field is handled differently:
 	The application can ask to configure the quantization of the capture
 	device when calling the :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl with
 	:ref:`V4L2_PIX_FMT_FLAG_SET_CSC <v4l2-pix-fmt-flag-set-csc>` set.
+    * - ``V4L2_FMT_FLAG_META_LINE_BASED``
+      - 0x0200
+      - The metadata format is line-based. In this case the ``width``,
+	``height`` and ``bytesperline`` fields of :c:type:`v4l2_meta_format` are
+	valid. The buffer consists of ``height`` lines, each having ``width``
+	bytes of data and offset between the beginning of each two consecutive
+	lines is ``bytesperline``.
 
 Return Value
 ============
diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b/Documentation/userspace-api/media/videodev2.h.rst.exceptions
index 3e58aac4ef0b..bdc628e8c1d6 100644
--- a/Documentation/userspace-api/media/videodev2.h.rst.exceptions
+++ b/Documentation/userspace-api/media/videodev2.h.rst.exceptions
@@ -215,6 +215,7 @@ replace define V4L2_FMT_FLAG_CSC_XFER_FUNC fmtdesc-flags
 replace define V4L2_FMT_FLAG_CSC_YCBCR_ENC fmtdesc-flags
 replace define V4L2_FMT_FLAG_CSC_HSV_ENC fmtdesc-flags
 replace define V4L2_FMT_FLAG_CSC_QUANTIZATION fmtdesc-flags
+replace define V4L2_FMT_FLAG_META_LINE_BASED fmtdesc-flags
 
 # V4L2 timecode types
 replace define V4L2_TC_TYPE_24FPS timecode-type
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index fbbddc333a30..971d784e7429 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -343,8 +343,9 @@ static void v4l_print_format(const void *arg, bool write_only)
 	case V4L2_BUF_TYPE_META_OUTPUT:
 		meta = &p->fmt.meta;
 		pixelformat = meta->dataformat;
-		pr_cont(", dataformat=%p4cc, buffersize=%u\n",
-			&pixelformat, meta->buffersize);
+		pr_cont(", dataformat=%p4cc, buffersize=%u, width=%u, height=%u, bytesperline=%u\n",
+			&pixelformat, meta->buffersize, meta->width,
+			meta->height, meta->bytesperline);
 		break;
 	}
 }
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index b4284a564025..d26c0650c6a7 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -877,6 +877,7 @@ struct v4l2_fmtdesc {
 #define V4L2_FMT_FLAG_CSC_YCBCR_ENC		0x0080
 #define V4L2_FMT_FLAG_CSC_HSV_ENC		V4L2_FMT_FLAG_CSC_YCBCR_ENC
 #define V4L2_FMT_FLAG_CSC_QUANTIZATION		0x0100
+#define V4L2_FMT_FLAG_META_LINE_BASED		0x0200
 
 	/* Frame Size and frame rate enumeration */
 /*
@@ -2420,10 +2421,19 @@ struct v4l2_sdr_format {
  * struct v4l2_meta_format - metadata format definition
  * @dataformat:		little endian four character code (fourcc)
  * @buffersize:		maximum size in bytes required for data
+ * @width:		number of bytes of data per line (valid for line based
+ *			formats only, see format documentation)
+ * @height:		number of lines of data per buffer (valid for line based
+ *			formats only)
+ * @bytesperline:	offset between the beginnings of two adjacent lines in
+ *			bytes (valid for line based formats only)
  */
 struct v4l2_meta_format {
 	__u32				dataformat;
 	__u32				buffersize;
+	__u32				width;
+	__u32				height;
+	__u32				bytesperline;
 } __attribute__ ((packed));
 
 /**
-- 
2.39.2

[PATCH v3 09/10] media: Add media bus codes for MIPI CCS embedded data

[Sakari Ailus]

Add new MIPI CCS embedded data media bus formats.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../media/v4l/subdev-formats.rst              | 32 +++++++++++++++++++
 include/uapi/linux/media-bus-format.h         | 10 +++++-
 2 files changed, 41 insertions(+), 1 deletion(-)

diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
index c615da08502d..5d5407738af9 100644
--- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
+++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
@@ -8491,3 +8491,35 @@ and finally the bit number in subscript. "p" indicates a padding bit.
       - p
       - p
       - p
+
+MIPI CCS Embedded Data Formats
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+`MIPI CCS <https://www.mipi.org/specifications/camera-command-set>`_ defines an
+metadata format for sensor embedded data, which is used to store the register
+configuration used for capturing a given frame. The format is defined in the CCS
+specification.
+
+The bit depth of the CCS embedded data matches the pixel data bit depth
+configured on the sensor. The formats used and their corresponding generic
+formats are listed in the table below.
+
+.. flat-table: CCS embedded data mbus formats and corresponding generic formats
+    :header-rows: 1
+
+    * - CCS embedded data mbus format
+      - Generic metadata format
+    * - MEDIA_BUS_FMT_CCS_EMBEDDED_8
+      - MEDIA_BUS_FMT_META_8
+    * - MEDIA_BUS_FMT_CCS_EMBEDDED_10
+      - MEDIA_BUS_FMT_META_10
+    * - MEDIA_BUS_FMT_CCS_EMBEDDED_12
+      - MEDIA_BUS_FMT_META_12
+    * - MEDIA_BUS_FMT_CCS_EMBEDDED_14
+      - MEDIA_BUS_FMT_META_14
+    * - MEDIA_BUS_FMT_CCS_EMBEDDED_16
+      - MEDIA_BUS_FMT_META_16
+    * - MEDIA_BUS_FMT_CCS_EMBEDDED_20
+      - MEDIA_BUS_FMT_META_20
+    * - MEDIA_BUS_FMT_CCS_EMBEDDED_24
+      - MEDIA_BUS_FMT_META_24
diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
index 2486b4178c5f..31d532446b23 100644
--- a/include/uapi/linux/media-bus-format.h
+++ b/include/uapi/linux/media-bus-format.h
@@ -182,7 +182,15 @@
 #define MEDIA_BUS_FMT_META_20			0x8006
 #define MEDIA_BUS_FMT_META_24			0x8007
 
+/* Specific metadata formats. Next is 0x9008. */
+#define MEDIA_BUS_FMT_CCS_EMBEDDED_8		0x9001
+#define MEDIA_BUS_FMT_CCS_EMBEDDED_10		0x9002
+#define MEDIA_BUS_FMT_CCS_EMBEDDED_12		0x9003
+#define MEDIA_BUS_FMT_CCS_EMBEDDED_14		0x9004
+#define MEDIA_BUS_FMT_CCS_EMBEDDED_16		0x9005
+#define MEDIA_BUS_FMT_CCS_EMBEDDED_20		0x9006
+#define MEDIA_BUS_FMT_CCS_EMBEDDED_24		0x9007
+
 #define MEDIA_BUS_FMT_IS_META(code)		\
 	((code) & 0xf000 == 0x7000 || (code) & 0xf000 == 0x8000)
-
 #endif /* __LINUX_MEDIA_BUS_FORMAT_H */
-- 
2.39.2

[PATCH v3 10/10] media: uapi: v4l: Document source routes

[Sakari Ailus]

Document how internal pads are used on source routes.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../userspace-api/media/v4l/dev-subdev.rst    | 179 ++++++++++++++++++
 1 file changed, 179 insertions(+)

diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index b023918177b5..27b0fe2dc83a 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -551,6 +551,27 @@ A stream at a specific point in the media pipeline is identified by the
 sub-device and a (pad, stream) pair. For sub-devices that do not support
 multiplexed streams the 'stream' field is always 0.
 
+.. _v4l2-subdev-source-routes:
+
+Internal pads and source routes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Cases where a single sub-device source pad is traversed by multiple streams one
+or more of which originate from within the sub-device itself are special as
+there is no external sink pad for such routes. In those cases, the sources of
+the internally generated streams are represented by internal sink pads, which
+are sink pads that have the :ref:`MEDIA_PAD_FL_INTERNAL <MEDIA-PAD-FL-INTERNAL>`
+pad flag set.
+
+Internal pads have all the properties of an external pad, including formats and
+selections. The format in this case is the source format of the stream. An
+internal pad always has a single stream only (0).
+
+/Source routes/ are routes from an internal sink pad to a(n external) source
+pad. Generally source routes are not modifiable but they can be activated and
+deactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE
+<v4l2-subdev-routing-flags>` flag, depending on driver capabilities.
+
 Interaction between routes, streams, formats and selections
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -666,3 +687,161 @@ A common way to accomplish this is to start from the sensors and propagate the
 configurations along the stream towards the receiver,
 using :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each
 stream endpoint in each sub-device.
+
+Internal pads setup example
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A simple example of a multiplexed stream setup might be as follows:
+
+- A CCS camera sensor source sub-device, with one sink pad (0), one source pad
+  (1), an internal sink pad (2) that represents the source of embedded
+  data. There are two routes, one from the sink pad to the source, and another
+  from the internal sink pad to the source pad. The embedded data stream needs
+  to be enabled by activating the related route. The configuration of the rest
+  of the CCS sub-devices is omitted from this example.
+
+- Multiplexer bridge (Bridge). The bridge has one sink pad, connected to the
+  sensor (pad 0), and one source pad (pad 1), which outputs two streams.
+
+- Receiver in the SoC (Receiver). The receiver has a single sink pad (pad 0),
+  connected to the bridge, and two source pads (pads 1-2), going to the DMA
+  engine. The receiver demultiplexes the incoming streams to the source pads.
+
+- DMA Engines in the SoC (DMA Engine), one for each stream. Each DMA engine is
+  connected to a single source pad in the receiver.
+
+The sensor, the bridge and the receiver are modeled as V4L2 sub-devices,
+exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are
+modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
+
+To configure this pipeline, the userspace must take the following steps:
+
+1) Set up media links between entities: connect the sensors to the bridge,
+   bridge to the receiver, and the receiver to the DMA engines. This step does
+   not differ from normal non-multiplexed media controller setup.
+
+2) Configure routing
+
+.. flat-table:: Camera sensor
+    :header-rows: 1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0
+      - 1/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from the sink pad
+    * - 2/0
+      - 1/1
+      - **V4L2_SUBDEV_ROUTE_FL_ACTIVE**
+      - Metadata stream from the internal sink pad
+
+.. flat-table:: Bridge routing table
+    :header-rows: 1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0
+      - 1/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from camera sensor
+    * - 0/1
+      - 1/1
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from camera sensor
+
+.. flat-table:: Receiver routing table
+    :header-rows:  1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0
+      - 1/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from camera sensor
+    * - 0/1
+      - 2/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from camera sensor
+
+The options available in sensor's routing configuration are dictated by hardware
+capabilities: typically camera sensors always produce image data stream while
+the embedded data stream typically can be either enabled or disabled.
+
+3) Configure formats and selections
+
+This example assumes that the formats are propagated from sink pad to the source
+pad as-is. The tables contain fields of both struct v4l2_subdev_format and
+struct v4l2_mbus_framefmt. The full configuration of CCS camera sensor is out of
+scope of this example.
+
+.. flat-table:: Formats set on the sub-devices. Bold values are set, others are
+                static or propagated.
+    :header-rows: 1
+    :fill-cells:
+
+    * - Sub-device
+      - Pad/Stream
+      - Width
+      - Height
+      - Code
+    * - :rspan:`3` Camera sensor sub-device (CCS source sub-device)
+      - 0/0
+      - 640
+      - 480
+      - MEDIA_BUS_FMT_SGRBG10
+    * - 1/0
+      - 640
+      - 480
+      - **MEDIA_BUS_FMT_SGRBG10**
+    * - 2/0
+      - 640
+      - 2
+      - MEDIA_BUS_FMT_CCS_EMBEDDED_10
+    * - 1/1
+      - 640
+      - 2
+      - MEDIA_BUS_FMT_META_10
+    * - :rspan:`3` Bridge
+      - 0/0
+      - **640**
+      - **480**
+      - **MEDIA_BUS_FMT_SGRBG10**
+    * - 1/0
+      - 640
+      - 480
+      - MEDIA_BUS_FMT_SGRBG10
+    * - 0/1
+      - **640**
+      - **2**
+      - **MEDIA_BUS_FMT_META_10**
+    * - 1/1
+      - 640
+      - 2
+      - MEDIA_BUS_FMT_META_10
+    * - :rspan:`3` Receiver
+      - 0/0
+      - **640**
+      - **480**
+      - **MEDIA_BUS_FMT_SGRBG10**
+    * - 1/0
+      - 640
+      - 480
+      - MEDIA_BUS_FMT_SGRBG10
+    * - 0/1
+      - **640**
+      - **2**
+      - **MEDIA_BUS_FMT_META_10**
+    * - 2/0
+      - 640
+      - 2
+      - MEDIA_BUS_FMT_META_10
+
+The embedded data format does not need to be configured as the format is
+dictated by the pixel data format in this case.
-- 
2.39.2

Re: [PATCH v3 06/10] media: uapi: Add a macro to tell whether an mbus code is metadata

[Hans Verkuil]

On 08/08/2023 09:55, Sakari Ailus wrote:
> Add a macro to tell whether a given mbus code is metadata.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  include/uapi/linux/media-bus-format.h | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
> index 9ee031397372..2486b4178c5f 100644
> --- a/include/uapi/linux/media-bus-format.h
> +++ b/include/uapi/linux/media-bus-format.h
> @@ -182,4 +182,7 @@
>  #define MEDIA_BUS_FMT_META_20			0x8006
>  #define MEDIA_BUS_FMT_META_24			0x8007
>  
> +#define MEDIA_BUS_FMT_IS_META(code)		\
> +	((code) & 0xf000 == 0x7000 || (code) & 0xf000 == 0x8000)

I don't think this is right: == has a higher precedence than &, so this
actually reads:

	code & (0xf000 == 0x7000)

Regards,

	Hans

> +
>  #endif /* __LINUX_MEDIA_BUS_FORMAT_H */

Re: [PATCH v3 03/10] media: mc: Add INTERNAL pad flag

[Hans Verkuil]

On 08/08/2023 09:55, Sakari Ailus wrote:
> Internal source pads will be used as routing endpoints in V4L2
> [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
> 
> Also prevent creating links to pads that have been flagged as internal.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  Documentation/userspace-api/media/glossary.rst             | 6 ++++++
>  Documentation/userspace-api/media/mediactl/media-types.rst | 6 ++++++
>  drivers/media/mc/mc-entity.c                               | 6 +++++-
>  include/uapi/linux/media.h                                 | 1 +
>  4 files changed, 18 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/userspace-api/media/glossary.rst b/Documentation/userspace-api/media/glossary.rst
> index 96a360edbf3b..f7b99a4527c7 100644
> --- a/Documentation/userspace-api/media/glossary.rst
> +++ b/Documentation/userspace-api/media/glossary.rst
> @@ -173,6 +173,12 @@ Glossary
>  	An integrated circuit that integrates all components of a computer
>  	or other electronic systems.
>  
> +_media-glossary-stream:
> +    Stream
> +	A distinct flow of data (image data or metadata) over a media pipeline
> +	from source to sink. A source may be e.g. an image sensor and a sink
> +	e.g. a memory buffer.
> +
>      V4L2 API
>  	**V4L2 userspace API**
>  
> diff --git a/Documentation/userspace-api/media/mediactl/media-types.rst b/Documentation/userspace-api/media/mediactl/media-types.rst
> index 0ffeece1e0c8..28941da27790 100644
> --- a/Documentation/userspace-api/media/mediactl/media-types.rst
> +++ b/Documentation/userspace-api/media/mediactl/media-types.rst
> @@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
>  .. _MEDIA-PAD-FL-SINK:
>  .. _MEDIA-PAD-FL-SOURCE:
>  .. _MEDIA-PAD-FL-MUST-CONNECT:
> +.. _MEDIA-PAD-FL-INTERNAL:
>  
>  .. flat-table:: Media pad flags
>      :header-rows:  0
> @@ -382,6 +383,11 @@ Types and flags used to represent the media graph elements
>  	  when this flag isn't set; the absence of the flag doesn't imply
>  	  there is none.
>  
> +    *  -  ``MEDIA_PAD_FL_INTERNAL``
> +       -  The internal flag indicates an internal pad that has no external
> +	  connections. Such a pad shall not be connected with a link. The
> +	  internal flag indicates that the :ref:``stream
> +	  <media-glossary-stream>`` either starts or ends in the entity.
>  
>  One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
>  must be set for every pad.
> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> index 4991281dcccc..03a9e0b8ebab 100644
> --- a/drivers/media/mc/mc-entity.c
> +++ b/drivers/media/mc/mc-entity.c
> @@ -1071,7 +1071,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
>  
>  	for (i = 0; i < entity->num_pads; i++) {
>  		if ((entity->pads[i].flags &
> -		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
> +		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
> +		      MEDIA_PAD_FL_INTERNAL)) != pad_type)
>  			continue;
>  
>  		if (entity->pads[i].sig_type == sig_type)
> @@ -1094,6 +1095,9 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
>  		return -EINVAL;
>  	if (WARN_ON(!(source->pads[source_pad].flags & MEDIA_PAD_FL_SOURCE)))
>  		return -EINVAL;
> +	if (WARN_ON(source->pads[source_pad].flags & MEDIA_PAD_FL_SOURCE &&
> +		    source->pads[source_pad].flags & MEDIA_PAD_FL_INTERNAL))

Even though this code is correct, I prefer to have () around the 'x & y' parts.
It avoids confusion for the reader.

Up to you, though.

Regards,

	Hans

> +		return -EINVAL;
>  	if (WARN_ON(!(sink->pads[sink_pad].flags & MEDIA_PAD_FL_SINK)))
>  		return -EINVAL;
>  
> diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
> index 1c80b1d6bbaf..80cfd12a43fc 100644
> --- a/include/uapi/linux/media.h
> +++ b/include/uapi/linux/media.h
> @@ -208,6 +208,7 @@ struct media_entity_desc {
>  #define MEDIA_PAD_FL_SINK			(1U << 0)
>  #define MEDIA_PAD_FL_SOURCE			(1U << 1)
>  #define MEDIA_PAD_FL_MUST_CONNECT		(1U << 2)
> +#define MEDIA_PAD_FL_INTERNAL			(1U << 3)
>  
>  struct media_pad_desc {
>  	__u32 entity;		/* entity ID */

Re: [PATCH v3 06/10] media: uapi: Add a macro to tell whether an mbus code is metadata

[Sakari Ailus]

On Tue, Aug 08, 2023 at 10:14:11AM +0200, Hans Verkuil wrote:
> On 08/08/2023 09:55, Sakari Ailus wrote:
> > Add a macro to tell whether a given mbus code is metadata.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  include/uapi/linux/media-bus-format.h | 3 +++
> >  1 file changed, 3 insertions(+)
> > 
> > diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
> > index 9ee031397372..2486b4178c5f 100644
> > --- a/include/uapi/linux/media-bus-format.h
> > +++ b/include/uapi/linux/media-bus-format.h
> > @@ -182,4 +182,7 @@
> >  #define MEDIA_BUS_FMT_META_20			0x8006
> >  #define MEDIA_BUS_FMT_META_24			0x8007
> >  
> > +#define MEDIA_BUS_FMT_IS_META(code)		\
> > +	((code) & 0xf000 == 0x7000 || (code) & 0xf000 == 0x8000)
> 
> I don't think this is right: == has a higher precedence than &, so this
> actually reads:
> 
> 	code & (0xf000 == 0x7000)

You're right, I'll fix this for v4.

-- 
Sakari Ailus

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

[Hans Verkuil]

On 08/08/2023 09:55, Sakari Ailus wrote:
> Generic 8-bit metadata formats define the in-memory data layout but not
> the format of the data itself. The reasoning for having such formats is to
> allow CSI-2 receiver drivers to receive and DMA drivers to write the data
> to memory without knowing a large number of device specific formats.
> 
> These formats may be used only in conjunction of a Media controller
> pipeline where the internal pad of the source sub-device defines the
> specific format of the data (using an mbus code).
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../userspace-api/media/v4l/meta-formats.rst  |   1 +
>  .../media/v4l/metafmt-generic.rst             | 331 ++++++++++++++++++
>  drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
>  include/uapi/linux/videodev2.h                |   9 +
>  4 files changed, 349 insertions(+)
>  create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
> 
> diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
> index 0bb61fc5bc00..919f595576b9 100644
> --- a/Documentation/userspace-api/media/v4l/meta-formats.rst
> +++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
> @@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only.
>      metafmt-vsp1-hgo
>      metafmt-vsp1-hgt
>      metafmt-vivid
> +    metafmt-generic
> diff --git a/Documentation/userspace-api/media/v4l/metafmt-generic.rst b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> new file mode 100644
> index 000000000000..a27bfc721edf
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> @@ -0,0 +1,331 @@
> +.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
> +
> +**************************************************************************************************************************************************************************************************************************************************************************************************************************
> +V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O'), V4L2_META_FMT_GENERIC_CSI2_2_24 ('MC2O')
> +**************************************************************************************************************************************************************************************************************************************************************************************************************************
> +
> +
> +Generic line-based metadata formats
> +
> +
> +Description
> +===========
> +
> +These generic line-based metadata formats define the memory layout of the data
> +without defining the format or meaning of the metadata itself. These formats may
> +only be used with a Media controller pipeline where the more specific format is
> +defined in an :ref:`internal source pad <MEDIA-PAD-FL-INTERNAL>` of the source
> +sub-device. See also :ref:`source routes <v4l2-subdev-source-routes>`.
> +
> +.. _v4l2-meta-fmt-generic-8:
> +
> +V4L2_META_FMT_GENERIC_8
> +-----------------------
> +
> +The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format.
> +
> +This format is also used on CSI-2 on both 8 bits per sample as well as on

on both 8 bits per sample as well as on ->
for both 8 bits per sample as well as for

Same elsewhere.

> +16 bits per sample when two bytes of metadata are packed into one sample.
> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_8.**
> +Each cell is one byte. "M" denotes a byte of metadata.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|
> +
> +.. flat-table::
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - M\ :sub:`10`
> +      - M\ :sub:`20`
> +      - M\ :sub:`30`
> +    * - start + 4:
> +      - M\ :sub:`01`
> +      - M\ :sub:`11`
> +      - M\ :sub:`21`
> +      - M\ :sub:`31`
> +
> +.. _v4l2-meta-fmt-generic-csi2-10:
> +
> +V4L2_META_FMT_GENERIC_CSI2_10
> +-----------------------------
> +
> +V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
> +for each 8 bits of data. Every four bytes of metadata is followed by a single
> +byte of padding. The way the data is stored follows the CSI-2 specification.
> +
> +This format is also used on CSI-2 on 20 bits per sample format that packs two
> +bytes of metadata into one sample.
> +
> +This format is little endian.
> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
> +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.

I think you should document whether the padding is always 0 or can be any value.
Perhaps 'X' is a better 'name' for the padding byte in the latter case.

> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
> +
> +.. flat-table::
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - M\ :sub:`10`
> +      - M\ :sub:`20`
> +      - M\ :sub:`30`
> +      - p
> +    * - start + 5:
> +      - M\ :sub:`01`
> +      - M\ :sub:`11`
> +      - M\ :sub:`21`
> +      - M\ :sub:`31`
> +      - p
> +
> +.. _v4l2-meta-fmt-generic-csi2-12:
> +
> +V4L2_META_FMT_GENERIC_CSI2_12
> +-----------------------------
> +
> +V4L2_META_FMT_GENERIC_CSI2_12 contains packed 8-bit generic metadata, 12 bits
> +for each 8 bits of data. Every four bytes of metadata is followed by two bytes
> +of padding. The way the data is stored follows the CSI-2 specification.
> +
> +This format is little endian.
> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_12.**
> +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> +
> +.. flat-table::
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - M\ :sub:`10`
> +      - M\ :sub:`20`
> +      - M\ :sub:`30`
> +      - p
> +      - p
> +    * - start + 6:
> +      - M\ :sub:`01`
> +      - M\ :sub:`11`
> +      - M\ :sub:`21`
> +      - M\ :sub:`31`
> +      - p
> +      - p
> +
> +.. _v4l2-meta-fmt-generic-csi2-14:
> +
> +V4L2_META_FMT_GENERIC_CSI2_14
> +-----------------------------
> +
> +V4L2_META_FMT_GENERIC_CSI2_14 contains packed 8-bit generic metadata, 14 bits
> +for each 8 bits of data. Every four bytes of metadata is followed by three
> +bytes of padding. The way the data is stored follows the CSI-2 specification.
> +
> +This format is little endian.
> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_14.**
> +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{.8cm}|
> +
> +.. flat-table::
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - M\ :sub:`10`
> +      - M\ :sub:`20`
> +      - M\ :sub:`30`
> +      - p
> +      - p
> +      - p
> +    * - start + 7:
> +      - M\ :sub:`01`
> +      - M\ :sub:`11`
> +      - M\ :sub:`21`
> +      - M\ :sub:`31`
> +      - p
> +      - p
> +      - p
> +
> +.. _v4l2-meta-fmt-generic-csi2-16:
> +
> +V4L2_META_FMT_GENERIC_CSI2_16
> +-----------------------------
> +
> +V4L2_META_FMT_GENERIC_CSI2_16 contains packed 8-bit generic metadata, 16 bits
> +for each 8 bits of data. Every byte of metadata is followed by one byte of
> +padding. The way the data is stored follows the CSI-2 specification.
> +
> +This format is little endian.
> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_16.**
> +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|
> +
> +.. flat-table::
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - p
> +      - M\ :sub:`10`
> +      - p
> +      - M\ :sub:`20`
> +      - p
> +      - M\ :sub:`30`
> +      - p
> +    * - start + 8:
> +      - M\ :sub:`01`
> +      - p
> +      - M\ :sub:`11`
> +      - p
> +      - M\ :sub:`21`
> +      - p
> +      - M\ :sub:`31`
> +      - p
> +
> +.. _v4l2-meta-fmt-generic-csi2-20:
> +
> +V4L2_META_FMT_GENERIC_CSI2_20
> +-----------------------------
> +
> +V4L2_META_FMT_GENERIC_CSI2_20 contains packed 8-bit generic metadata, 20 bits
> +for each 8 bits of data. Every byte of metadata is followed by alternating one
> +and two bytes of padding. The way the data is stored follows the CSI-2
> +specification.
> +
> +This format is little endian.
> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_20.**
> +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> +
> +.. flat-table::
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8 8 8 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - p
> +      - M\ :sub:`10`
> +      - p
> +      - p
> +      - M\ :sub:`20`
> +      - p
> +      - M\ :sub:`30`
> +      - p
> +      - p
> +    * - start + 10:
> +      - M\ :sub:`01`
> +      - p
> +      - M\ :sub:`11`
> +      - p
> +      - p
> +      - M\ :sub:`21`
> +      - p
> +      - M\ :sub:`31`
> +      - p
> +      - p
> +
> +.. _v4l2-meta-fmt-generic-csi2-24:
> +
> +V4L2_META_FMT_GENERIC_CSI2_24
> +-----------------------------
> +
> +V4L2_META_FMT_GENERIC_CSI2_24 contains packed 8-bit generic metadata, 24 bits
> +for each 8 bits of data. Every byte of metadata is followed by two bytes of
> +padding. The way the data is stored follows the CSI-2 specification.
> +
> +This format is little endian.
> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_24.**
> +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> +
> +.. flat-table::
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8 8 8 8 8 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - p
> +      - p
> +      - M\ :sub:`10`
> +      - p
> +      - p
> +      - M\ :sub:`20`
> +      - p
> +      - p
> +      - M\ :sub:`30`
> +      - p
> +      - p
> +    * - start + 12:
> +      - M\ :sub:`01`
> +      - p
> +      - p
> +      - M\ :sub:`11`
> +      - p
> +      - p
> +      - M\ :sub:`21`
> +      - p
> +      - p
> +      - M\ :sub:`31`
> +      - p
> +      - p
> +
> +.. _v4l2-meta-fmt-generic-csi2-2-24:
> +
> +V4L2_META_FMT_GENERIC_CSI2_2_24
> +-------------------------------
> +
> +V4L2_META_FMT_GENERIC_CSI2_2_24 contains packed 8-bit generic metadata, 24 bits
> +for each two times 8 bits of data. Every two bytes of metadata are followed by
> +one byte of padding. The way the data is stored follows the CSI-2
> +specification.
> +
> +This format is little endian.
> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_2_24.**
> +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
> +
> +.. flat-table::
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - M\ :sub:`10`
> +      - p
> +      - M\ :sub:`20`
> +      - M\ :sub:`30`
> +      - p
> +    * - start + 6:
> +      - M\ :sub:`01`
> +      - M\ :sub:`11`
> +      - p
> +      - M\ :sub:`21`
> +      - M\ :sub:`31`
> +      - p
> +
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index f4d9d6279094..fbbddc333a30 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -1452,6 +1452,14 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
>  	case V4L2_PIX_FMT_Y210:		descr = "10-bit YUYV Packed"; break;
>  	case V4L2_PIX_FMT_Y212:		descr = "12-bit YUYV Packed"; break;
>  	case V4L2_PIX_FMT_Y216:		descr = "16-bit YUYV Packed"; break;
> +	case V4L2_META_FMT_GENERIC_8:	descr = "8-bit Generic Metadata"; break;
> +	case V4L2_META_FMT_GENERIC_CSI2_10:	descr = "8b Generic Meta, 10b CSI-2"; break;
> +	case V4L2_META_FMT_GENERIC_CSI2_12:	descr = "8b Generic Meta, 12b CSI-2"; break;
> +	case V4L2_META_FMT_GENERIC_CSI2_14:	descr = "8b Generic Meta, 14b CSI-2"; break;
> +	case V4L2_META_FMT_GENERIC_CSI2_16:	descr = "8b Generic Meta, 16b CSI-2"; break;
> +	case V4L2_META_FMT_GENERIC_CSI2_20:	descr = "8b Generic Meta, 20b CSI-2"; break;
> +	case V4L2_META_FMT_GENERIC_CSI2_24:	descr = "8b Generic Meta, 24b CSI-2"; break;
> +	case V4L2_META_FMT_GENERIC_CSI2_2_24:	descr = "2x8b Generic Meta, 24b CSI-2"; break;
>  
>  	default:
>  		/* Compressed formats */
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index 78260e5d9985..b4284a564025 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -838,6 +838,15 @@ struct v4l2_pix_format {
>  #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
>  #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
>  
> +#define V4L2_META_FMT_GENERIC_8		v4l2_fourcc('M', 'E', 'T', '8') /* Generic 8-bit metadata */
> +#define V4L2_META_FMT_GENERIC_CSI2_10	v4l2_fourcc('M', 'C', '1', 'A') /* 10-bit CSI-2 packed 8-bit metadata */
> +#define V4L2_META_FMT_GENERIC_CSI2_12	v4l2_fourcc('M', 'C', '1', 'C') /* 12-bit CSI-2 packed 8-bit metadata */
> +#define V4L2_META_FMT_GENERIC_CSI2_14	v4l2_fourcc('M', 'C', '1', 'E') /* 14-bit CSI-2 packed 8-bit metadata */
> +#define V4L2_META_FMT_GENERIC_CSI2_16	v4l2_fourcc('M', 'C', '1', 'G') /* 16-bit CSI-2 packed 8-bit metadata */
> +#define V4L2_META_FMT_GENERIC_CSI2_20	v4l2_fourcc('M', 'C', '1', 'K') /* 20-bit CSI-2 packed 8-bit metadata */
> +#define V4L2_META_FMT_GENERIC_CSI2_24	v4l2_fourcc('M', 'C', '1', 'O') /* 24-bit CSI-2 packed 8-bit metadata */
> +#define V4L2_META_FMT_GENERIC_CSI2_2_24	v4l2_fourcc('M', 'C', '2', 'O') /* 2 bytes of 8-bit metadata, 24-bit CSI-2 packed */
> +
>  /* priv field value to indicates that subsequent fields are valid. */
>  #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
>  

Regards,

	Hans

Re: [PATCH v3 10/10] media: uapi: v4l: Document source routes

[Hans Verkuil]

On 08/08/2023 09:55, Sakari Ailus wrote:
> Document how internal pads are used on source routes.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../userspace-api/media/v4l/dev-subdev.rst    | 179 ++++++++++++++++++
>  1 file changed, 179 insertions(+)
> 
> diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> index b023918177b5..27b0fe2dc83a 100644
> --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> @@ -551,6 +551,27 @@ A stream at a specific point in the media pipeline is identified by the
>  sub-device and a (pad, stream) pair. For sub-devices that do not support
>  multiplexed streams the 'stream' field is always 0.
>  
> +.. _v4l2-subdev-source-routes:
> +
> +Internal pads and source routes
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +Cases where a single sub-device source pad is traversed by multiple streams one
> +or more of which originate from within the sub-device itself are special as
> +there is no external sink pad for such routes. In those cases, the sources of
> +the internally generated streams are represented by internal sink pads, which
> +are sink pads that have the :ref:`MEDIA_PAD_FL_INTERNAL <MEDIA-PAD-FL-INTERNAL>`
> +pad flag set.
> +
> +Internal pads have all the properties of an external pad, including formats and
> +selections. The format in this case is the source format of the stream. An
> +internal pad always has a single stream only (0).
> +
> +/Source routes/ are routes from an internal sink pad to a(n external) source

'/Source routes/' appears just like that in the generated documentation, that's
probably not what you intended.

> +pad. Generally source routes are not modifiable but they can be activated and
> +deactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +<v4l2-subdev-routing-flags>` flag, depending on driver capabilities.
> +
>  Interaction between routes, streams, formats and selections
>  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>  
> @@ -666,3 +687,161 @@ A common way to accomplish this is to start from the sensors and propagate the
>  configurations along the stream towards the receiver,
>  using :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each
>  stream endpoint in each sub-device.
> +
> +Internal pads setup example
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +A simple example of a multiplexed stream setup might be as follows:
> +
> +- A CCS camera sensor source sub-device, with one sink pad (0), one source pad

Explain what 'CCS' means: probably a link to MIPI CCS is sufficient, but for newbies
'CCS' doesn't provide any information.

> +  (1), an internal sink pad (2) that represents the source of embedded

So since the INTERNAL flag is associated with a sink pad, this is now called an
'internal sink' instead of 'internal source' as it was before. I agree with that,
but note that the phrase 'internal source' is still used in several places,
including the cover letter. For the next version you post, please check for that
and change the terminology since mixing the two is very confusing :-)

> +  data. There are two routes, one from the sink pad to the source, and another
> +  from the internal sink pad to the source pad. The embedded data stream needs
> +  to be enabled by activating the related route. The configuration of the rest
> +  of the CCS sub-devices is omitted from this example.
> +
> +- Multiplexer bridge (Bridge). The bridge has one sink pad, connected to the
> +  sensor (pad 0), and one source pad (pad 1), which outputs two streams.
> +
> +- Receiver in the SoC (Receiver). The receiver has a single sink pad (pad 0),
> +  connected to the bridge, and two source pads (pads 1-2), going to the DMA
> +  engine. The receiver demultiplexes the incoming streams to the source pads.
> +
> +- DMA Engines in the SoC (DMA Engine), one for each stream. Each DMA engine is
> +  connected to a single source pad in the receiver.
> +
> +The sensor, the bridge and the receiver are modeled as V4L2 sub-devices,
> +exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are
> +modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
> +
> +To configure this pipeline, the userspace must take the following steps:
> +
> +1) Set up media links between entities: connect the sensors to the bridge,
> +   bridge to the receiver, and the receiver to the DMA engines. This step does
> +   not differ from normal non-multiplexed media controller setup.
> +
> +2) Configure routing
> +
> +.. flat-table:: Camera sensor
> +    :header-rows: 1
> +
> +    * - Sink Pad/Stream
> +      - Source Pad/Stream
> +      - Routing Flags
> +      - Comments
> +    * - 0/0
> +      - 1/0
> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +      - Pixel data stream from the sink pad

Huh? It is a sensor subdev, so aren't both pixel data and metadata
coming from an internal sink?

I think you are actually describing the internal subdev of a CCS
sensor here, but that is not clear from the description at all.

Since CCS sensors are relatively common (I think), it might be
good to give a fully fledged example.

But suppose you have a non-CCS sensor that generates the pixel data
and metadata, then you would have two internal sink pads and one
source pad for two streams, right?

And very simple sensors that currently just have a single source pad
could be describes as having an internal sink pad connected to the
source pad by a fixed route. But we don't do that, because it is
overkill. Correct? If so, then this might be useful information to add
to answer the question why some sensors have internal sinks and others
do not.

> +    * - 2/0
> +      - 1/1
> +      - **V4L2_SUBDEV_ROUTE_FL_ACTIVE**
> +      - Metadata stream from the internal sink pad
> +
> +.. flat-table:: Bridge routing table
> +    :header-rows: 1
> +
> +    * - Sink Pad/Stream
> +      - Source Pad/Stream
> +      - Routing Flags
> +      - Comments
> +    * - 0/0
> +      - 1/0
> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +      - Pixel data stream from camera sensor
> +    * - 0/1
> +      - 1/1
> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +      - Metadata stream from camera sensor
> +
> +.. flat-table:: Receiver routing table
> +    :header-rows:  1
> +
> +    * - Sink Pad/Stream
> +      - Source Pad/Stream
> +      - Routing Flags
> +      - Comments
> +    * - 0/0
> +      - 1/0
> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +      - Pixel data stream from camera sensor
> +    * - 0/1
> +      - 2/0
> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +      - Metadata stream from camera sensor
> +
> +The options available in sensor's routing configuration are dictated by hardware
> +capabilities: typically camera sensors always produce image data stream while
> +the embedded data stream typically can be either enabled or disabled.
> +
> +3) Configure formats and selections
> +
> +This example assumes that the formats are propagated from sink pad to the source
> +pad as-is. The tables contain fields of both struct v4l2_subdev_format and
> +struct v4l2_mbus_framefmt. The full configuration of CCS camera sensor is out of
> +scope of this example.
> +
> +.. flat-table:: Formats set on the sub-devices. Bold values are set, others are
> +                static or propagated.
> +    :header-rows: 1
> +    :fill-cells:
> +
> +    * - Sub-device
> +      - Pad/Stream
> +      - Width
> +      - Height
> +      - Code
> +    * - :rspan:`3` Camera sensor sub-device (CCS source sub-device)
> +      - 0/0
> +      - 640
> +      - 480
> +      - MEDIA_BUS_FMT_SGRBG10
> +    * - 1/0
> +      - 640
> +      - 480
> +      - **MEDIA_BUS_FMT_SGRBG10**
> +    * - 2/0
> +      - 640
> +      - 2
> +      - MEDIA_BUS_FMT_CCS_EMBEDDED_10
> +    * - 1/1
> +      - 640
> +      - 2
> +      - MEDIA_BUS_FMT_META_10
> +    * - :rspan:`3` Bridge
> +      - 0/0
> +      - **640**
> +      - **480**
> +      - **MEDIA_BUS_FMT_SGRBG10**
> +    * - 1/0
> +      - 640
> +      - 480
> +      - MEDIA_BUS_FMT_SGRBG10
> +    * - 0/1
> +      - **640**
> +      - **2**
> +      - **MEDIA_BUS_FMT_META_10**
> +    * - 1/1
> +      - 640
> +      - 2
> +      - MEDIA_BUS_FMT_META_10
> +    * - :rspan:`3` Receiver
> +      - 0/0
> +      - **640**
> +      - **480**
> +      - **MEDIA_BUS_FMT_SGRBG10**
> +    * - 1/0
> +      - 640
> +      - 480
> +      - MEDIA_BUS_FMT_SGRBG10
> +    * - 0/1
> +      - **640**
> +      - **2**
> +      - **MEDIA_BUS_FMT_META_10**
> +    * - 2/0
> +      - 640
> +      - 2
> +      - MEDIA_BUS_FMT_META_10
> +
> +The embedded data format does not need to be configured as the format is
> +dictated by the pixel data format in this case.

Regards,

	Hans

Re: [PATCH v3 02/10] media: mc: Check pad flag validity

[Jacopo Mondi]

Hi Sakari

On Tue, Aug 08, 2023 at 10:55:30AM +0300, Sakari Ailus wrote:
> Check the validity of pad flags on entity init. Exactly one of the flags
> must be set.
>
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  drivers/media/mc/mc-entity.c | 11 ++++++++++-
>  1 file changed, 10 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> index 83468d4a440b..4991281dcccc 100644
> --- a/drivers/media/mc/mc-entity.c
> +++ b/drivers/media/mc/mc-entity.c
> @@ -197,6 +197,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
>  	struct media_device *mdev = entity->graph_obj.mdev;
>  	struct media_pad *iter;
>  	unsigned int i = 0;
> +	int ret = 0;
>
>  	if (num_pads >= MEDIA_ENTITY_MAX_PADS)
>  		return -E2BIG;
> @@ -210,6 +211,14 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
>  	media_entity_for_each_pad(entity, iter) {
>  		iter->entity = entity;
>  		iter->index = i++;
> +
> +		if (hweight32(iter->flags & (MEDIA_PAD_FL_SINK |
> +					     MEDIA_PAD_FL_SOURCE))
> +		    != 1) {

Fits on the previous line, if you like it

> +			ret = -EINVAL;
> +			break;

Now that we can interrupt the for loop earlier, do you need to cleanup
objects appended to the mdev list with the media_gobj_create() call here below ?

> +		}
> +
>  		if (mdev)
>  			media_gobj_create(mdev, MEDIA_GRAPH_PAD,
>  					  &iter->graph_obj);
> @@ -218,7 +227,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
>  	if (mdev)
>  		mutex_unlock(&mdev->graph_mutex);
>
> -	return 0;
> +	return ret;
>  }
>  EXPORT_SYMBOL_GPL(media_entity_pads_init);
>
> --
> 2.39.2
>

Re: [PATCH v3 03/10] media: mc: Add INTERNAL pad flag

[Jacopo Mondi]

Hi Sakari

On Tue, Aug 08, 2023 at 10:55:31AM +0300, Sakari Ailus wrote:
> Internal source pads will be used as routing endpoints in V4L2
> [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
>
> Also prevent creating links to pads that have been flagged as internal.
>
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  Documentation/userspace-api/media/glossary.rst             | 6 ++++++
>  Documentation/userspace-api/media/mediactl/media-types.rst | 6 ++++++
>  drivers/media/mc/mc-entity.c                               | 6 +++++-
>  include/uapi/linux/media.h                                 | 1 +
>  4 files changed, 18 insertions(+), 1 deletion(-)
>
> diff --git a/Documentation/userspace-api/media/glossary.rst b/Documentation/userspace-api/media/glossary.rst
> index 96a360edbf3b..f7b99a4527c7 100644
> --- a/Documentation/userspace-api/media/glossary.rst
> +++ b/Documentation/userspace-api/media/glossary.rst
> @@ -173,6 +173,12 @@ Glossary
>  	An integrated circuit that integrates all components of a computer
>  	or other electronic systems.
>
> +_media-glossary-stream:
> +    Stream
> +	A distinct flow of data (image data or metadata) over a media pipeline
> +	from source to sink. A source may be e.g. an image sensor and a sink
> +	e.g. a memory buffer.
> +
>      V4L2 API
>  	**V4L2 userspace API**
>
> diff --git a/Documentation/userspace-api/media/mediactl/media-types.rst b/Documentation/userspace-api/media/mediactl/media-types.rst
> index 0ffeece1e0c8..28941da27790 100644
> --- a/Documentation/userspace-api/media/mediactl/media-types.rst
> +++ b/Documentation/userspace-api/media/mediactl/media-types.rst
> @@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
>  .. _MEDIA-PAD-FL-SINK:
>  .. _MEDIA-PAD-FL-SOURCE:
>  .. _MEDIA-PAD-FL-MUST-CONNECT:
> +.. _MEDIA-PAD-FL-INTERNAL:
>
>  .. flat-table:: Media pad flags
>      :header-rows:  0
> @@ -382,6 +383,11 @@ Types and flags used to represent the media graph elements
>  	  when this flag isn't set; the absence of the flag doesn't imply
>  	  there is none.
>
> +    *  -  ``MEDIA_PAD_FL_INTERNAL``
> +       -  The internal flag indicates an internal pad that has no external
> +	  connections. Such a pad shall not be connected with a link. The
> +	  internal flag indicates that the :ref:``stream
> +	  <media-glossary-stream>`` either starts or ends in the entity.
>
>  One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
>  must be set for every pad.
> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> index 4991281dcccc..03a9e0b8ebab 100644
> --- a/drivers/media/mc/mc-entity.c
> +++ b/drivers/media/mc/mc-entity.c
> @@ -1071,7 +1071,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
>
>  	for (i = 0; i < entity->num_pads; i++) {
>  		if ((entity->pads[i].flags &
> -		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
> +		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
> +		      MEDIA_PAD_FL_INTERNAL)) != pad_type)
>  			continue;
>
>  		if (entity->pads[i].sig_type == sig_type)
> @@ -1094,6 +1095,9 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
>  		return -EINVAL;
>  	if (WARN_ON(!(source->pads[source_pad].flags & MEDIA_PAD_FL_SOURCE)))
>  		return -EINVAL;
> +	if (WARN_ON(source->pads[source_pad].flags & MEDIA_PAD_FL_SOURCE &&

Why do you need to check for this again here ? Hasn't the previous
if() clause just validated that

        source->pads[source_pad].flags & MEDIA_PAD_FL_SOURCE ?

> +		    source->pads[source_pad].flags & MEDIA_PAD_FL_INTERNAL))
> +		return -EINVAL;
>  	if (WARN_ON(!(sink->pads[sink_pad].flags & MEDIA_PAD_FL_SINK)))
>  		return -EINVAL;
>
> diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
> index 1c80b1d6bbaf..80cfd12a43fc 100644
> --- a/include/uapi/linux/media.h
> +++ b/include/uapi/linux/media.h
> @@ -208,6 +208,7 @@ struct media_entity_desc {
>  #define MEDIA_PAD_FL_SINK			(1U << 0)
>  #define MEDIA_PAD_FL_SOURCE			(1U << 1)
>  #define MEDIA_PAD_FL_MUST_CONNECT		(1U << 2)
> +#define MEDIA_PAD_FL_INTERNAL			(1U << 3)
>
>  struct media_pad_desc {
>  	__u32 entity;		/* entity ID */
> --
> 2.39.2
>

Re: [PATCH v3 05/10] media: uapi: Document which mbus format fields are valid for metadata

[Jacopo Mondi]

Hi Sakari

On Tue, Aug 08, 2023 at 10:55:33AM +0300, Sakari Ailus wrote:
> Now that metadata mbus formats have been added, it is necessary to define
> which fields in struct v4l2_mbus_format are applicable to them (not many).
>
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  include/uapi/linux/v4l2-mediabus.h | 18 ++++++++++++------
>  1 file changed, 12 insertions(+), 6 deletions(-)
>
> diff --git a/include/uapi/linux/v4l2-mediabus.h b/include/uapi/linux/v4l2-mediabus.h
> index 6b07b73473b5..3cadb3b58b85 100644
> --- a/include/uapi/linux/v4l2-mediabus.h
> +++ b/include/uapi/linux/v4l2-mediabus.h
> @@ -19,12 +19,18 @@
>   * @width:	image width
>   * @height:	image height
>   * @code:	data format code (from enum v4l2_mbus_pixelcode)
> - * @field:	used interlacing type (from enum v4l2_field)
> - * @colorspace:	colorspace of the data (from enum v4l2_colorspace)
> - * @ycbcr_enc:	YCbCr encoding of the data (from enum v4l2_ycbcr_encoding)
> - * @hsv_enc:	HSV encoding of the data (from enum v4l2_hsv_encoding)
> - * @quantization: quantization of the data (from enum v4l2_quantization)
> - * @xfer_func:  transfer function of the data (from enum v4l2_xfer_func)
> + * @field:	used interlacing type (from enum v4l2_field), not applicable
> + *		to metadata mbus codes

"not applicable" is a bit geeric. Should this be set to
V4L2_FIELD_NONE (for metadata, and progressive image formats maybe ?)

> + * @colorspace:	colorspace of the data (from enum v4l2_colorspace), zero on
> + *		metadata mbus codes
> + * @ycbcr_enc:	YCbCr encoding of the data (from enum v4l2_ycbcr_encoding), zero
> + *		on metadata mbus codes
> + * @hsv_enc:	HSV encoding of the data (from enum v4l2_hsv_encoding), zero on
> + *		metadata mbus codes

Can this be zero ?

enum v4l2_hsv_encoding {

	/* Hue mapped to 0 - 179 */
	V4L2_HSV_ENC_180		= 128,

	/* Hue mapped to 0-255 */
	V4L2_HSV_ENC_256		= 129,
};

> + * @quantization: quantization of the data (from enum v4l2_quantization), zero
> + *		on metadata mbus codes
> + * @xfer_func:  transfer function of the data (from enum v4l2_xfer_func), zero
> + *		on metadata mbus codes
>   * @flags:	flags (V4L2_MBUS_FRAMEFMT_*)
>   * @reserved:  reserved bytes that can be later used
>   */
> --
> 2.39.2
>

Re: [PATCH v3 08/10] media: v4l: Support line-based metadata capture

[Jacopo Mondi]

Hi Sakari

On Tue, Aug 08, 2023 at 10:55:36AM +0300, Sakari Ailus wrote:
> many camera sensors, among other devices, transmit embedded data and image
> data for each CSI-2 frame. This embedded data typically contains register
> configuration of the sensor that has been used to capture the image data
> of the same frame.
>
> The embedded data is received by the CSI-2 receiver and has the same
> properties as the image data, including that it is line based: it has
> width, height and bytesperline (stride).
>
> Add these fields to struct v4l2_meta_format and document them.
>
> Also add V4L2_FMT_FLAG_META_LINE_BASED to tell a given format is
> line-based i.e. these fields of struct v4l2_meta_format are valid for it.
>
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../userspace-api/media/v4l/dev-meta.rst          | 15 +++++++++++++++
>  .../userspace-api/media/v4l/vidioc-enum-fmt.rst   |  7 +++++++
>  .../media/videodev2.h.rst.exceptions              |  1 +
>  drivers/media/v4l2-core/v4l2-ioctl.c              |  5 +++--
>  include/uapi/linux/videodev2.h                    | 10 ++++++++++
>  5 files changed, 36 insertions(+), 2 deletions(-)
>
> diff --git a/Documentation/userspace-api/media/v4l/dev-meta.rst b/Documentation/userspace-api/media/v4l/dev-meta.rst
> index 0e7e1ee1471a..4b24bae6e171 100644
> --- a/Documentation/userspace-api/media/v4l/dev-meta.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-meta.rst
> @@ -65,3 +65,18 @@ to 0.
>        - ``buffersize``
>        - Maximum buffer size in bytes required for data. The value is set by the
>          driver.
> +    * - __u32
> +      - ``width``
> +      - Width of a line of metadata in samples. Valid when :c:type`v4l2_fmtdesc`
> +	flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
> +	:c:func:`VIDIOC_ENUM_FMT`.
> +    * - __u32
> +      - ``height``
> +      - Number of rows of metadata. Valid when :c:type`v4l2_fmtdesc` flag
> +	``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
> +	:c:func:`VIDIOC_ENUM_FMT`.
> +    * - __u32
> +      - ``bytesperline``
> +      - Offset in bytes between the beginning of two consecutive lines. Valid
> +	when :c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is
> +	set, otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> index 000c154b0f98..6d7664345a4e 100644
> --- a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> +++ b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> @@ -227,6 +227,13 @@ the ``mbus_code`` field is handled differently:
>  	The application can ask to configure the quantization of the capture
>  	device when calling the :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl with
>  	:ref:`V4L2_PIX_FMT_FLAG_SET_CSC <v4l2-pix-fmt-flag-set-csc>` set.
> +    * - ``V4L2_FMT_FLAG_META_LINE_BASED``
> +      - 0x0200
> +      - The metadata format is line-based. In this case the ``width``,
> +	``height`` and ``bytesperline`` fields of :c:type:`v4l2_meta_format` are
> +	valid. The buffer consists of ``height`` lines, each having ``width``
> +	bytes of data and offset between the beginning of each two consecutive

Isn't ``width`` in samples ?

> +	lines is ``bytesperline``.
>
>  Return Value
>  ============
> diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b/Documentation/userspace-api/media/videodev2.h.rst.exceptions
> index 3e58aac4ef0b..bdc628e8c1d6 100644
> --- a/Documentation/userspace-api/media/videodev2.h.rst.exceptions
> +++ b/Documentation/userspace-api/media/videodev2.h.rst.exceptions
> @@ -215,6 +215,7 @@ replace define V4L2_FMT_FLAG_CSC_XFER_FUNC fmtdesc-flags
>  replace define V4L2_FMT_FLAG_CSC_YCBCR_ENC fmtdesc-flags
>  replace define V4L2_FMT_FLAG_CSC_HSV_ENC fmtdesc-flags
>  replace define V4L2_FMT_FLAG_CSC_QUANTIZATION fmtdesc-flags
> +replace define V4L2_FMT_FLAG_META_LINE_BASED fmtdesc-flags
>
>  # V4L2 timecode types
>  replace define V4L2_TC_TYPE_24FPS timecode-type
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index fbbddc333a30..971d784e7429 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -343,8 +343,9 @@ static void v4l_print_format(const void *arg, bool write_only)
>  	case V4L2_BUF_TYPE_META_OUTPUT:
>  		meta = &p->fmt.meta;
>  		pixelformat = meta->dataformat;
> -		pr_cont(", dataformat=%p4cc, buffersize=%u\n",
> -			&pixelformat, meta->buffersize);
> +		pr_cont(", dataformat=%p4cc, buffersize=%u, width=%u, height=%u, bytesperline=%u\n",
> +			&pixelformat, meta->buffersize, meta->width,
> +			meta->height, meta->bytesperline);
>  		break;
>  	}
>  }
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index b4284a564025..d26c0650c6a7 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -877,6 +877,7 @@ struct v4l2_fmtdesc {
>  #define V4L2_FMT_FLAG_CSC_YCBCR_ENC		0x0080
>  #define V4L2_FMT_FLAG_CSC_HSV_ENC		V4L2_FMT_FLAG_CSC_YCBCR_ENC
>  #define V4L2_FMT_FLAG_CSC_QUANTIZATION		0x0100
> +#define V4L2_FMT_FLAG_META_LINE_BASED		0x0200
>
>  	/* Frame Size and frame rate enumeration */
>  /*
> @@ -2420,10 +2421,19 @@ struct v4l2_sdr_format {
>   * struct v4l2_meta_format - metadata format definition
>   * @dataformat:		little endian four character code (fourcc)
>   * @buffersize:		maximum size in bytes required for data
> + * @width:		number of bytes of data per line (valid for line based

I'm a bit confused here as well, isn't width in samples ?

> + *			formats only, see format documentation)
> + * @height:		number of lines of data per buffer (valid for line based
> + *			formats only)
> + * @bytesperline:	offset between the beginnings of two adjacent lines in
> + *			bytes (valid for line based formats only)
>   */
>  struct v4l2_meta_format {
>  	__u32				dataformat;
>  	__u32				buffersize;
> +	__u32				width;
> +	__u32				height;
> +	__u32				bytesperline;
>  } __attribute__ ((packed));
>
>  /**
> --
> 2.39.2
>

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

[Jacopo Mondi]

Hi Sakari, Hans

On Tue, Aug 08, 2023 at 10:22:21AM +0200, Hans Verkuil wrote:
> On 08/08/2023 09:55, Sakari Ailus wrote:
> > Generic 8-bit metadata formats define the in-memory data layout but not
> > the format of the data itself. The reasoning for having such formats is to
> > allow CSI-2 receiver drivers to receive and DMA drivers to write the data
> > to memory without knowing a large number of device specific formats.
> >
> > These formats may be used only in conjunction of a Media controller
> > pipeline where the internal pad of the source sub-device defines the
> > specific format of the data (using an mbus code).
> >
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  .../userspace-api/media/v4l/meta-formats.rst  |   1 +
> >  .../media/v4l/metafmt-generic.rst             | 331 ++++++++++++++++++
> >  drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
> >  include/uapi/linux/videodev2.h                |   9 +
> >  4 files changed, 349 insertions(+)
> >  create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
> >
> > diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
> > index 0bb61fc5bc00..919f595576b9 100644
> > --- a/Documentation/userspace-api/media/v4l/meta-formats.rst
> > +++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
> > @@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only.
> >      metafmt-vsp1-hgo
> >      metafmt-vsp1-hgt
> >      metafmt-vivid
> > +    metafmt-generic
> > diff --git a/Documentation/userspace-api/media/v4l/metafmt-generic.rst b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > new file mode 100644
> > index 000000000000..a27bfc721edf
> > --- /dev/null
> > +++ b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > @@ -0,0 +1,331 @@
> > +.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
> > +
> > +**************************************************************************************************************************************************************************************************************************************************************************************************************************
> > +V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O'), V4L2_META_FMT_GENERIC_CSI2_2_24 ('MC2O')
> > +**************************************************************************************************************************************************************************************************************************************************************************************************************************
> > +
> > +
> > +Generic line-based metadata formats
> > +
> > +
> > +Description
> > +===========
> > +
> > +These generic line-based metadata formats define the memory layout of the data
> > +without defining the format or meaning of the metadata itself. These formats may
> > +only be used with a Media controller pipeline where the more specific format is
> > +defined in an :ref:`internal source pad <MEDIA-PAD-FL-INTERNAL>` of the source
> > +sub-device. See also :ref:`source routes <v4l2-subdev-source-routes>`.
> > +
> > +.. _v4l2-meta-fmt-generic-8:
> > +
> > +V4L2_META_FMT_GENERIC_8
> > +-----------------------
> > +
> > +The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format.
> > +
> > +This format is also used on CSI-2 on both 8 bits per sample as well as on
>
> on both 8 bits per sample as well as on ->
> for both 8 bits per sample as well as for
>
> Same elsewhere.
>
> > +16 bits per sample when two bytes of metadata are packed into one sample.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_8.**
> > +Each cell is one byte. "M" denotes a byte of metadata.
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|
> > +
> > +.. flat-table::
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - M\ :sub:`10`
> > +      - M\ :sub:`20`
> > +      - M\ :sub:`30`
> > +    * - start + 4:
> > +      - M\ :sub:`01`
> > +      - M\ :sub:`11`
> > +      - M\ :sub:`21`
> > +      - M\ :sub:`31`
> > +
> > +.. _v4l2-meta-fmt-generic-csi2-10:
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_10
> > +-----------------------------
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
> > +for each 8 bits of data. Every four bytes of metadata is followed by a single
> > +byte of padding. The way the data is stored follows the CSI-2 specification.
> > +
> > +This format is also used on CSI-2 on 20 bits per sample format that packs two
> > +bytes of metadata into one sample.
> > +
> > +This format is little endian.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
> > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
>
> I think you should document whether the padding is always 0 or can be any value.
> Perhaps 'X' is a better 'name' for the padding byte in the latter case.
>

Did I get this right that this format is supposed to work as the RAW10
CSI-2 packed image format, where 4 bytes contain the higher 8 bits of
the 10 bits sample and the 5th byte every 4 contains the lower 2 bits of
the previous 4 sample ?

If that's the case, is 'padding' the correct term here ?

> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
> > +
> > +.. flat-table::
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - M\ :sub:`10`
> > +      - M\ :sub:`20`
> > +      - M\ :sub:`30`
> > +      - p
> > +    * - start + 5:
> > +      - M\ :sub:`01`
> > +      - M\ :sub:`11`
> > +      - M\ :sub:`21`
> > +      - M\ :sub:`31`
> > +      - p
> > +
> > +.. _v4l2-meta-fmt-generic-csi2-12:
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_12
> > +-----------------------------
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_12 contains packed 8-bit generic metadata, 12 bits
> > +for each 8 bits of data. Every four bytes of metadata is followed by two bytes
> > +of padding. The way the data is stored follows the CSI-2 specification.
> > +
> > +This format is little endian.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_12.**
> > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> > +
> > +.. flat-table::
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - M\ :sub:`10`
> > +      - M\ :sub:`20`
> > +      - M\ :sub:`30`
> > +      - p
> > +      - p
> > +    * - start + 6:
> > +      - M\ :sub:`01`
> > +      - M\ :sub:`11`
> > +      - M\ :sub:`21`
> > +      - M\ :sub:`31`
> > +      - p
> > +      - p
> > +
> > +.. _v4l2-meta-fmt-generic-csi2-14:
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_14
> > +-----------------------------
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_14 contains packed 8-bit generic metadata, 14 bits
> > +for each 8 bits of data. Every four bytes of metadata is followed by three
> > +bytes of padding. The way the data is stored follows the CSI-2 specification.
> > +
> > +This format is little endian.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_14.**
> > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{.8cm}|
> > +
> > +.. flat-table::
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - M\ :sub:`10`
> > +      - M\ :sub:`20`
> > +      - M\ :sub:`30`
> > +      - p
> > +      - p
> > +      - p
> > +    * - start + 7:
> > +      - M\ :sub:`01`
> > +      - M\ :sub:`11`
> > +      - M\ :sub:`21`
> > +      - M\ :sub:`31`
> > +      - p
> > +      - p
> > +      - p
> > +
> > +.. _v4l2-meta-fmt-generic-csi2-16:
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_16
> > +-----------------------------
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_16 contains packed 8-bit generic metadata, 16 bits
> > +for each 8 bits of data. Every byte of metadata is followed by one byte of
> > +padding. The way the data is stored follows the CSI-2 specification.
> > +
> > +This format is little endian.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_16.**
> > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|
> > +
> > +.. flat-table::
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - p
> > +      - M\ :sub:`10`
> > +      - p
> > +      - M\ :sub:`20`
> > +      - p
> > +      - M\ :sub:`30`
> > +      - p
> > +    * - start + 8:
> > +      - M\ :sub:`01`
> > +      - p
> > +      - M\ :sub:`11`
> > +      - p
> > +      - M\ :sub:`21`
> > +      - p
> > +      - M\ :sub:`31`
> > +      - p
> > +
> > +.. _v4l2-meta-fmt-generic-csi2-20:
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_20
> > +-----------------------------
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_20 contains packed 8-bit generic metadata, 20 bits
> > +for each 8 bits of data. Every byte of metadata is followed by alternating one
> > +and two bytes of padding. The way the data is stored follows the CSI-2
> > +specification.
> > +
> > +This format is little endian.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_20.**
> > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> > +
> > +.. flat-table::
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8 8 8 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - p
> > +      - M\ :sub:`10`
> > +      - p
> > +      - p
> > +      - M\ :sub:`20`
> > +      - p
> > +      - M\ :sub:`30`
> > +      - p
> > +      - p
> > +    * - start + 10:
> > +      - M\ :sub:`01`
> > +      - p
> > +      - M\ :sub:`11`
> > +      - p
> > +      - p
> > +      - M\ :sub:`21`
> > +      - p
> > +      - M\ :sub:`31`
> > +      - p
> > +      - p
> > +
> > +.. _v4l2-meta-fmt-generic-csi2-24:
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_24
> > +-----------------------------
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_24 contains packed 8-bit generic metadata, 24 bits
> > +for each 8 bits of data. Every byte of metadata is followed by two bytes of
> > +padding. The way the data is stored follows the CSI-2 specification.
> > +
> > +This format is little endian.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_24.**
> > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> > +
> > +.. flat-table::
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8 8 8 8 8 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - p
> > +      - p
> > +      - M\ :sub:`10`
> > +      - p
> > +      - p
> > +      - M\ :sub:`20`
> > +      - p
> > +      - p
> > +      - M\ :sub:`30`
> > +      - p
> > +      - p
> > +    * - start + 12:
> > +      - M\ :sub:`01`
> > +      - p
> > +      - p
> > +      - M\ :sub:`11`
> > +      - p
> > +      - p
> > +      - M\ :sub:`21`
> > +      - p
> > +      - p
> > +      - M\ :sub:`31`
> > +      - p
> > +      - p
> > +
> > +.. _v4l2-meta-fmt-generic-csi2-2-24:
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_2_24
> > +-------------------------------
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_2_24 contains packed 8-bit generic metadata, 24 bits
> > +for each two times 8 bits of data. Every two bytes of metadata are followed by
> > +one byte of padding. The way the data is stored follows the CSI-2
> > +specification.
> > +
> > +This format is little endian.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_2_24.**
> > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
> > +
> > +.. flat-table::
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - M\ :sub:`10`
> > +      - p
> > +      - M\ :sub:`20`
> > +      - M\ :sub:`30`
> > +      - p
> > +    * - start + 6:
> > +      - M\ :sub:`01`
> > +      - M\ :sub:`11`
> > +      - p
> > +      - M\ :sub:`21`
> > +      - M\ :sub:`31`
> > +      - p
> > +
> > diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> > index f4d9d6279094..fbbddc333a30 100644
> > --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> > +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> > @@ -1452,6 +1452,14 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
> >  	case V4L2_PIX_FMT_Y210:		descr = "10-bit YUYV Packed"; break;
> >  	case V4L2_PIX_FMT_Y212:		descr = "12-bit YUYV Packed"; break;
> >  	case V4L2_PIX_FMT_Y216:		descr = "16-bit YUYV Packed"; break;
> > +	case V4L2_META_FMT_GENERIC_8:	descr = "8-bit Generic Metadata"; break;
> > +	case V4L2_META_FMT_GENERIC_CSI2_10:	descr = "8b Generic Meta, 10b CSI-2"; break;
> > +	case V4L2_META_FMT_GENERIC_CSI2_12:	descr = "8b Generic Meta, 12b CSI-2"; break;
> > +	case V4L2_META_FMT_GENERIC_CSI2_14:	descr = "8b Generic Meta, 14b CSI-2"; break;
> > +	case V4L2_META_FMT_GENERIC_CSI2_16:	descr = "8b Generic Meta, 16b CSI-2"; break;
> > +	case V4L2_META_FMT_GENERIC_CSI2_20:	descr = "8b Generic Meta, 20b CSI-2"; break;
> > +	case V4L2_META_FMT_GENERIC_CSI2_24:	descr = "8b Generic Meta, 24b CSI-2"; break;
> > +	case V4L2_META_FMT_GENERIC_CSI2_2_24:	descr = "2x8b Generic Meta, 24b CSI-2"; break;
> >
> >  	default:
> >  		/* Compressed formats */
> > diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> > index 78260e5d9985..b4284a564025 100644
> > --- a/include/uapi/linux/videodev2.h
> > +++ b/include/uapi/linux/videodev2.h
> > @@ -838,6 +838,15 @@ struct v4l2_pix_format {
> >  #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
> >  #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
> >
> > +#define V4L2_META_FMT_GENERIC_8		v4l2_fourcc('M', 'E', 'T', '8') /* Generic 8-bit metadata */
> > +#define V4L2_META_FMT_GENERIC_CSI2_10	v4l2_fourcc('M', 'C', '1', 'A') /* 10-bit CSI-2 packed 8-bit metadata */
> > +#define V4L2_META_FMT_GENERIC_CSI2_12	v4l2_fourcc('M', 'C', '1', 'C') /* 12-bit CSI-2 packed 8-bit metadata */
> > +#define V4L2_META_FMT_GENERIC_CSI2_14	v4l2_fourcc('M', 'C', '1', 'E') /* 14-bit CSI-2 packed 8-bit metadata */
> > +#define V4L2_META_FMT_GENERIC_CSI2_16	v4l2_fourcc('M', 'C', '1', 'G') /* 16-bit CSI-2 packed 8-bit metadata */
> > +#define V4L2_META_FMT_GENERIC_CSI2_20	v4l2_fourcc('M', 'C', '1', 'K') /* 20-bit CSI-2 packed 8-bit metadata */
> > +#define V4L2_META_FMT_GENERIC_CSI2_24	v4l2_fourcc('M', 'C', '1', 'O') /* 24-bit CSI-2 packed 8-bit metadata */
> > +#define V4L2_META_FMT_GENERIC_CSI2_2_24	v4l2_fourcc('M', 'C', '2', 'O') /* 2 bytes of 8-bit metadata, 24-bit CSI-2 packed */
> > +
> >  /* priv field value to indicates that subsequent fields are valid. */
> >  #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
> >
>
> Regards,
>
> 	Hans

Re: [PATCH v3 03/10] media: mc: Add INTERNAL pad flag

[Sakari Ailus]

Hi Jacopo,

Thanks for the review.

On Thu, Aug 10, 2023 at 04:12:18PM +0200, Jacopo Mondi wrote:
> Hi Sakari
> 
> On Tue, Aug 08, 2023 at 10:55:31AM +0300, Sakari Ailus wrote:
> > Internal source pads will be used as routing endpoints in V4L2
> > [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
> >
> > Also prevent creating links to pads that have been flagged as internal.
> >
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  Documentation/userspace-api/media/glossary.rst             | 6 ++++++
> >  Documentation/userspace-api/media/mediactl/media-types.rst | 6 ++++++
> >  drivers/media/mc/mc-entity.c                               | 6 +++++-
> >  include/uapi/linux/media.h                                 | 1 +
> >  4 files changed, 18 insertions(+), 1 deletion(-)
> >
> > diff --git a/Documentation/userspace-api/media/glossary.rst b/Documentation/userspace-api/media/glossary.rst
> > index 96a360edbf3b..f7b99a4527c7 100644
> > --- a/Documentation/userspace-api/media/glossary.rst
> > +++ b/Documentation/userspace-api/media/glossary.rst
> > @@ -173,6 +173,12 @@ Glossary
> >  	An integrated circuit that integrates all components of a computer
> >  	or other electronic systems.
> >
> > +_media-glossary-stream:
> > +    Stream
> > +	A distinct flow of data (image data or metadata) over a media pipeline
> > +	from source to sink. A source may be e.g. an image sensor and a sink
> > +	e.g. a memory buffer.
> > +
> >      V4L2 API
> >  	**V4L2 userspace API**
> >
> > diff --git a/Documentation/userspace-api/media/mediactl/media-types.rst b/Documentation/userspace-api/media/mediactl/media-types.rst
> > index 0ffeece1e0c8..28941da27790 100644
> > --- a/Documentation/userspace-api/media/mediactl/media-types.rst
> > +++ b/Documentation/userspace-api/media/mediactl/media-types.rst
> > @@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
> >  .. _MEDIA-PAD-FL-SINK:
> >  .. _MEDIA-PAD-FL-SOURCE:
> >  .. _MEDIA-PAD-FL-MUST-CONNECT:
> > +.. _MEDIA-PAD-FL-INTERNAL:
> >
> >  .. flat-table:: Media pad flags
> >      :header-rows:  0
> > @@ -382,6 +383,11 @@ Types and flags used to represent the media graph elements
> >  	  when this flag isn't set; the absence of the flag doesn't imply
> >  	  there is none.
> >
> > +    *  -  ``MEDIA_PAD_FL_INTERNAL``
> > +       -  The internal flag indicates an internal pad that has no external
> > +	  connections. Such a pad shall not be connected with a link. The
> > +	  internal flag indicates that the :ref:``stream
> > +	  <media-glossary-stream>`` either starts or ends in the entity.
> >
> >  One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
> >  must be set for every pad.
> > diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> > index 4991281dcccc..03a9e0b8ebab 100644
> > --- a/drivers/media/mc/mc-entity.c
> > +++ b/drivers/media/mc/mc-entity.c
> > @@ -1071,7 +1071,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
> >
> >  	for (i = 0; i < entity->num_pads; i++) {
> >  		if ((entity->pads[i].flags &
> > -		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
> > +		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
> > +		      MEDIA_PAD_FL_INTERNAL)) != pad_type)
> >  			continue;
> >
> >  		if (entity->pads[i].sig_type == sig_type)
> > @@ -1094,6 +1095,9 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
> >  		return -EINVAL;
> >  	if (WARN_ON(!(source->pads[source_pad].flags & MEDIA_PAD_FL_SOURCE)))
> >  		return -EINVAL;
> > +	if (WARN_ON(source->pads[source_pad].flags & MEDIA_PAD_FL_SOURCE &&
> 
> Why do you need to check for this again here ? Hasn't the previous
> if() clause just validated that
> 
>         source->pads[source_pad].flags & MEDIA_PAD_FL_SOURCE ?

Yes, this needs to be changed. As a link is created here, the internal flag
shouldn't be present in neither sink nor source pads. I may have forgotten
to address this previously, I'll do this for v4.

> 
> > +		    source->pads[source_pad].flags & MEDIA_PAD_FL_INTERNAL))
> > +		return -EINVAL;
> >  	if (WARN_ON(!(sink->pads[sink_pad].flags & MEDIA_PAD_FL_SINK)))
> >  		return -EINVAL;
> >
> > diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
> > index 1c80b1d6bbaf..80cfd12a43fc 100644
> > --- a/include/uapi/linux/media.h
> > +++ b/include/uapi/linux/media.h
> > @@ -208,6 +208,7 @@ struct media_entity_desc {
> >  #define MEDIA_PAD_FL_SINK			(1U << 0)
> >  #define MEDIA_PAD_FL_SOURCE			(1U << 1)
> >  #define MEDIA_PAD_FL_MUST_CONNECT		(1U << 2)
> > +#define MEDIA_PAD_FL_INTERNAL			(1U << 3)
> >
> >  struct media_pad_desc {
> >  	__u32 entity;		/* entity ID */

-- 
Regards,

Sakari Ailus

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

[Sakari Ailus]

Hi Jacopo,

On Fri, Aug 11, 2023 at 08:31:16AM +0200, Jacopo Mondi wrote:
> > > +V4L2_META_FMT_GENERIC_CSI2_10
> > > +-----------------------------
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
> > > +for each 8 bits of data. Every four bytes of metadata is followed by a single
> > > +byte of padding. The way the data is stored follows the CSI-2 specification.
> > > +
> > > +This format is also used on CSI-2 on 20 bits per sample format that packs two
> > > +bytes of metadata into one sample.
> > > +
> > > +This format is little endian.
> > > +
> > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
> > > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> >
> > I think you should document whether the padding is always 0 or can be any value.
> > Perhaps 'X' is a better 'name' for the padding byte in the latter case.
> >
> 
> Did I get this right that this format is supposed to work as the RAW10
> CSI-2 packed image format, where 4 bytes contain the higher 8 bits of
> the 10 bits sample and the 5th byte every 4 contains the lower 2 bits of
> the previous 4 sample ?
> 
> If that's the case, is 'padding' the correct term here ?

What else would you call it? It'll be zeros that exist just due to the bit
depth used and as such not interesting at all.

-- 
Regards,

Sakari Ailus

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

[Jacopo Mondi]

Hi Sakari

On Fri, Aug 11, 2023 at 09:11:39AM +0000, Sakari Ailus wrote:
> Hi Jacopo,
>
> On Fri, Aug 11, 2023 at 08:31:16AM +0200, Jacopo Mondi wrote:
> > > > +V4L2_META_FMT_GENERIC_CSI2_10
> > > > +-----------------------------
> > > > +
> > > > +V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
> > > > +for each 8 bits of data. Every four bytes of metadata is followed by a single
> > > > +byte of padding. The way the data is stored follows the CSI-2 specification.
> > > > +
> > > > +This format is also used on CSI-2 on 20 bits per sample format that packs two
> > > > +bytes of metadata into one sample.
> > > > +
> > > > +This format is little endian.
> > > > +
> > > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
> > > > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > >
> > > I think you should document whether the padding is always 0 or can be any value.
> > > Perhaps 'X' is a better 'name' for the padding byte in the latter case.
> > >
> >
> > Did I get this right that this format is supposed to work as the RAW10
> > CSI-2 packed image format, where 4 bytes contain the higher 8 bits of
> > the 10 bits sample and the 5th byte every 4 contains the lower 2 bits of
> > the previous 4 sample ?
> >
> > If that's the case, is 'padding' the correct term here ?
>
> What else would you call it? It'll be zeros that exist just due to the bit
> depth used and as such not interesting at all.

Ah, they will be 0s! They won't contain the 2 lower bits of the
previous 4 bytes then ?

I guess this is due to the fact the number of valid bits in each
metadata sample is always 8, regardless of the sample size on the
media bus ?

The DMA engine will re-pack them to CSI2_10 (or CSI2_12 etc) just to match
the image format sizes then ?

In this case, padding is indeed correct, sorry for the noise!

Thanks
  j


>
> --
> Regards,
>
> Sakari Ailus

Re: [PATCH v3 10/10] media: uapi: v4l: Document source routes

[Sakari Ailus]

Hi Hans,

Thank you for the review.

On Tue, Aug 08, 2023 at 10:55:24AM +0200, Hans Verkuil wrote:
> On 08/08/2023 09:55, Sakari Ailus wrote:
> > Document how internal pads are used on source routes.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  .../userspace-api/media/v4l/dev-subdev.rst    | 179 ++++++++++++++++++
> >  1 file changed, 179 insertions(+)
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > index b023918177b5..27b0fe2dc83a 100644
> > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > @@ -551,6 +551,27 @@ A stream at a specific point in the media pipeline is identified by the
> >  sub-device and a (pad, stream) pair. For sub-devices that do not support
> >  multiplexed streams the 'stream' field is always 0.
> >  
> > +.. _v4l2-subdev-source-routes:
> > +
> > +Internal pads and source routes
> > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > +
> > +Cases where a single sub-device source pad is traversed by multiple streams one
> > +or more of which originate from within the sub-device itself are special as
> > +there is no external sink pad for such routes. In those cases, the sources of
> > +the internally generated streams are represented by internal sink pads, which
> > +are sink pads that have the :ref:`MEDIA_PAD_FL_INTERNAL <MEDIA-PAD-FL-INTERNAL>`
> > +pad flag set.
> > +
> > +Internal pads have all the properties of an external pad, including formats and
> > +selections. The format in this case is the source format of the stream. An
> > +internal pad always has a single stream only (0).
> > +
> > +/Source routes/ are routes from an internal sink pad to a(n external) source
> 
> '/Source routes/' appears just like that in the generated documentation, that's
> probably not what you intended.

This was intended to be *Source routes*. I'll address it in v4.

> 
> > +pad. Generally source routes are not modifiable but they can be activated and
> > +deactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > +<v4l2-subdev-routing-flags>` flag, depending on driver capabilities.
> > +
> >  Interaction between routes, streams, formats and selections
> >  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> >  
> > @@ -666,3 +687,161 @@ A common way to accomplish this is to start from the sensors and propagate the
> >  configurations along the stream towards the receiver,
> >  using :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each
> >  stream endpoint in each sub-device.
> > +
> > +Internal pads setup example
> > +^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > +
> > +A simple example of a multiplexed stream setup might be as follows:
> > +
> > +- A CCS camera sensor source sub-device, with one sink pad (0), one source pad
> 
> Explain what 'CCS' means: probably a link to MIPI CCS is sufficient, but for newbies
> 'CCS' doesn't provide any information.
> 
> > +  (1), an internal sink pad (2) that represents the source of embedded
> 
> So since the INTERNAL flag is associated with a sink pad, this is now called an
> 'internal sink' instead of 'internal source' as it was before. I agree with that,
> but note that the phrase 'internal source' is still used in several places,
> including the cover letter. For the next version you post, please check for that
> and change the terminology since mixing the two is very confusing :-)

Are you referring to this patch? There don't seem to be any references to
"intenal source pads" left.

> 
> > +  data. There are two routes, one from the sink pad to the source, and another
> > +  from the internal sink pad to the source pad. The embedded data stream needs
> > +  to be enabled by activating the related route. The configuration of the rest
> > +  of the CCS sub-devices is omitted from this example.
> > +
> > +- Multiplexer bridge (Bridge). The bridge has one sink pad, connected to the
> > +  sensor (pad 0), and one source pad (pad 1), which outputs two streams.
> > +
> > +- Receiver in the SoC (Receiver). The receiver has a single sink pad (pad 0),
> > +  connected to the bridge, and two source pads (pads 1-2), going to the DMA
> > +  engine. The receiver demultiplexes the incoming streams to the source pads.
> > +
> > +- DMA Engines in the SoC (DMA Engine), one for each stream. Each DMA engine is
> > +  connected to a single source pad in the receiver.
> > +
> > +The sensor, the bridge and the receiver are modeled as V4L2 sub-devices,
> > +exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are
> > +modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
> > +
> > +To configure this pipeline, the userspace must take the following steps:
> > +
> > +1) Set up media links between entities: connect the sensors to the bridge,
> > +   bridge to the receiver, and the receiver to the DMA engines. This step does
> > +   not differ from normal non-multiplexed media controller setup.
> > +
> > +2) Configure routing
> > +
> > +.. flat-table:: Camera sensor
> > +    :header-rows: 1
> > +
> > +    * - Sink Pad/Stream
> > +      - Source Pad/Stream
> > +      - Routing Flags
> > +      - Comments
> > +    * - 0/0
> > +      - 1/0
> > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > +      - Pixel data stream from the sink pad
> 
> Huh? It is a sensor subdev, so aren't both pixel data and metadata
> coming from an internal sink?

The CCS driver exposes three sub-devices, so in this case the sink pad
isn't internal. Almost (?) all other sensor drivers have a single
sub-device only, perhaps I'll change the example to align with one of
those.

> 
> I think you are actually describing the internal subdev of a CCS
> sensor here, but that is not clear from the description at all.
> 
> Since CCS sensors are relatively common (I think), it might be
> good to give a fully fledged example.
> 
> But suppose you have a non-CCS sensor that generates the pixel data
> and metadata, then you would have two internal sink pads and one
> source pad for two streams, right?
> 
> And very simple sensors that currently just have a single source pad
> could be describes as having an internal sink pad connected to the
> source pad by a fixed route. But we don't do that, because it is
> overkill. Correct? If so, then this might be useful information to add
> to answer the question why some sensors have internal sinks and others
> do not.
> 
> > +    * - 2/0
> > +      - 1/1
> > +      - **V4L2_SUBDEV_ROUTE_FL_ACTIVE**
> > +      - Metadata stream from the internal sink pad
> > +
> > +.. flat-table:: Bridge routing table
> > +    :header-rows: 1
> > +
> > +    * - Sink Pad/Stream
> > +      - Source Pad/Stream
> > +      - Routing Flags
> > +      - Comments
> > +    * - 0/0
> > +      - 1/0
> > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > +      - Pixel data stream from camera sensor
> > +    * - 0/1
> > +      - 1/1
> > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > +      - Metadata stream from camera sensor
> > +
> > +.. flat-table:: Receiver routing table
> > +    :header-rows:  1
> > +
> > +    * - Sink Pad/Stream
> > +      - Source Pad/Stream
> > +      - Routing Flags
> > +      - Comments
> > +    * - 0/0
> > +      - 1/0
> > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > +      - Pixel data stream from camera sensor
> > +    * - 0/1
> > +      - 2/0
> > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > +      - Metadata stream from camera sensor
> > +
> > +The options available in sensor's routing configuration are dictated by hardware
> > +capabilities: typically camera sensors always produce image data stream while
> > +the embedded data stream typically can be either enabled or disabled.
> > +
> > +3) Configure formats and selections
> > +
> > +This example assumes that the formats are propagated from sink pad to the source
> > +pad as-is. The tables contain fields of both struct v4l2_subdev_format and
> > +struct v4l2_mbus_framefmt. The full configuration of CCS camera sensor is out of
> > +scope of this example.
> > +
> > +.. flat-table:: Formats set on the sub-devices. Bold values are set, others are
> > +                static or propagated.
> > +    :header-rows: 1
> > +    :fill-cells:
> > +
> > +    * - Sub-device
> > +      - Pad/Stream
> > +      - Width
> > +      - Height
> > +      - Code
> > +    * - :rspan:`3` Camera sensor sub-device (CCS source sub-device)
> > +      - 0/0
> > +      - 640
> > +      - 480
> > +      - MEDIA_BUS_FMT_SGRBG10
> > +    * - 1/0
> > +      - 640
> > +      - 480
> > +      - **MEDIA_BUS_FMT_SGRBG10**
> > +    * - 2/0
> > +      - 640
> > +      - 2
> > +      - MEDIA_BUS_FMT_CCS_EMBEDDED_10
> > +    * - 1/1
> > +      - 640
> > +      - 2
> > +      - MEDIA_BUS_FMT_META_10
> > +    * - :rspan:`3` Bridge
> > +      - 0/0
> > +      - **640**
> > +      - **480**
> > +      - **MEDIA_BUS_FMT_SGRBG10**
> > +    * - 1/0
> > +      - 640
> > +      - 480
> > +      - MEDIA_BUS_FMT_SGRBG10
> > +    * - 0/1
> > +      - **640**
> > +      - **2**
> > +      - **MEDIA_BUS_FMT_META_10**
> > +    * - 1/1
> > +      - 640
> > +      - 2
> > +      - MEDIA_BUS_FMT_META_10
> > +    * - :rspan:`3` Receiver
> > +      - 0/0
> > +      - **640**
> > +      - **480**
> > +      - **MEDIA_BUS_FMT_SGRBG10**
> > +    * - 1/0
> > +      - 640
> > +      - 480
> > +      - MEDIA_BUS_FMT_SGRBG10
> > +    * - 0/1
> > +      - **640**
> > +      - **2**
> > +      - **MEDIA_BUS_FMT_META_10**
> > +    * - 2/0
> > +      - 640
> > +      - 2
> > +      - MEDIA_BUS_FMT_META_10
> > +
> > +The embedded data format does not need to be configured as the format is
> > +dictated by the pixel data format in this case.

-- 
Kind regards,

Sakari Ailus

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

[Sakari Ailus]

Hi Jacopo,

On Fri, Aug 11, 2023 at 11:43:26AM +0200, Jacopo Mondi wrote:
> Hi Sakari
> 
> On Fri, Aug 11, 2023 at 09:11:39AM +0000, Sakari Ailus wrote:
> > Hi Jacopo,
> >
> > On Fri, Aug 11, 2023 at 08:31:16AM +0200, Jacopo Mondi wrote:
> > > > > +V4L2_META_FMT_GENERIC_CSI2_10
> > > > > +-----------------------------
> > > > > +
> > > > > +V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
> > > > > +for each 8 bits of data. Every four bytes of metadata is followed by a single
> > > > > +byte of padding. The way the data is stored follows the CSI-2 specification.
> > > > > +
> > > > > +This format is also used on CSI-2 on 20 bits per sample format that packs two
> > > > > +bytes of metadata into one sample.
> > > > > +
> > > > > +This format is little endian.
> > > > > +
> > > > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
> > > > > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > > >
> > > > I think you should document whether the padding is always 0 or can be any value.
> > > > Perhaps 'X' is a better 'name' for the padding byte in the latter case.
> > > >
> > >
> > > Did I get this right that this format is supposed to work as the RAW10
> > > CSI-2 packed image format, where 4 bytes contain the higher 8 bits of
> > > the 10 bits sample and the 5th byte every 4 contains the lower 2 bits of
> > > the previous 4 sample ?
> > >
> > > If that's the case, is 'padding' the correct term here ?
> >
> > What else would you call it? It'll be zeros that exist just due to the bit
> > depth used and as such not interesting at all.
> 
> Ah, they will be 0s! They won't contain the 2 lower bits of the
> previous 4 bytes then ?
> 
> I guess this is due to the fact the number of valid bits in each
> metadata sample is always 8, regardless of the sample size on the
> media bus ?

Correct. This is also documented for the generic metadata formats.

> 
> The DMA engine will re-pack them to CSI2_10 (or CSI2_12 etc) just to match
> the image format sizes then ?

Generally yes. The receiver hardware / DMA is of course free to strip out
the padding but in practice that doesn't seem to be done.

> 
> In this case, padding is indeed correct, sorry for the noise!

No problem.

-- 
Regards,

Sakari Ailus

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

[Sakari Ailus]

Hi Hans,

Thanks for the review.

On Tue, Aug 08, 2023 at 10:22:21AM +0200, Hans Verkuil wrote:
> On 08/08/2023 09:55, Sakari Ailus wrote:
> > Generic 8-bit metadata formats define the in-memory data layout but not
> > the format of the data itself. The reasoning for having such formats is to
> > allow CSI-2 receiver drivers to receive and DMA drivers to write the data
> > to memory without knowing a large number of device specific formats.
> > 
> > These formats may be used only in conjunction of a Media controller
> > pipeline where the internal pad of the source sub-device defines the
> > specific format of the data (using an mbus code).
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  .../userspace-api/media/v4l/meta-formats.rst  |   1 +
> >  .../media/v4l/metafmt-generic.rst             | 331 ++++++++++++++++++
> >  drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
> >  include/uapi/linux/videodev2.h                |   9 +
> >  4 files changed, 349 insertions(+)
> >  create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
> > index 0bb61fc5bc00..919f595576b9 100644
> > --- a/Documentation/userspace-api/media/v4l/meta-formats.rst
> > +++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
> > @@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only.
> >      metafmt-vsp1-hgo
> >      metafmt-vsp1-hgt
> >      metafmt-vivid
> > +    metafmt-generic
> > diff --git a/Documentation/userspace-api/media/v4l/metafmt-generic.rst b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > new file mode 100644
> > index 000000000000..a27bfc721edf
> > --- /dev/null
> > +++ b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > @@ -0,0 +1,331 @@
> > +.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
> > +
> > +**************************************************************************************************************************************************************************************************************************************************************************************************************************
> > +V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O'), V4L2_META_FMT_GENERIC_CSI2_2_24 ('MC2O')
> > +**************************************************************************************************************************************************************************************************************************************************************************************************************************
> > +
> > +
> > +Generic line-based metadata formats
> > +
> > +
> > +Description
> > +===========
> > +
> > +These generic line-based metadata formats define the memory layout of the data
> > +without defining the format or meaning of the metadata itself. These formats may
> > +only be used with a Media controller pipeline where the more specific format is
> > +defined in an :ref:`internal source pad <MEDIA-PAD-FL-INTERNAL>` of the source
> > +sub-device. See also :ref:`source routes <v4l2-subdev-source-routes>`.
> > +
> > +.. _v4l2-meta-fmt-generic-8:
> > +
> > +V4L2_META_FMT_GENERIC_8
> > +-----------------------
> > +
> > +The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format.
> > +
> > +This format is also used on CSI-2 on both 8 bits per sample as well as on
> 
> on both 8 bits per sample as well as on ->
> for both 8 bits per sample as well as for
> 
> Same elsewhere.

There's another for 10 bits per sample, the usage is different elsewhere as
there is only a single bit depth in other cases. 

> 
> > +16 bits per sample when two bytes of metadata are packed into one sample.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_8.**
> > +Each cell is one byte. "M" denotes a byte of metadata.
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|
> > +
> > +.. flat-table::
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - M\ :sub:`10`
> > +      - M\ :sub:`20`
> > +      - M\ :sub:`30`
> > +    * - start + 4:
> > +      - M\ :sub:`01`
> > +      - M\ :sub:`11`
> > +      - M\ :sub:`21`
> > +      - M\ :sub:`31`
> > +
> > +.. _v4l2-meta-fmt-generic-csi2-10:
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_10
> > +-----------------------------
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
> > +for each 8 bits of data. Every four bytes of metadata is followed by a single
> > +byte of padding. The way the data is stored follows the CSI-2 specification.
> > +
> > +This format is also used on CSI-2 on 20 bits per sample format that packs two
> > +bytes of metadata into one sample.
> > +
> > +This format is little endian.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
> > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> 
> I think you should document whether the padding is always 0 or can be any value.
> Perhaps 'X' is a better 'name' for the padding byte in the latter case.

I'll change these for v4.

> 
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
> > +
> > +.. flat-table::
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - M\ :sub:`10`
> > +      - M\ :sub:`20`
> > +      - M\ :sub:`30`
> > +      - p
> > +    * - start + 5:
> > +      - M\ :sub:`01`
> > +      - M\ :sub:`11`
> > +      - M\ :sub:`21`
> > +      - M\ :sub:`31`
> > +      - p
> > +
> > +.. _v4l2-meta-fmt-generic-csi2-12:
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_12
> > +-----------------------------
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_12 contains packed 8-bit generic metadata, 12 bits
> > +for each 8 bits of data. Every four bytes of metadata is followed by two bytes
> > +of padding. The way the data is stored follows the CSI-2 specification.
> > +
> > +This format is little endian.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_12.**
> > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> > +
> > +.. flat-table::
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - M\ :sub:`10`
> > +      - M\ :sub:`20`
> > +      - M\ :sub:`30`
> > +      - p
> > +      - p
> > +    * - start + 6:
> > +      - M\ :sub:`01`
> > +      - M\ :sub:`11`
> > +      - M\ :sub:`21`
> > +      - M\ :sub:`31`
> > +      - p
> > +      - p
> > +
> > +.. _v4l2-meta-fmt-generic-csi2-14:
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_14
> > +-----------------------------
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_14 contains packed 8-bit generic metadata, 14 bits
> > +for each 8 bits of data. Every four bytes of metadata is followed by three
> > +bytes of padding. The way the data is stored follows the CSI-2 specification.
> > +
> > +This format is little endian.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_14.**
> > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{.8cm}|
> > +
> > +.. flat-table::
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - M\ :sub:`10`
> > +      - M\ :sub:`20`
> > +      - M\ :sub:`30`
> > +      - p
> > +      - p
> > +      - p
> > +    * - start + 7:
> > +      - M\ :sub:`01`
> > +      - M\ :sub:`11`
> > +      - M\ :sub:`21`
> > +      - M\ :sub:`31`
> > +      - p
> > +      - p
> > +      - p
> > +
> > +.. _v4l2-meta-fmt-generic-csi2-16:
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_16
> > +-----------------------------
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_16 contains packed 8-bit generic metadata, 16 bits
> > +for each 8 bits of data. Every byte of metadata is followed by one byte of
> > +padding. The way the data is stored follows the CSI-2 specification.
> > +
> > +This format is little endian.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_16.**
> > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|
> > +
> > +.. flat-table::
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - p
> > +      - M\ :sub:`10`
> > +      - p
> > +      - M\ :sub:`20`
> > +      - p
> > +      - M\ :sub:`30`
> > +      - p
> > +    * - start + 8:
> > +      - M\ :sub:`01`
> > +      - p
> > +      - M\ :sub:`11`
> > +      - p
> > +      - M\ :sub:`21`
> > +      - p
> > +      - M\ :sub:`31`
> > +      - p
> > +
> > +.. _v4l2-meta-fmt-generic-csi2-20:
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_20
> > +-----------------------------
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_20 contains packed 8-bit generic metadata, 20 bits
> > +for each 8 bits of data. Every byte of metadata is followed by alternating one
> > +and two bytes of padding. The way the data is stored follows the CSI-2
> > +specification.
> > +
> > +This format is little endian.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_20.**
> > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> > +
> > +.. flat-table::
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8 8 8 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - p
> > +      - M\ :sub:`10`
> > +      - p
> > +      - p
> > +      - M\ :sub:`20`
> > +      - p
> > +      - M\ :sub:`30`
> > +      - p
> > +      - p
> > +    * - start + 10:
> > +      - M\ :sub:`01`
> > +      - p
> > +      - M\ :sub:`11`
> > +      - p
> > +      - p
> > +      - M\ :sub:`21`
> > +      - p
> > +      - M\ :sub:`31`
> > +      - p
> > +      - p
> > +
> > +.. _v4l2-meta-fmt-generic-csi2-24:
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_24
> > +-----------------------------
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_24 contains packed 8-bit generic metadata, 24 bits
> > +for each 8 bits of data. Every byte of metadata is followed by two bytes of
> > +padding. The way the data is stored follows the CSI-2 specification.
> > +
> > +This format is little endian.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_24.**
> > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> > +
> > +.. flat-table::
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8 8 8 8 8 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - p
> > +      - p
> > +      - M\ :sub:`10`
> > +      - p
> > +      - p
> > +      - M\ :sub:`20`
> > +      - p
> > +      - p
> > +      - M\ :sub:`30`
> > +      - p
> > +      - p
> > +    * - start + 12:
> > +      - M\ :sub:`01`
> > +      - p
> > +      - p
> > +      - M\ :sub:`11`
> > +      - p
> > +      - p
> > +      - M\ :sub:`21`
> > +      - p
> > +      - p
> > +      - M\ :sub:`31`
> > +      - p
> > +      - p
> > +
> > +.. _v4l2-meta-fmt-generic-csi2-2-24:
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_2_24
> > +-------------------------------
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_2_24 contains packed 8-bit generic metadata, 24 bits
> > +for each two times 8 bits of data. Every two bytes of metadata are followed by
> > +one byte of padding. The way the data is stored follows the CSI-2
> > +specification.
> > +
> > +This format is little endian.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_2_24.**
> > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
> > +
> > +.. flat-table::
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - M\ :sub:`10`
> > +      - p
> > +      - M\ :sub:`20`
> > +      - M\ :sub:`30`
> > +      - p
> > +    * - start + 6:
> > +      - M\ :sub:`01`
> > +      - M\ :sub:`11`
> > +      - p
> > +      - M\ :sub:`21`
> > +      - M\ :sub:`31`
> > +      - p
> > +
> > diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> > index f4d9d6279094..fbbddc333a30 100644
> > --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> > +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> > @@ -1452,6 +1452,14 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
> >  	case V4L2_PIX_FMT_Y210:		descr = "10-bit YUYV Packed"; break;
> >  	case V4L2_PIX_FMT_Y212:		descr = "12-bit YUYV Packed"; break;
> >  	case V4L2_PIX_FMT_Y216:		descr = "16-bit YUYV Packed"; break;
> > +	case V4L2_META_FMT_GENERIC_8:	descr = "8-bit Generic Metadata"; break;
> > +	case V4L2_META_FMT_GENERIC_CSI2_10:	descr = "8b Generic Meta, 10b CSI-2"; break;
> > +	case V4L2_META_FMT_GENERIC_CSI2_12:	descr = "8b Generic Meta, 12b CSI-2"; break;
> > +	case V4L2_META_FMT_GENERIC_CSI2_14:	descr = "8b Generic Meta, 14b CSI-2"; break;
> > +	case V4L2_META_FMT_GENERIC_CSI2_16:	descr = "8b Generic Meta, 16b CSI-2"; break;
> > +	case V4L2_META_FMT_GENERIC_CSI2_20:	descr = "8b Generic Meta, 20b CSI-2"; break;
> > +	case V4L2_META_FMT_GENERIC_CSI2_24:	descr = "8b Generic Meta, 24b CSI-2"; break;
> > +	case V4L2_META_FMT_GENERIC_CSI2_2_24:	descr = "2x8b Generic Meta, 24b CSI-2"; break;
> >  
> >  	default:
> >  		/* Compressed formats */
> > diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> > index 78260e5d9985..b4284a564025 100644
> > --- a/include/uapi/linux/videodev2.h
> > +++ b/include/uapi/linux/videodev2.h
> > @@ -838,6 +838,15 @@ struct v4l2_pix_format {
> >  #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
> >  #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
> >  
> > +#define V4L2_META_FMT_GENERIC_8		v4l2_fourcc('M', 'E', 'T', '8') /* Generic 8-bit metadata */
> > +#define V4L2_META_FMT_GENERIC_CSI2_10	v4l2_fourcc('M', 'C', '1', 'A') /* 10-bit CSI-2 packed 8-bit metadata */
> > +#define V4L2_META_FMT_GENERIC_CSI2_12	v4l2_fourcc('M', 'C', '1', 'C') /* 12-bit CSI-2 packed 8-bit metadata */
> > +#define V4L2_META_FMT_GENERIC_CSI2_14	v4l2_fourcc('M', 'C', '1', 'E') /* 14-bit CSI-2 packed 8-bit metadata */
> > +#define V4L2_META_FMT_GENERIC_CSI2_16	v4l2_fourcc('M', 'C', '1', 'G') /* 16-bit CSI-2 packed 8-bit metadata */
> > +#define V4L2_META_FMT_GENERIC_CSI2_20	v4l2_fourcc('M', 'C', '1', 'K') /* 20-bit CSI-2 packed 8-bit metadata */
> > +#define V4L2_META_FMT_GENERIC_CSI2_24	v4l2_fourcc('M', 'C', '1', 'O') /* 24-bit CSI-2 packed 8-bit metadata */
> > +#define V4L2_META_FMT_GENERIC_CSI2_2_24	v4l2_fourcc('M', 'C', '2', 'O') /* 2 bytes of 8-bit metadata, 24-bit CSI-2 packed */
> > +
> >  /* priv field value to indicates that subsequent fields are valid. */
> >  #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
> >  
> 
> Regards,
> 
> 	Hans

-- 
Regards,

Sakari Ailus

Re: [PATCH v3 03/10] media: mc: Add INTERNAL pad flag

[Sakari Ailus]

Hi Hans,

On Tue, Aug 08, 2023 at 10:15:45AM +0200, Hans Verkuil wrote:
> On 08/08/2023 09:55, Sakari Ailus wrote:
> > Internal source pads will be used as routing endpoints in V4L2
> > [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
> > 
> > Also prevent creating links to pads that have been flagged as internal.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  Documentation/userspace-api/media/glossary.rst             | 6 ++++++
> >  Documentation/userspace-api/media/mediactl/media-types.rst | 6 ++++++
> >  drivers/media/mc/mc-entity.c                               | 6 +++++-
> >  include/uapi/linux/media.h                                 | 1 +
> >  4 files changed, 18 insertions(+), 1 deletion(-)
> > 
> > diff --git a/Documentation/userspace-api/media/glossary.rst b/Documentation/userspace-api/media/glossary.rst
> > index 96a360edbf3b..f7b99a4527c7 100644
> > --- a/Documentation/userspace-api/media/glossary.rst
> > +++ b/Documentation/userspace-api/media/glossary.rst
> > @@ -173,6 +173,12 @@ Glossary
> >  	An integrated circuit that integrates all components of a computer
> >  	or other electronic systems.
> >  
> > +_media-glossary-stream:
> > +    Stream
> > +	A distinct flow of data (image data or metadata) over a media pipeline
> > +	from source to sink. A source may be e.g. an image sensor and a sink
> > +	e.g. a memory buffer.
> > +
> >      V4L2 API
> >  	**V4L2 userspace API**
> >  
> > diff --git a/Documentation/userspace-api/media/mediactl/media-types.rst b/Documentation/userspace-api/media/mediactl/media-types.rst
> > index 0ffeece1e0c8..28941da27790 100644
> > --- a/Documentation/userspace-api/media/mediactl/media-types.rst
> > +++ b/Documentation/userspace-api/media/mediactl/media-types.rst
> > @@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
> >  .. _MEDIA-PAD-FL-SINK:
> >  .. _MEDIA-PAD-FL-SOURCE:
> >  .. _MEDIA-PAD-FL-MUST-CONNECT:
> > +.. _MEDIA-PAD-FL-INTERNAL:
> >  
> >  .. flat-table:: Media pad flags
> >      :header-rows:  0
> > @@ -382,6 +383,11 @@ Types and flags used to represent the media graph elements
> >  	  when this flag isn't set; the absence of the flag doesn't imply
> >  	  there is none.
> >  
> > +    *  -  ``MEDIA_PAD_FL_INTERNAL``
> > +       -  The internal flag indicates an internal pad that has no external
> > +	  connections. Such a pad shall not be connected with a link. The
> > +	  internal flag indicates that the :ref:``stream
> > +	  <media-glossary-stream>`` either starts or ends in the entity.
> >  
> >  One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
> >  must be set for every pad.
> > diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> > index 4991281dcccc..03a9e0b8ebab 100644
> > --- a/drivers/media/mc/mc-entity.c
> > +++ b/drivers/media/mc/mc-entity.c
> > @@ -1071,7 +1071,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
> >  
> >  	for (i = 0; i < entity->num_pads; i++) {
> >  		if ((entity->pads[i].flags &
> > -		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
> > +		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
> > +		      MEDIA_PAD_FL_INTERNAL)) != pad_type)
> >  			continue;
> >  
> >  		if (entity->pads[i].sig_type == sig_type)
> > @@ -1094,6 +1095,9 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
> >  		return -EINVAL;
> >  	if (WARN_ON(!(source->pads[source_pad].flags & MEDIA_PAD_FL_SOURCE)))
> >  		return -EINVAL;
> > +	if (WARN_ON(source->pads[source_pad].flags & MEDIA_PAD_FL_SOURCE &&
> > +		    source->pads[source_pad].flags & MEDIA_PAD_FL_INTERNAL))
> 
> Even though this code is correct, I prefer to have () around the 'x & y' parts.
> It avoids confusion for the reader.

There actually should be a check here that neither of the pads have
internal flag set --- internal pads may not be connected by a link. I'll
fix that for v4. This was actually a leftover from the "internal source"
flag.

> 
> Up to you, though.

I'd prefer to keep it as-is. :-)

-- 
Regards,

Sakari Ailus

Re: [PATCH v3 02/10] media: mc: Check pad flag validity

[Sakari Ailus]

Hi Jacopo,

On Thu, Aug 10, 2023 at 03:54:40PM +0200, Jacopo Mondi wrote:
> Hi Sakari
> 
> On Tue, Aug 08, 2023 at 10:55:30AM +0300, Sakari Ailus wrote:
> > Check the validity of pad flags on entity init. Exactly one of the flags
> > must be set.
> >
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  drivers/media/mc/mc-entity.c | 11 ++++++++++-
> >  1 file changed, 10 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> > index 83468d4a440b..4991281dcccc 100644
> > --- a/drivers/media/mc/mc-entity.c
> > +++ b/drivers/media/mc/mc-entity.c
> > @@ -197,6 +197,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
> >  	struct media_device *mdev = entity->graph_obj.mdev;
> >  	struct media_pad *iter;
> >  	unsigned int i = 0;
> > +	int ret = 0;
> >
> >  	if (num_pads >= MEDIA_ENTITY_MAX_PADS)
> >  		return -E2BIG;
> > @@ -210,6 +211,14 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
> >  	media_entity_for_each_pad(entity, iter) {
> >  		iter->entity = entity;
> >  		iter->index = i++;
> > +
> > +		if (hweight32(iter->flags & (MEDIA_PAD_FL_SINK |
> > +					     MEDIA_PAD_FL_SOURCE))
> > +		    != 1) {
> 
> Fits on the previous line, if you like it

Not quite.

> 
> > +			ret = -EINVAL;
> > +			break;
> 
> Now that we can interrupt the for loop earlier, do you need to cleanup
> objects appended to the mdev list with the media_gobj_create() call here below ?

Nice find. I'll fix this for v4.

> 
> > +		}
> > +
> >  		if (mdev)
> >  			media_gobj_create(mdev, MEDIA_GRAPH_PAD,
> >  					  &iter->graph_obj);
> > @@ -218,7 +227,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
> >  	if (mdev)
> >  		mutex_unlock(&mdev->graph_mutex);
> >
> > -	return 0;
> > +	return ret;
> >  }
> >  EXPORT_SYMBOL_GPL(media_entity_pads_init);
> >

-- 
Kind regards,

Sakari Ailus

Re: [PATCH v3 05/10] media: uapi: Document which mbus format fields are valid for metadata

[Sakari Ailus]

Hi Jacopo,

Thanks for the review.

On Thu, Aug 10, 2023 at 05:19:02PM +0200, Jacopo Mondi wrote:
> Hi Sakari
> 
> On Tue, Aug 08, 2023 at 10:55:33AM +0300, Sakari Ailus wrote:
> > Now that metadata mbus formats have been added, it is necessary to define
> > which fields in struct v4l2_mbus_format are applicable to them (not many).
> >
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  include/uapi/linux/v4l2-mediabus.h | 18 ++++++++++++------
> >  1 file changed, 12 insertions(+), 6 deletions(-)
> >
> > diff --git a/include/uapi/linux/v4l2-mediabus.h b/include/uapi/linux/v4l2-mediabus.h
> > index 6b07b73473b5..3cadb3b58b85 100644
> > --- a/include/uapi/linux/v4l2-mediabus.h
> > +++ b/include/uapi/linux/v4l2-mediabus.h
> > @@ -19,12 +19,18 @@
> >   * @width:	image width
> >   * @height:	image height
> >   * @code:	data format code (from enum v4l2_mbus_pixelcode)
> > - * @field:	used interlacing type (from enum v4l2_field)
> > - * @colorspace:	colorspace of the data (from enum v4l2_colorspace)
> > - * @ycbcr_enc:	YCbCr encoding of the data (from enum v4l2_ycbcr_encoding)
> > - * @hsv_enc:	HSV encoding of the data (from enum v4l2_hsv_encoding)
> > - * @quantization: quantization of the data (from enum v4l2_quantization)
> > - * @xfer_func:  transfer function of the data (from enum v4l2_xfer_func)
> > + * @field:	used interlacing type (from enum v4l2_field), not applicable
> > + *		to metadata mbus codes
> 
> "not applicable" is a bit geeric. Should this be set to
> V4L2_FIELD_NONE (for metadata, and progressive image formats maybe ?)

I actually intended to have the same wording here than for the other fields
but missed changing this.

0 corresponds to V4L2_FIELD_ANY.

> 
> > + * @colorspace:	colorspace of the data (from enum v4l2_colorspace), zero on
> > + *		metadata mbus codes
> > + * @ycbcr_enc:	YCbCr encoding of the data (from enum v4l2_ycbcr_encoding), zero
> > + *		on metadata mbus codes
> > + * @hsv_enc:	HSV encoding of the data (from enum v4l2_hsv_encoding), zero on
> > + *		metadata mbus codes
> 
> Can this be zero ?
> 
> enum v4l2_hsv_encoding {
> 
> 	/* Hue mapped to 0 - 179 */
> 	V4L2_HSV_ENC_180		= 128,
> 
> 	/* Hue mapped to 0-255 */
> 	V4L2_HSV_ENC_256		= 129,
> };

Good question. Neither value is meaningful for metadata.

The values appear to be such in the enum to avoid colliding with ycbcr
encoding values (another field above).

Generally, what doesn't matter will be zero. These fields have been added
at some point and a lot of drivers do not set them, even for pixel data.

I wonder what Hans and Laurent think.

> 
> > + * @quantization: quantization of the data (from enum v4l2_quantization), zero
> > + *		on metadata mbus codes
> > + * @xfer_func:  transfer function of the data (from enum v4l2_xfer_func), zero
> > + *		on metadata mbus codes
> >   * @flags:	flags (V4L2_MBUS_FRAMEFMT_*)
> >   * @reserved:  reserved bytes that can be later used
> >   */

-- 
Kind regards,

Sakari Ailus

Re: [PATCH v3 08/10] media: v4l: Support line-based metadata capture

[Sakari Ailus]

Hi Jacopo,

On Thu, Aug 10, 2023 at 05:24:14PM +0200, Jacopo Mondi wrote:
> Hi Sakari
> 
> On Tue, Aug 08, 2023 at 10:55:36AM +0300, Sakari Ailus wrote:
> > many camera sensors, among other devices, transmit embedded data and image
> > data for each CSI-2 frame. This embedded data typically contains register
> > configuration of the sensor that has been used to capture the image data
> > of the same frame.
> >
> > The embedded data is received by the CSI-2 receiver and has the same
> > properties as the image data, including that it is line based: it has
> > width, height and bytesperline (stride).
> >
> > Add these fields to struct v4l2_meta_format and document them.
> >
> > Also add V4L2_FMT_FLAG_META_LINE_BASED to tell a given format is
> > line-based i.e. these fields of struct v4l2_meta_format are valid for it.
> >
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  .../userspace-api/media/v4l/dev-meta.rst          | 15 +++++++++++++++
> >  .../userspace-api/media/v4l/vidioc-enum-fmt.rst   |  7 +++++++
> >  .../media/videodev2.h.rst.exceptions              |  1 +
> >  drivers/media/v4l2-core/v4l2-ioctl.c              |  5 +++--
> >  include/uapi/linux/videodev2.h                    | 10 ++++++++++
> >  5 files changed, 36 insertions(+), 2 deletions(-)
> >
> > diff --git a/Documentation/userspace-api/media/v4l/dev-meta.rst b/Documentation/userspace-api/media/v4l/dev-meta.rst
> > index 0e7e1ee1471a..4b24bae6e171 100644
> > --- a/Documentation/userspace-api/media/v4l/dev-meta.rst
> > +++ b/Documentation/userspace-api/media/v4l/dev-meta.rst
> > @@ -65,3 +65,18 @@ to 0.
> >        - ``buffersize``
> >        - Maximum buffer size in bytes required for data. The value is set by the
> >          driver.
> > +    * - __u32
> > +      - ``width``
> > +      - Width of a line of metadata in samples. Valid when :c:type`v4l2_fmtdesc`
> > +	flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
> > +	:c:func:`VIDIOC_ENUM_FMT`.
> > +    * - __u32
> > +      - ``height``
> > +      - Number of rows of metadata. Valid when :c:type`v4l2_fmtdesc` flag
> > +	``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
> > +	:c:func:`VIDIOC_ENUM_FMT`.
> > +    * - __u32
> > +      - ``bytesperline``
> > +      - Offset in bytes between the beginning of two consecutive lines. Valid
> > +	when :c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is
> > +	set, otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
> > diff --git a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> > index 000c154b0f98..6d7664345a4e 100644
> > --- a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> > +++ b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> > @@ -227,6 +227,13 @@ the ``mbus_code`` field is handled differently:
> >  	The application can ask to configure the quantization of the capture
> >  	device when calling the :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl with
> >  	:ref:`V4L2_PIX_FMT_FLAG_SET_CSC <v4l2-pix-fmt-flag-set-csc>` set.
> > +    * - ``V4L2_FMT_FLAG_META_LINE_BASED``
> > +      - 0x0200
> > +      - The metadata format is line-based. In this case the ``width``,
> > +	``height`` and ``bytesperline`` fields of :c:type:`v4l2_meta_format` are
> > +	valid. The buffer consists of ``height`` lines, each having ``width``
> > +	bytes of data and offset between the beginning of each two consecutive
> 
> Isn't ``width`` in samples ?

Indeed, it's better to refer to samples for clarity. I'll fix for v4.

I'll also add bytesperline is in bytes (and not in samples).

> 
> > +	lines is ``bytesperline``.
> >
> >  Return Value
> >  ============
> > diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b/Documentation/userspace-api/media/videodev2.h.rst.exceptions
> > index 3e58aac4ef0b..bdc628e8c1d6 100644
> > --- a/Documentation/userspace-api/media/videodev2.h.rst.exceptions
> > +++ b/Documentation/userspace-api/media/videodev2.h.rst.exceptions
> > @@ -215,6 +215,7 @@ replace define V4L2_FMT_FLAG_CSC_XFER_FUNC fmtdesc-flags
> >  replace define V4L2_FMT_FLAG_CSC_YCBCR_ENC fmtdesc-flags
> >  replace define V4L2_FMT_FLAG_CSC_HSV_ENC fmtdesc-flags
> >  replace define V4L2_FMT_FLAG_CSC_QUANTIZATION fmtdesc-flags
> > +replace define V4L2_FMT_FLAG_META_LINE_BASED fmtdesc-flags
> >
> >  # V4L2 timecode types
> >  replace define V4L2_TC_TYPE_24FPS timecode-type
> > diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> > index fbbddc333a30..971d784e7429 100644
> > --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> > +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> > @@ -343,8 +343,9 @@ static void v4l_print_format(const void *arg, bool write_only)
> >  	case V4L2_BUF_TYPE_META_OUTPUT:
> >  		meta = &p->fmt.meta;
> >  		pixelformat = meta->dataformat;
> > -		pr_cont(", dataformat=%p4cc, buffersize=%u\n",
> > -			&pixelformat, meta->buffersize);
> > +		pr_cont(", dataformat=%p4cc, buffersize=%u, width=%u, height=%u, bytesperline=%u\n",
> > +			&pixelformat, meta->buffersize, meta->width,
> > +			meta->height, meta->bytesperline);
> >  		break;
> >  	}
> >  }
> > diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> > index b4284a564025..d26c0650c6a7 100644
> > --- a/include/uapi/linux/videodev2.h
> > +++ b/include/uapi/linux/videodev2.h
> > @@ -877,6 +877,7 @@ struct v4l2_fmtdesc {
> >  #define V4L2_FMT_FLAG_CSC_YCBCR_ENC		0x0080
> >  #define V4L2_FMT_FLAG_CSC_HSV_ENC		V4L2_FMT_FLAG_CSC_YCBCR_ENC
> >  #define V4L2_FMT_FLAG_CSC_QUANTIZATION		0x0100
> > +#define V4L2_FMT_FLAG_META_LINE_BASED		0x0200
> >
> >  	/* Frame Size and frame rate enumeration */
> >  /*
> > @@ -2420,10 +2421,19 @@ struct v4l2_sdr_format {
> >   * struct v4l2_meta_format - metadata format definition
> >   * @dataformat:		little endian four character code (fourcc)
> >   * @buffersize:		maximum size in bytes required for data
> > + * @width:		number of bytes of data per line (valid for line based
> 
> I'm a bit confused here as well, isn't width in samples ?

I'll change this one as well.

> 
> > + *			formats only, see format documentation)
> > + * @height:		number of lines of data per buffer (valid for line based
> > + *			formats only)
> > + * @bytesperline:	offset between the beginnings of two adjacent lines in
> > + *			bytes (valid for line based formats only)
> >   */
> >  struct v4l2_meta_format {
> >  	__u32				dataformat;
> >  	__u32				buffersize;
> > +	__u32				width;
> > +	__u32				height;
> > +	__u32				bytesperline;
> >  } __attribute__ ((packed));
> >
> >  /**

-- 
Kind regards,

Sakari Ailus

Re: [PATCH v3 04/10] media: uapi: Add generic serial metadata mbus formats

[Tomi Valkeinen]

Hi Sakari,

On 08/08/2023 10:55, Sakari Ailus wrote:
> Add generic serial metadata mbus formats. These formats describe data
> width and packing but not the content itself. The reason for specifying
> such formats is that the formats as such are fairly device specific but
> they are still handled by CSI-2 receiver drivers that should not be aware
> of device specific formats. What makes generic metadata formats possible
> is that these formats are parsed by software only, after capturing the
> data to system memory.

If I'm not mistaken, the CSI-2 spec doesn't say much about embedded 
data, except that it may exist. Afaics, in CSI-2, the embedded data is 
split into "lines", although the amount of data can be different than in 
the video lines.

The CCS specs talks more about embedded data. Some of it is quite odd, 
like: "The length of the embedded data line shall not exceed the length 
of the image data line. The embedded data line should have the same 
length as the image data line.". I think it means the embedded line can 
be shorter than image line, but not longer.

CCS also says that an embedded line should use 0x07 as padding at the 
end of the line, if there's less data than the image line.

CCS also talks about how the embedded data would be packed, and in some 
cases that packing would be the same as for pixel data.

But I don't think these formats are generic. They're defined in CCS, so 
shouldn't the format be, e.g., MEDIA_BUS_FMT_META_CCS_RAW_12 or such?

The MEDIA_BUS_FMT_META_8 is quite safe one, as it just means byte data 
without any padding or packing.

  Tomi

> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>   .../media/v4l/subdev-formats.rst              | 257 ++++++++++++++++++
>   include/uapi/linux/media-bus-format.h         |   9 +
>   2 files changed, 266 insertions(+)
> 
> diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> index a3a35eeed708..c615da08502d 100644
> --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
> +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> @@ -8234,3 +8234,260 @@ The following table lists the existing metadata formats.
>   	both sides of the link and the bus format is a fixed
>   	metadata format that is not configurable from userspace.
>   	Width and height will be set to 0 for this format.
> +
> +Generic Serial Metadata Formats
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +Generic serial metadata formats are used on serial busses where the actual data
> +content is more or less device specific but the data is transmitted and received
> +by multiple devices that do not process the data in any way, simply writing
> +it to system memory for processing in software at the end of the pipeline.
> +
> +The more specific variant describing the actual data is used on the internal
> +source pad of the originating sub-device.
> +
> +"b" in an array cell signifies a byte of data, followed by the number of byte
> +and finally the bit number in subscript. "p" indicates a padding bit.
> +
> +.. _media-bus-format-generic-meta:
> +
> +.. cssclass: longtable
> +
> +.. flat-table:: Generic Serial Metadata Formats
> +    :header-rows:  2
> +    :stub-columns: 0
> +
> +    * - Identifier
> +      - Code
> +      -
> +      - :cspan:`23` Data organization
> +    * -
> +      -
> +      - Bit
> +      - 23
> +      - 22
> +      - 21
> +      - 20
> +      - 19
> +      - 18
> +      - 17
> +      - 16
> +      - 15
> +      - 14
> +      - 13
> +      - 12
> +      - 11
> +      - 10
> +      - 9
> +      - 8
> +      - 7
> +      - 6
> +      - 5
> +      - 4
> +      - 3
> +      - 2
> +      - 1
> +      - 0
> +    * .. _MEDIA-BUS-FMT-META-8:
> +
> +      - MEDIA_BUS_FMT_META_8
> +      - 0x8001
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      - b0\ :sub:`7`
> +      - b0\ :sub:`6`
> +      - b0\ :sub:`5`
> +      - b0\ :sub:`4`
> +      - b0\ :sub:`3`
> +      - b0\ :sub:`2`
> +      - b0\ :sub:`1`
> +      - b0\ :sub:`0`
> +    * .. _MEDIA-BUS-FMT-META-10:
> +
> +      - MEDIA_BUS_FMT_META_10
> +      - 0x8002
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      - b0\ :sub:`7`
> +      - b0\ :sub:`6`
> +      - b0\ :sub:`5`
> +      - b0\ :sub:`4`
> +      - b0\ :sub:`3`
> +      - b0\ :sub:`2`
> +      - b0\ :sub:`1`
> +      - b0\ :sub:`0`
> +      - p
> +      - p
> +    * .. _MEDIA-BUS-FMT-META-12:
> +
> +      - MEDIA_BUS_FMT_META_12
> +      - 0x8003
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      - b0\ :sub:`7`
> +      - b0\ :sub:`6`
> +      - b0\ :sub:`5`
> +      - b0\ :sub:`4`
> +      - b0\ :sub:`3`
> +      - b0\ :sub:`2`
> +      - b0\ :sub:`1`
> +      - b0\ :sub:`0`
> +      - p
> +      - p
> +      - p
> +      - p
> +    * .. _MEDIA-BUS-FMT-META-14:
> +
> +      - MEDIA_BUS_FMT_META_14
> +      - 0x8004
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      - b0\ :sub:`7`
> +      - b0\ :sub:`6`
> +      - b0\ :sub:`5`
> +      - b0\ :sub:`4`
> +      - b0\ :sub:`3`
> +      - b0\ :sub:`2`
> +      - b0\ :sub:`1`
> +      - b0\ :sub:`0`
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +    * .. _MEDIA-BUS-FMT-META-16:
> +
> +      - MEDIA_BUS_FMT_META_16
> +      - 0x8005
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      - b0\ :sub:`7`
> +      - b0\ :sub:`6`
> +      - b0\ :sub:`5`
> +      - b0\ :sub:`4`
> +      - b0\ :sub:`3`
> +      - b0\ :sub:`2`
> +      - b0\ :sub:`1`
> +      - b0\ :sub:`0`
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +    * .. _MEDIA-BUS-FMT-META-20:
> +
> +      - MEDIA_BUS_FMT_META_20
> +      - 0x8007
> +      -
> +      -
> +      -
> +      -
> +      -
> +      - b0\ :sub:`7`
> +      - b0\ :sub:`6`
> +      - b0\ :sub:`5`
> +      - b0\ :sub:`4`
> +      - b0\ :sub:`3`
> +      - b0\ :sub:`2`
> +      - b0\ :sub:`1`
> +      - b0\ :sub:`0`
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +    * .. _MEDIA-BUS-FMT-META-24:
> +
> +      - MEDIA_BUS_FMT_META_24
> +      - 0x8009
> +      -
> +      - b0\ :sub:`7`
> +      - b0\ :sub:`6`
> +      - b0\ :sub:`5`
> +      - b0\ :sub:`4`
> +      - b0\ :sub:`3`
> +      - b0\ :sub:`2`
> +      - b0\ :sub:`1`
> +      - b0\ :sub:`0`
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
> index a03c543cb072..9ee031397372 100644
> --- a/include/uapi/linux/media-bus-format.h
> +++ b/include/uapi/linux/media-bus-format.h
> @@ -173,4 +173,13 @@
>    */
>   #define MEDIA_BUS_FMT_METADATA_FIXED		0x7001
>   
> +/* Generic line based metadata formats for serial buses. Next is 0x8008. */
> +#define MEDIA_BUS_FMT_META_8			0x8001
> +#define MEDIA_BUS_FMT_META_10			0x8002
> +#define MEDIA_BUS_FMT_META_12			0x8003
> +#define MEDIA_BUS_FMT_META_14			0x8004
> +#define MEDIA_BUS_FMT_META_16			0x8005
> +#define MEDIA_BUS_FMT_META_20			0x8006
> +#define MEDIA_BUS_FMT_META_24			0x8007
> +
>   #endif /* __LINUX_MEDIA_BUS_FORMAT_H */

Re: [PATCH v3 04/10] media: uapi: Add generic serial metadata mbus formats

[Sakari Ailus]

Moi,

On Wed, Aug 23, 2023 at 04:16:13PM +0300, Tomi Valkeinen wrote:
> Hi Sakari,
> 
> On 08/08/2023 10:55, Sakari Ailus wrote:
> > Add generic serial metadata mbus formats. These formats describe data
> > width and packing but not the content itself. The reason for specifying
> > such formats is that the formats as such are fairly device specific but
> > they are still handled by CSI-2 receiver drivers that should not be aware
> > of device specific formats. What makes generic metadata formats possible
> > is that these formats are parsed by software only, after capturing the
> > data to system memory.
> 
> If I'm not mistaken, the CSI-2 spec doesn't say much about embedded data,
> except that it may exist. Afaics, in CSI-2, the embedded data is split into
> "lines", although the amount of data can be different than in the video
> lines.
> 
> The CCS specs talks more about embedded data. Some of it is quite odd, like:
> "The length of the embedded data line shall not exceed the length of the
> image data line. The embedded data line should have the same length as the
> image data line.". I think it means the embedded line can be shorter than
> image line, but not longer.

That's what it means, yes. The CCS also has means to obtain the actual
length --- frame format descriptors.

> 
> CCS also says that an embedded line should use 0x07 as padding at the end of
> the line, if there's less data than the image line.
> 
> CCS also talks about how the embedded data would be packed, and in some
> cases that packing would be the same as for pixel data.

In fact the packing is the same as for pixel data: the CSI-2 does not
really differentiate there.

> 
> But I don't think these formats are generic. They're defined in CCS, so
> shouldn't the format be, e.g., MEDIA_BUS_FMT_META_CCS_RAW_12 or such?

The reason for having generic definitions is that we do not need receiver
drivers to be aware of formats that are specific to another driver.

If there is a need for new generic formats that do not match this, we can
always add more. But the point is: drivers for devices that do not produce
the data should never deal with (device) specific formats.

What comes to CSI-2 and these formats --- on parallel buses you might have
the data aligned to the least significant bits instead. But is there a need
to transport such data on parallel buses, at least so it would be expressed
in mbus formats?

> 
> The MEDIA_BUS_FMT_META_8 is quite safe one, as it just means byte data
> without any padding or packing.

As you're always dealing with 8 bits only, there is of course less room for
variation.

-- 
Regards,

Sakari Ailus

Re: [PATCH v3 04/10] media: uapi: Add generic serial metadata mbus formats

[Tomi Valkeinen]

On 24/08/2023 10:24, Sakari Ailus wrote:
> Moi,
> 
> On Wed, Aug 23, 2023 at 04:16:13PM +0300, Tomi Valkeinen wrote:
>> Hi Sakari,
>>
>> On 08/08/2023 10:55, Sakari Ailus wrote:
>>> Add generic serial metadata mbus formats. These formats describe data
>>> width and packing but not the content itself. The reason for specifying
>>> such formats is that the formats as such are fairly device specific but
>>> they are still handled by CSI-2 receiver drivers that should not be aware
>>> of device specific formats. What makes generic metadata formats possible
>>> is that these formats are parsed by software only, after capturing the
>>> data to system memory.
>>
>> If I'm not mistaken, the CSI-2 spec doesn't say much about embedded data,
>> except that it may exist. Afaics, in CSI-2, the embedded data is split into
>> "lines", although the amount of data can be different than in the video
>> lines.
>>
>> The CCS specs talks more about embedded data. Some of it is quite odd, like:
>> "The length of the embedded data line shall not exceed the length of the
>> image data line. The embedded data line should have the same length as the
>> image data line.". I think it means the embedded line can be shorter than
>> image line, but not longer.
> 
> That's what it means, yes. The CCS also has means to obtain the actual
> length --- frame format descriptors.
> 
>>
>> CCS also says that an embedded line should use 0x07 as padding at the end of
>> the line, if there's less data than the image line.
>>
>> CCS also talks about how the embedded data would be packed, and in some
>> cases that packing would be the same as for pixel data.
> 
> In fact the packing is the same as for pixel data: the CSI-2 does not
> really differentiate there.

If I understand the CCS spec right, the packing is not the same as for 
the pixel data. You can have RAW12 pixel data but 8-byte embedded data. 
But if you meant that the different packing style options are the same 
for pixel and embedded data, yes.

>> But I don't think these formats are generic. They're defined in CCS, so
>> shouldn't the format be, e.g., MEDIA_BUS_FMT_META_CCS_RAW_12 or such?
> 
> The reason for having generic definitions is that we do not need receiver
> drivers to be aware of formats that are specific to another driver.

Yes, I agree with that, and that's not my point here. But "generic" 
doesn't mean the definitions are not for some particular bus or 
standard, "generic" just means that it doesn't specify the content, only 
the packing.

My point is that these packings seem to be specific to CCS. While 
non-CCS compliant sensors may use the same packing, the packing itself 
is still a "CCS packing". So why not call them that? The 8-bit format is 
fully generic, whereas the rest are CCS packings, as defined in the CCS 
spec (the CCS spec also specifies the content, but here were only using 
the CCS packing).

Maybe they shouldn't be called "generic", but... umm... Content unaware 
metadata formats... doesn't sound very good =).

Also, I'm a bit confused about CSI-2 pixel and embedded data formats and 
how we handle them.

For MEDIA_BUS_FMT_SBGGR10_1X10, we define that the data contains 10 bits 
per pixel, from bit 0 to 9. But CSI-2, for RAW10, actually sends it 
differently, with the higher bits first, and the lowest bits in the 
fourth byte. So that CSI-2 packing is implicit, kind of hidden here. 
Which is probably fine, as we're really only interested in the unpacked 
data, not the CSI-2 packed. And when writing this data to memory, the 
DMA engine can write it either as is, or unpack each pixel to a 16-bit 
container.

For MEDIA_BUS_FMT_META_10, we define it similarly to 
MEDIA_BUS_FMT_SBGGR10_1X10, except the lowest 2 bits are marked as 
padding bits. And, I think, MEDIA_BUS_FMT_META_10 implies RAW10 CSI-2 
format. However, when writing the data to memory, we don't want either 
of the modes used in the above pixel data case, but rather we want to 
write the data as it is in the CSI-2 bus. So, the DMA engine can either 
reverse the RAW10 unpacking to get the wanted output format, or 
alternatively the CSI-2 receiver could instead use RAW8 mode to avoid 
any unpacking.

Does the above make any sense? I'm a bit confused about all the packings 
and unpackings =). Does MEDIA_BUS_FMT_META_10 mean that the CSI-2 TX 
uses RAW10 CSI-2 packing, but the receiver uses RAW8 CSI-2 packing?

> If there is a need for new generic formats that do not match this, we can
> always add more. But the point is: drivers for devices that do not produce
> the data should never deal with (device) specific formats.
> 
> What comes to CSI-2 and these formats --- on parallel buses you might have
> the data aligned to the least significant bits instead. But is there a need
> to transport such data on parallel buses, at least so it would be expressed
> in mbus formats?

I have a parallel sensor that sends embedded data. I'll have a look how 
it organizes the data.

  Tomi

Re: [PATCH v3 06/10] media: uapi: Add a macro to tell whether an mbus code is metadata

[Tomi Valkeinen]

Hi,

On 08/08/2023 10:55, Sakari Ailus wrote:
> Add a macro to tell whether a given mbus code is metadata.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>   include/uapi/linux/media-bus-format.h | 3 +++
>   1 file changed, 3 insertions(+)
> 
> diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
> index 9ee031397372..2486b4178c5f 100644
> --- a/include/uapi/linux/media-bus-format.h
> +++ b/include/uapi/linux/media-bus-format.h
> @@ -182,4 +182,7 @@
>   #define MEDIA_BUS_FMT_META_20			0x8006
>   #define MEDIA_BUS_FMT_META_24			0x8007
>   
> +#define MEDIA_BUS_FMT_IS_META(code)		\
> +	((code) & 0xf000 == 0x7000 || (code) & 0xf000 == 0x8000)
> +
>   #endif /* __LINUX_MEDIA_BUS_FORMAT_H */

mbus code seems to be u32, so the above won't work. Maybe:

(((code) & ~0xfffULL) == 0x7000 || ((code) & ~0xfffULL) == 0x8000)

Also, embedded formats with 0x9nnn codes are added later in the series.

  Tomi

Re: [PATCH v3 06/10] media: uapi: Add a macro to tell whether an mbus code is metadata

[Sakari Ailus]

Moi,

On Tue, Sep 05, 2023 at 12:47:21PM +0300, Tomi Valkeinen wrote:
> Hi,
> 
> On 08/08/2023 10:55, Sakari Ailus wrote:
> > Add a macro to tell whether a given mbus code is metadata.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >   include/uapi/linux/media-bus-format.h | 3 +++
> >   1 file changed, 3 insertions(+)
> > 
> > diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
> > index 9ee031397372..2486b4178c5f 100644
> > --- a/include/uapi/linux/media-bus-format.h
> > +++ b/include/uapi/linux/media-bus-format.h
> > @@ -182,4 +182,7 @@
> >   #define MEDIA_BUS_FMT_META_20			0x8006
> >   #define MEDIA_BUS_FMT_META_24			0x8007
> > +#define MEDIA_BUS_FMT_IS_META(code)		\
> > +	((code) & 0xf000 == 0x7000 || (code) & 0xf000 == 0x8000)
> > +
> >   #endif /* __LINUX_MEDIA_BUS_FORMAT_H */
> 
> mbus code seems to be u32, so the above won't work. Maybe:
> 
> (((code) & ~0xfffULL) == 0x7000 || ((code) & ~0xfffULL) == 0x8000)
> 
> Also, embedded formats with 0x9nnn codes are added later in the series.

Thanks, I'll address these for v4.

-- 
Sakari Ailus

Re: [PATCH v3 01/10] media: Documentation: Align numbered list

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Tue, Aug 08, 2023 at 10:55:29AM +0300, Sakari Ailus wrote:
> Align lines for numbered list so that Sphinx produces an uniform output
> for all list entries.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../userspace-api/media/v4l/dev-subdev.rst    | 27 +++++++++----------
>  1 file changed, 13 insertions(+), 14 deletions(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> index a4f1df7093e8..b023918177b5 100644
> --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> @@ -579,20 +579,19 @@ is started.
>  
>  There are three steps in configuring the streams:
>  
> -1) Set up links. Connect the pads between sub-devices using the :ref:`Media
> -Controller API <media_controller>`
> +1) Set up links. Connect the pads between sub-devices using the
> +   :ref:`Media Controller API <media_controller>`

While at it, could we standardize on 1. vs. 1) ? Both are used in the
same file. After this patch they both render to "1.", but it would be
nice to be consistent in the source too.

>  
> -2) Streams. Streams are declared and their routing is configured by
> -setting the routing table for the sub-device using
> -:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that
> -setting the routing table will reset formats and selections in the
> -sub-device to default values.
> +2) Streams. Streams are declared and their routing is configured by setting the
> +   routing table for the sub-device using :ref:`VIDIOC_SUBDEV_S_ROUTING
> +   <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that setting the routing table will
> +   reset formats and selections in the sub-device to default values.
>  
> -3) Configure formats and selections. Formats and selections of each stream
> -are configured separately as documented for plain sub-devices in
> -:ref:`format-propagation`. The stream ID is set to the same stream ID
> -associated with either sink or source pads of routes configured using the
> -:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl.
> +3) Configure formats and selections. Formats and selections of each stream are
> +   configured separately as documented for plain sub-devices in
> +   :ref:`format-propagation`. The stream ID is set to the same stream ID
> +   associated with either sink or source pads of routes configured using the
> +   :ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl.
>  
>  Multiplexed streams setup example
>  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> @@ -619,8 +618,8 @@ modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
>  To configure this pipeline, the userspace must take the following steps:
>  
>  1) Set up media links between entities: connect the sensors to the bridge,
> -bridge to the receiver, and the receiver to the DMA engines. This step does
> -not differ from normal non-multiplexed media controller setup.
> +   bridge to the receiver, and the receiver to the DMA engines. This step does
> +   not differ from normal non-multiplexed media controller setup.
>  
>  2) Configure routing
>  

3) below contains two paragraphs that are not indented.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v3 02/10] media: mc: Check pad flag validity

[Laurent Pinchart]

On Mon, Aug 14, 2023 at 08:49:47AM +0000, Sakari Ailus wrote:
> On Thu, Aug 10, 2023 at 03:54:40PM +0200, Jacopo Mondi wrote:
> > On Tue, Aug 08, 2023 at 10:55:30AM +0300, Sakari Ailus wrote:
> > > Check the validity of pad flags on entity init. Exactly one of the flags
> > > must be set.
> > >
> > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > ---
> > >  drivers/media/mc/mc-entity.c | 11 ++++++++++-
> > >  1 file changed, 10 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> > > index 83468d4a440b..4991281dcccc 100644
> > > --- a/drivers/media/mc/mc-entity.c
> > > +++ b/drivers/media/mc/mc-entity.c
> > > @@ -197,6 +197,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
> > >  	struct media_device *mdev = entity->graph_obj.mdev;
> > >  	struct media_pad *iter;
> > >  	unsigned int i = 0;
> > > +	int ret = 0;
> > >
> > >  	if (num_pads >= MEDIA_ENTITY_MAX_PADS)
> > >  		return -E2BIG;
> > > @@ -210,6 +211,14 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
> > >  	media_entity_for_each_pad(entity, iter) {
> > >  		iter->entity = entity;
> > >  		iter->index = i++;
> > > +
> > > +		if (hweight32(iter->flags & (MEDIA_PAD_FL_SINK |
> > > +					     MEDIA_PAD_FL_SOURCE))
> > > +		    != 1) {
> > 
> > Fits on the previous line, if you like it
> 
> Not quite.

The '!= 1) {' does. unless you want to lower the line length limit to 73
or less :-)

> > > +			ret = -EINVAL;
> > > +			break;
> > 
> > Now that we can interrupt the for loop earlier, do you need to cleanup
> > objects appended to the mdev list with the media_gobj_create() call here below ?
> 
> Nice find. I'll fix this for v4.
> 
> > > +		}
> > > +
> > >  		if (mdev)
> > >  			media_gobj_create(mdev, MEDIA_GRAPH_PAD,
> > >  					  &iter->graph_obj);
> > > @@ -218,7 +227,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
> > >  	if (mdev)
> > >  		mutex_unlock(&mdev->graph_mutex);
> > >
> > > -	return 0;
> > > +	return ret;
> > >  }
> > >  EXPORT_SYMBOL_GPL(media_entity_pads_init);
> > >

-- 
Regards,

Laurent Pinchart

Re: [PATCH v3 03/10] media: mc: Add INTERNAL pad flag

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Tue, Aug 08, 2023 at 10:55:31AM +0300, Sakari Ailus wrote:
> Internal source pads will be used as routing endpoints in V4L2
> [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
> 
> Also prevent creating links to pads that have been flagged as internal.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  Documentation/userspace-api/media/glossary.rst             | 6 ++++++
>  Documentation/userspace-api/media/mediactl/media-types.rst | 6 ++++++
>  drivers/media/mc/mc-entity.c                               | 6 +++++-
>  include/uapi/linux/media.h                                 | 1 +
>  4 files changed, 18 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/userspace-api/media/glossary.rst b/Documentation/userspace-api/media/glossary.rst
> index 96a360edbf3b..f7b99a4527c7 100644
> --- a/Documentation/userspace-api/media/glossary.rst
> +++ b/Documentation/userspace-api/media/glossary.rst
> @@ -173,6 +173,12 @@ Glossary
>  	An integrated circuit that integrates all components of a computer
>  	or other electronic systems.
>  
> +_media-glossary-stream:
> +    Stream
> +	A distinct flow of data (image data or metadata) over a media pipeline
> +	from source to sink. A source may be e.g. an image sensor and a sink
> +	e.g. a memory buffer.
> +
>      V4L2 API
>  	**V4L2 userspace API**
>  
> diff --git a/Documentation/userspace-api/media/mediactl/media-types.rst b/Documentation/userspace-api/media/mediactl/media-types.rst
> index 0ffeece1e0c8..28941da27790 100644
> --- a/Documentation/userspace-api/media/mediactl/media-types.rst
> +++ b/Documentation/userspace-api/media/mediactl/media-types.rst
> @@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
>  .. _MEDIA-PAD-FL-SINK:
>  .. _MEDIA-PAD-FL-SOURCE:
>  .. _MEDIA-PAD-FL-MUST-CONNECT:
> +.. _MEDIA-PAD-FL-INTERNAL:
>  
>  .. flat-table:: Media pad flags
>      :header-rows:  0
> @@ -382,6 +383,11 @@ Types and flags used to represent the media graph elements
>  	  when this flag isn't set; the absence of the flag doesn't imply
>  	  there is none.
>  
> +    *  -  ``MEDIA_PAD_FL_INTERNAL``
> +       -  The internal flag indicates an internal pad that has no external
> +	  connections. Such a pad shall not be connected with a link. The
> +	  internal flag indicates that the :ref:``stream
> +	  <media-glossary-stream>`` either starts or ends in the entity.
>  
>  One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
>  must be set for every pad.

You also need to update the definition of "pad" in
media-controller-model.rst. That's where you should explain what an
internal pad *is*, with enough information to make it understandable to
people who haven't worked on this patches series :-) You'll probably
need a couple of paragraphs, or possibly adding text below the list of
objects.

It would be nice to fix the indentation inconsistency in that file by
the way. I'd send a patch, but it will conflict, it would be easier if
you could handle it.

> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> index 4991281dcccc..03a9e0b8ebab 100644
> --- a/drivers/media/mc/mc-entity.c
> +++ b/drivers/media/mc/mc-entity.c
> @@ -1071,7 +1071,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
>  
>  	for (i = 0; i < entity->num_pads; i++) {
>  		if ((entity->pads[i].flags &
> -		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
> +		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
> +		      MEDIA_PAD_FL_INTERNAL)) != pad_type)
>  			continue;
>  
>  		if (entity->pads[i].sig_type == sig_type)
> @@ -1094,6 +1095,9 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
>  		return -EINVAL;
>  	if (WARN_ON(!(source->pads[source_pad].flags & MEDIA_PAD_FL_SOURCE)))
>  		return -EINVAL;
> +	if (WARN_ON(source->pads[source_pad].flags & MEDIA_PAD_FL_SOURCE &&
> +		    source->pads[source_pad].flags & MEDIA_PAD_FL_INTERNAL))
> +		return -EINVAL;
>  	if (WARN_ON(!(sink->pads[sink_pad].flags & MEDIA_PAD_FL_SINK)))
>  		return -EINVAL;
>  
> diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
> index 1c80b1d6bbaf..80cfd12a43fc 100644
> --- a/include/uapi/linux/media.h
> +++ b/include/uapi/linux/media.h
> @@ -208,6 +208,7 @@ struct media_entity_desc {
>  #define MEDIA_PAD_FL_SINK			(1U << 0)
>  #define MEDIA_PAD_FL_SOURCE			(1U << 1)
>  #define MEDIA_PAD_FL_MUST_CONNECT		(1U << 2)
> +#define MEDIA_PAD_FL_INTERNAL			(1U << 3)
>  
>  struct media_pad_desc {
>  	__u32 entity;		/* entity ID */

-- 
Regards,

Laurent Pinchart

Re: [PATCH v3 04/10] media: uapi: Add generic serial metadata mbus formats

[Laurent Pinchart]

On Thu, Aug 24, 2023 at 11:26:56AM +0300, Tomi Valkeinen wrote:
> On 24/08/2023 10:24, Sakari Ailus wrote:
> > On Wed, Aug 23, 2023 at 04:16:13PM +0300, Tomi Valkeinen wrote:
> >> On 08/08/2023 10:55, Sakari Ailus wrote:
> >>> Add generic serial metadata mbus formats. These formats describe data
> >>> width and packing but not the content itself. The reason for specifying
> >>> such formats is that the formats as such are fairly device specific but
> >>> they are still handled by CSI-2 receiver drivers that should not be aware

Do we want these formats to be CSI-2-specific ?

> >>> of device specific formats. What makes generic metadata formats possible
> >>> is that these formats are parsed by software only, after capturing the
> >>> data to system memory.
> >>
> >> If I'm not mistaken, the CSI-2 spec doesn't say much about embedded data,
> >> except that it may exist. Afaics, in CSI-2, the embedded data is split into
> >> "lines", although the amount of data can be different than in the video
> >> lines.
> >>
> >> The CCS specs talks more about embedded data. Some of it is quite odd, like:
> >> "The length of the embedded data line shall not exceed the length of the
> >> image data line. The embedded data line should have the same length as the
> >> image data line.". I think it means the embedded line can be shorter than
> >> image line, but not longer.
> > 
> > That's what it means, yes. The CCS also has means to obtain the actual
> > length --- frame format descriptors.
> > 
> >> CCS also says that an embedded line should use 0x07 as padding at the end of
> >> the line, if there's less data than the image line.
> >>
> >> CCS also talks about how the embedded data would be packed, and in some
> >> cases that packing would be the same as for pixel data.
> > 
> > In fact the packing is the same as for pixel data: the CSI-2 does not
> > really differentiate there.
> 
> If I understand the CCS spec right, the packing is not the same as for 
> the pixel data. You can have RAW12 pixel data but 8-byte embedded data. 
> But if you meant that the different packing style options are the same 
> for pixel and embedded data, yes.

The idea is to allow CSI-2 receivers to unpack embedded data the same
way as pixel data at the hardware level, and guarantee that no embedded
data significant byte gets split across multiple samples. When unpacking
RAW10 pixel data, the CSI-2 receiver will take 5 bytes and output 4
samples on a 10-bit wide bus. For embedded data, it can conceptually do
that same, and output embedded data with one byte per sample, aligned on
the MSB of the 10-bit bus. However, in practice, I think the CSI-2
receiver will simply receive the bytes and send them to the DMA engine,
without unpacking and then repacking.

This can also be viewed as away to allow CSI-2 transmitters to serialize
pixel data and embedded data the same way. For a RAW10 sensor, for
instance, the embedded data would be generated on an internal 10-bit
wide bus, aligned to the MSB, and would go through CSI-2 packing the
same way that pixels do. That's nice, but it's internal to the sensor,
so we don't need to expose it to userspace on that side.

> >> But I don't think these formats are generic. They're defined in CCS, so
> >> shouldn't the format be, e.g., MEDIA_BUS_FMT_META_CCS_RAW_12 or such?
> > 
> > The reason for having generic definitions is that we do not need receiver
> > drivers to be aware of formats that are specific to another driver.
> 
> Yes, I agree with that, and that's not my point here. But "generic" 
> doesn't mean the definitions are not for some particular bus or 
> standard, "generic" just means that it doesn't specify the content, only 
> the packing.
> 
> My point is that these packings seem to be specific to CCS. While 
> non-CCS compliant sensors may use the same packing, the packing itself 
> is still a "CCS packing".

Agreed.

> So why not call them that? The 8-bit format is 
> fully generic, whereas the rest are CCS packings, as defined in the CCS 
> spec (the CCS spec also specifies the content, but here were only using 
> the CCS packing).

I think an additional question is whether we actually need multiple
media bus codes here. Are there CSI-2 receivers, or other downstream
blocks, that will need to be configured differently to receive embedded
data from a CCS (in the sense of using CCS packing, not being fully
CCS-compliant) RAW10 sensor compared to a RAW12 sensor ? Or will the
CSI-2 receiver just send bytes to memory without caring ? In the latter
case, it will be the application that will need to know how the data is
packed to ignore one byte out of three for RAW12 and out of five for
RAW10.

> Maybe they shouldn't be called "generic", but... umm... Content unaware 
> metadata formats... doesn't sound very good =).
> 
> Also, I'm a bit confused about CSI-2 pixel and embedded data formats and 
> how we handle them.
> 
> For MEDIA_BUS_FMT_SBGGR10_1X10, we define that the data contains 10 bits 
> per pixel, from bit 0 to 9. But CSI-2, for RAW10, actually sends it 
> differently, with the higher bits first, and the lowest bits in the 
> fourth byte. So that CSI-2 packing is implicit, kind of hidden here. 

Usage of MEDIA_BUS_FMT_SBGGR10_1X10 for CSI-2 is a historical decision
(or mistake, depending on how you look at it). It doesn't match what
actually happens on the bus.

> Which is probably fine, as we're really only interested in the unpacked 
> data, not the CSI-2 packed. And when writing this data to memory, the 
> DMA engine can write it either as is, or unpack each pixel to a 16-bit 
> container.
> 
> For MEDIA_BUS_FMT_META_10, we define it similarly to 
> MEDIA_BUS_FMT_SBGGR10_1X10, except the lowest 2 bits are marked as 
> padding bits. And, I think, MEDIA_BUS_FMT_META_10 implies RAW10 CSI-2 
> format.

That's how I understand it too

> However, when writing the data to memory, we don't want either 
> of the modes used in the above pixel data case, but rather we want to 
> write the data as it is in the CSI-2 bus. So, the DMA engine can either 
> reverse the RAW10 unpacking to get the wanted output format, or 
> alternatively the CSI-2 receiver could instead use RAW8 mode to avoid 
> any unpacking.

I don't think the DMA engine would "reverse the RAW10 packing" here. It
will write bytes as they arrive (or more likely, a set of bytes at a
time to operate on bursts). The other option would be for the DMA engine
to drop the padding bytes, but I don't know of any CSI-2 receiver that
can do that.

> Does the above make any sense? I'm a bit confused about all the packings 
> and unpackings =). Does MEDIA_BUS_FMT_META_10 mean that the CSI-2 TX 
> uses RAW10 CSI-2 packing, but the receiver uses RAW8 CSI-2 packing?
> 
> > If there is a need for new generic formats that do not match this, we can
> > always add more. But the point is: drivers for devices that do not produce
> > the data should never deal with (device) specific formats.
> > 
> > What comes to CSI-2 and these formats --- on parallel buses you might have
> > the data aligned to the least significant bits instead. But is there a need
> > to transport such data on parallel buses, at least so it would be expressed
> > in mbus formats?
> 
> I have a parallel sensor that sends embedded data. I'll have a look how 
> it organizes the data.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v3 05/10] media: uapi: Document which mbus format fields are valid for metadata

[Laurent Pinchart]

On Mon, Aug 14, 2023 at 10:23:59AM +0000, Sakari Ailus wrote:
> On Thu, Aug 10, 2023 at 05:19:02PM +0200, Jacopo Mondi wrote:
> > On Tue, Aug 08, 2023 at 10:55:33AM +0300, Sakari Ailus wrote:
> > > Now that metadata mbus formats have been added, it is necessary to define
> > > which fields in struct v4l2_mbus_format are applicable to them (not many).
> > >
> > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > ---
> > >  include/uapi/linux/v4l2-mediabus.h | 18 ++++++++++++------
> > >  1 file changed, 12 insertions(+), 6 deletions(-)
> > >
> > > diff --git a/include/uapi/linux/v4l2-mediabus.h b/include/uapi/linux/v4l2-mediabus.h
> > > index 6b07b73473b5..3cadb3b58b85 100644
> > > --- a/include/uapi/linux/v4l2-mediabus.h
> > > +++ b/include/uapi/linux/v4l2-mediabus.h
> > > @@ -19,12 +19,18 @@
> > >   * @width:	image width
> > >   * @height:	image height
> > >   * @code:	data format code (from enum v4l2_mbus_pixelcode)
> > > - * @field:	used interlacing type (from enum v4l2_field)
> > > - * @colorspace:	colorspace of the data (from enum v4l2_colorspace)
> > > - * @ycbcr_enc:	YCbCr encoding of the data (from enum v4l2_ycbcr_encoding)
> > > - * @hsv_enc:	HSV encoding of the data (from enum v4l2_hsv_encoding)
> > > - * @quantization: quantization of the data (from enum v4l2_quantization)
> > > - * @xfer_func:  transfer function of the data (from enum v4l2_xfer_func)
> > > + * @field:	used interlacing type (from enum v4l2_field), not applicable
> > > + *		to metadata mbus codes
> > 
> > "not applicable" is a bit geeric. Should this be set to
> > V4L2_FIELD_NONE (for metadata, and progressive image formats maybe ?)
> 
> I actually intended to have the same wording here than for the other fields
> but missed changing this.
> 
> 0 corresponds to V4L2_FIELD_ANY.
> 
> > > + * @colorspace:	colorspace of the data (from enum v4l2_colorspace), zero on
> > > + *		metadata mbus codes
> > > + * @ycbcr_enc:	YCbCr encoding of the data (from enum v4l2_ycbcr_encoding), zero
> > > + *		on metadata mbus codes
> > > + * @hsv_enc:	HSV encoding of the data (from enum v4l2_hsv_encoding), zero on
> > > + *		metadata mbus codes
> > 
> > Can this be zero ?
> > 
> > enum v4l2_hsv_encoding {
> > 
> > 	/* Hue mapped to 0 - 179 */
> > 	V4L2_HSV_ENC_180		= 128,
> > 
> > 	/* Hue mapped to 0-255 */
> > 	V4L2_HSV_ENC_256		= 129,
> > };
> 
> Good question. Neither value is meaningful for metadata.
> 
> The values appear to be such in the enum to avoid colliding with ycbcr
> encoding values (another field above).
> 
> Generally, what doesn't matter will be zero. These fields have been added
> at some point and a lot of drivers do not set them, even for pixel data.
> 
> I wonder what Hans and Laurent think.

ycbcr_enc and hsv_enc are stored in an unnamed union.

> > > + * @quantization: quantization of the data (from enum v4l2_quantization), zero
> > > + *		on metadata mbus codes
> > > + * @xfer_func:  transfer function of the data (from enum v4l2_xfer_func), zero
> > > + *		on metadata mbus codes
> > >   * @flags:	flags (V4L2_MBUS_FRAMEFMT_*)
> > >   * @reserved:  reserved bytes that can be later used
> > >   */

-- 
Regards,

Laurent Pinchart

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

[Laurent Pinchart]

Hi Sakari,

On Fri, Aug 11, 2023 at 09:11:39AM +0000, Sakari Ailus wrote:
> On Fri, Aug 11, 2023 at 08:31:16AM +0200, Jacopo Mondi wrote:
> > > > +V4L2_META_FMT_GENERIC_CSI2_10
> > > > +-----------------------------
> > > > +
> > > > +V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
> > > > +for each 8 bits of data. Every four bytes of metadata is followed by a single
> > > > +byte of padding. The way the data is stored follows the CSI-2 specification.
> > > > +
> > > > +This format is also used on CSI-2 on 20 bits per sample format that packs two
> > > > +bytes of metadata into one sample.
> > > > +
> > > > +This format is little endian.
> > > > +
> > > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
> > > > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > >
> > > I think you should document whether the padding is always 0 or can be any value.
> > > Perhaps 'X' is a better 'name' for the padding byte in the latter case.
> > 
> > Did I get this right that this format is supposed to work as the RAW10
> > CSI-2 packed image format, where 4 bytes contain the higher 8 bits of
> > the 10 bits sample and the 5th byte every 4 contains the lower 2 bits of
> > the previous 4 sample ?
> > 
> > If that's the case, is 'padding' the correct term here ?
> 
> What else would you call it? It'll be zeros that exist just due to the bit
> depth used and as such not interesting at all.

It's actually not 0, CCS requires the padding bytes to be 0x55.

I wonder if the conformance test suite tests the contents of the padding
bytes.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

[Laurent Pinchart]

On Tue, Aug 08, 2023 at 10:55:35AM +0300, Sakari Ailus wrote:
> Generic 8-bit metadata formats define the in-memory data layout but not
> the format of the data itself. The reasoning for having such formats is to
> allow CSI-2 receiver drivers to receive and DMA drivers to write the data
> to memory without knowing a large number of device specific formats.
> 
> These formats may be used only in conjunction of a Media controller
> pipeline where the internal pad of the source sub-device defines the
> specific format of the data (using an mbus code).
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../userspace-api/media/v4l/meta-formats.rst  |   1 +
>  .../media/v4l/metafmt-generic.rst             | 331 ++++++++++++++++++
>  drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
>  include/uapi/linux/videodev2.h                |   9 +
>  4 files changed, 349 insertions(+)
>  create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
> 
> diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
> index 0bb61fc5bc00..919f595576b9 100644
> --- a/Documentation/userspace-api/media/v4l/meta-formats.rst
> +++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
> @@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only.
>      metafmt-vsp1-hgo
>      metafmt-vsp1-hgt
>      metafmt-vivid
> +    metafmt-generic
> diff --git a/Documentation/userspace-api/media/v4l/metafmt-generic.rst b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> new file mode 100644
> index 000000000000..a27bfc721edf
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> @@ -0,0 +1,331 @@
> +.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
> +
> +**************************************************************************************************************************************************************************************************************************************************************************************************************************
> +V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O'), V4L2_META_FMT_GENERIC_CSI2_2_24 ('MC2O')
> +**************************************************************************************************************************************************************************************************************************************************************************************************************************
> +
> +
> +Generic line-based metadata formats
> +
> +
> +Description
> +===========
> +
> +These generic line-based metadata formats define the memory layout of the data
> +without defining the format or meaning of the metadata itself. These formats may
> +only be used with a Media controller pipeline where the more specific format is
> +defined in an :ref:`internal source pad <MEDIA-PAD-FL-INTERNAL>` of the source
> +sub-device. See also :ref:`source routes <v4l2-subdev-source-routes>`.
> +
> +.. _v4l2-meta-fmt-generic-8:
> +
> +V4L2_META_FMT_GENERIC_8
> +-----------------------
> +
> +The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format.
> +
> +This format is also used on CSI-2 on both 8 bits per sample as well as on

s/also on/by/

I would also mention "MIPI CCS" instead of "CSI-2".

> +16 bits per sample when two bytes of metadata are packed into one sample.

"bits per sample" is very ill-defined for metadata, as there's no
sample. I would write "for both the RAW8 packing and the 2 bytes RAW16
packing" or something similar.

Similar comments for below.

> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_8.**
> +Each cell is one byte. "M" denotes a byte of metadata.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|
> +
> +.. flat-table::
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - M\ :sub:`10`
> +      - M\ :sub:`20`
> +      - M\ :sub:`30`
> +    * - start + 4:
> +      - M\ :sub:`01`
> +      - M\ :sub:`11`
> +      - M\ :sub:`21`
> +      - M\ :sub:`31`
> +
> +.. _v4l2-meta-fmt-generic-csi2-10:
> +
> +V4L2_META_FMT_GENERIC_CSI2_10
> +-----------------------------
> +
> +V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
> +for each 8 bits of data. Every four bytes of metadata is followed by a single
> +byte of padding.

It sounds really weird to write that this format writes 10 bits for each
8 bits of data, when essentially it adds a packing byte every four
bytes.

> The way the data is stored follows the CSI-2 specification.

Again, it's CCS, not CSI-2.

> +
> +This format is also used on CSI-2 on 20 bits per sample format that packs two
> +bytes of metadata into one sample.
> +
> +This format is little endian.
> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
> +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
> +
> +.. flat-table::
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - M\ :sub:`10`
> +      - M\ :sub:`20`
> +      - M\ :sub:`30`
> +      - p
> +    * - start + 5:
> +      - M\ :sub:`01`
> +      - M\ :sub:`11`
> +      - M\ :sub:`21`
> +      - M\ :sub:`31`
> +      - p
> +
> +.. _v4l2-meta-fmt-generic-csi2-12:
> +
> +V4L2_META_FMT_GENERIC_CSI2_12
> +-----------------------------
> +
> +V4L2_META_FMT_GENERIC_CSI2_12 contains packed 8-bit generic metadata, 12 bits
> +for each 8 bits of data. Every four bytes of metadata is followed by two bytes
> +of padding. The way the data is stored follows the CSI-2 specification.
> +
> +This format is little endian.
> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_12.**
> +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> +
> +.. flat-table::
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - M\ :sub:`10`
> +      - M\ :sub:`20`
> +      - M\ :sub:`30`
> +      - p
> +      - p
> +    * - start + 6:
> +      - M\ :sub:`01`
> +      - M\ :sub:`11`
> +      - M\ :sub:`21`
> +      - M\ :sub:`31`
> +      - p
> +      - p
> +
> +.. _v4l2-meta-fmt-generic-csi2-14:
> +
> +V4L2_META_FMT_GENERIC_CSI2_14
> +-----------------------------
> +
> +V4L2_META_FMT_GENERIC_CSI2_14 contains packed 8-bit generic metadata, 14 bits
> +for each 8 bits of data. Every four bytes of metadata is followed by three
> +bytes of padding. The way the data is stored follows the CSI-2 specification.
> +
> +This format is little endian.
> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_14.**
> +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{.8cm}|
> +
> +.. flat-table::
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - M\ :sub:`10`
> +      - M\ :sub:`20`
> +      - M\ :sub:`30`
> +      - p
> +      - p
> +      - p
> +    * - start + 7:
> +      - M\ :sub:`01`
> +      - M\ :sub:`11`
> +      - M\ :sub:`21`
> +      - M\ :sub:`31`
> +      - p
> +      - p
> +      - p
> +
> +.. _v4l2-meta-fmt-generic-csi2-16:
> +
> +V4L2_META_FMT_GENERIC_CSI2_16
> +-----------------------------
> +
> +V4L2_META_FMT_GENERIC_CSI2_16 contains packed 8-bit generic metadata, 16 bits
> +for each 8 bits of data. Every byte of metadata is followed by one byte of
> +padding. The way the data is stored follows the CSI-2 specification.
> +
> +This format is little endian.
> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_16.**
> +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|
> +
> +.. flat-table::
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - p
> +      - M\ :sub:`10`
> +      - p
> +      - M\ :sub:`20`
> +      - p
> +      - M\ :sub:`30`
> +      - p
> +    * - start + 8:
> +      - M\ :sub:`01`
> +      - p
> +      - M\ :sub:`11`
> +      - p
> +      - M\ :sub:`21`
> +      - p
> +      - M\ :sub:`31`
> +      - p
> +
> +.. _v4l2-meta-fmt-generic-csi2-20:
> +
> +V4L2_META_FMT_GENERIC_CSI2_20
> +-----------------------------
> +
> +V4L2_META_FMT_GENERIC_CSI2_20 contains packed 8-bit generic metadata, 20 bits
> +for each 8 bits of data. Every byte of metadata is followed by alternating one
> +and two bytes of padding. The way the data is stored follows the CSI-2
> +specification.
> +
> +This format is little endian.
> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_20.**
> +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> +
> +.. flat-table::
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8 8 8 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - p
> +      - M\ :sub:`10`
> +      - p
> +      - p
> +      - M\ :sub:`20`
> +      - p
> +      - M\ :sub:`30`
> +      - p
> +      - p
> +    * - start + 10:
> +      - M\ :sub:`01`
> +      - p
> +      - M\ :sub:`11`
> +      - p
> +      - p
> +      - M\ :sub:`21`
> +      - p
> +      - M\ :sub:`31`
> +      - p
> +      - p
> +
> +.. _v4l2-meta-fmt-generic-csi2-24:
> +
> +V4L2_META_FMT_GENERIC_CSI2_24
> +-----------------------------
> +
> +V4L2_META_FMT_GENERIC_CSI2_24 contains packed 8-bit generic metadata, 24 bits
> +for each 8 bits of data. Every byte of metadata is followed by two bytes of
> +padding. The way the data is stored follows the CSI-2 specification.
> +
> +This format is little endian.
> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_24.**
> +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> +
> +.. flat-table::
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8 8 8 8 8 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - p
> +      - p
> +      - M\ :sub:`10`
> +      - p
> +      - p
> +      - M\ :sub:`20`
> +      - p
> +      - p
> +      - M\ :sub:`30`
> +      - p
> +      - p
> +    * - start + 12:
> +      - M\ :sub:`01`
> +      - p
> +      - p
> +      - M\ :sub:`11`
> +      - p
> +      - p
> +      - M\ :sub:`21`
> +      - p
> +      - p
> +      - M\ :sub:`31`
> +      - p
> +      - p
> +
> +.. _v4l2-meta-fmt-generic-csi2-2-24:
> +
> +V4L2_META_FMT_GENERIC_CSI2_2_24
> +-------------------------------
> +
> +V4L2_META_FMT_GENERIC_CSI2_2_24 contains packed 8-bit generic metadata, 24 bits
> +for each two times 8 bits of data. Every two bytes of metadata are followed by
> +one byte of padding. The way the data is stored follows the CSI-2
> +specification.
> +
> +This format is little endian.
> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_2_24.**
> +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
> +
> +.. flat-table::
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - M\ :sub:`10`
> +      - p
> +      - M\ :sub:`20`
> +      - M\ :sub:`30`
> +      - p
> +    * - start + 6:
> +      - M\ :sub:`01`
> +      - M\ :sub:`11`
> +      - p
> +      - M\ :sub:`21`
> +      - M\ :sub:`31`
> +      - p
> +
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index f4d9d6279094..fbbddc333a30 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -1452,6 +1452,14 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
>  	case V4L2_PIX_FMT_Y210:		descr = "10-bit YUYV Packed"; break;
>  	case V4L2_PIX_FMT_Y212:		descr = "12-bit YUYV Packed"; break;
>  	case V4L2_PIX_FMT_Y216:		descr = "16-bit YUYV Packed"; break;
> +	case V4L2_META_FMT_GENERIC_8:	descr = "8-bit Generic Metadata"; break;
> +	case V4L2_META_FMT_GENERIC_CSI2_10:	descr = "8b Generic Meta, 10b CSI-2"; break;
> +	case V4L2_META_FMT_GENERIC_CSI2_12:	descr = "8b Generic Meta, 12b CSI-2"; break;
> +	case V4L2_META_FMT_GENERIC_CSI2_14:	descr = "8b Generic Meta, 14b CSI-2"; break;
> +	case V4L2_META_FMT_GENERIC_CSI2_16:	descr = "8b Generic Meta, 16b CSI-2"; break;
> +	case V4L2_META_FMT_GENERIC_CSI2_20:	descr = "8b Generic Meta, 20b CSI-2"; break;
> +	case V4L2_META_FMT_GENERIC_CSI2_24:	descr = "8b Generic Meta, 24b CSI-2"; break;
> +	case V4L2_META_FMT_GENERIC_CSI2_2_24:	descr = "2x8b Generic Meta, 24b CSI-2"; break;
>  
>  	default:
>  		/* Compressed formats */
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index 78260e5d9985..b4284a564025 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -838,6 +838,15 @@ struct v4l2_pix_format {
>  #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
>  #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
>  
> +#define V4L2_META_FMT_GENERIC_8		v4l2_fourcc('M', 'E', 'T', '8') /* Generic 8-bit metadata */
> +#define V4L2_META_FMT_GENERIC_CSI2_10	v4l2_fourcc('M', 'C', '1', 'A') /* 10-bit CSI-2 packed 8-bit metadata */
> +#define V4L2_META_FMT_GENERIC_CSI2_12	v4l2_fourcc('M', 'C', '1', 'C') /* 12-bit CSI-2 packed 8-bit metadata */
> +#define V4L2_META_FMT_GENERIC_CSI2_14	v4l2_fourcc('M', 'C', '1', 'E') /* 14-bit CSI-2 packed 8-bit metadata */
> +#define V4L2_META_FMT_GENERIC_CSI2_16	v4l2_fourcc('M', 'C', '1', 'G') /* 16-bit CSI-2 packed 8-bit metadata */
> +#define V4L2_META_FMT_GENERIC_CSI2_20	v4l2_fourcc('M', 'C', '1', 'K') /* 20-bit CSI-2 packed 8-bit metadata */
> +#define V4L2_META_FMT_GENERIC_CSI2_24	v4l2_fourcc('M', 'C', '1', 'O') /* 24-bit CSI-2 packed 8-bit metadata */
> +#define V4L2_META_FMT_GENERIC_CSI2_2_24	v4l2_fourcc('M', 'C', '2', 'O') /* 2 bytes of 8-bit metadata, 24-bit CSI-2 packed */
> +
>  /* priv field value to indicates that subsequent fields are valid. */
>  #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
>  

-- 
Regards,

Laurent Pinchart

Re: [PATCH v3 06/10] media: uapi: Add a macro to tell whether an mbus code is metadata

[Laurent Pinchart]

On Tue, Sep 05, 2023 at 10:37:45AM +0000, Sakari Ailus wrote:
> On Tue, Sep 05, 2023 at 12:47:21PM +0300, Tomi Valkeinen wrote:
> > On 08/08/2023 10:55, Sakari Ailus wrote:
> > > Add a macro to tell whether a given mbus code is metadata.
> > > 
> > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > ---
> > >   include/uapi/linux/media-bus-format.h | 3 +++
> > >   1 file changed, 3 insertions(+)
> > > 
> > > diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
> > > index 9ee031397372..2486b4178c5f 100644
> > > --- a/include/uapi/linux/media-bus-format.h
> > > +++ b/include/uapi/linux/media-bus-format.h
> > > @@ -182,4 +182,7 @@
> > >   #define MEDIA_BUS_FMT_META_20			0x8006
> > >   #define MEDIA_BUS_FMT_META_24			0x8007
> > > +#define MEDIA_BUS_FMT_IS_META(code)		\
> > > +	((code) & 0xf000 == 0x7000 || (code) & 0xf000 == 0x8000)
> > > +
> > >   #endif /* __LINUX_MEDIA_BUS_FORMAT_H */
> > 
> > mbus code seems to be u32, so the above won't work. Maybe:
> > 
> > (((code) & ~0xfffULL) == 0x7000 || ((code) & ~0xfffULL) == 0x8000)
> > 
> > Also, embedded formats with 0x9nnn codes are added later in the series.
> 
> Thanks, I'll address these for v4.

It would be nice to make this an inline function, to avoid evaluating
the code twice. I think you can move it to an internal kernel header, it
doesn't need to be exposed to userspace.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v3 08/10] media: v4l: Support line-based metadata capture

[Laurent Pinchart]

Hi Sakari,

On Mon, Aug 14, 2023 at 11:02:40AM +0000, Sakari Ailus wrote:
> On Thu, Aug 10, 2023 at 05:24:14PM +0200, Jacopo Mondi wrote:
> > On Tue, Aug 08, 2023 at 10:55:36AM +0300, Sakari Ailus wrote:
> > > many camera sensors, among other devices, transmit embedded data and image

s/many/Many/

> > > data for each CSI-2 frame. This embedded data typically contains register
> > > configuration of the sensor that has been used to capture the image data
> > > of the same frame.
> > >
> > > The embedded data is received by the CSI-2 receiver and has the same
> > > properties as the image data, including that it is line based: it has
> > > width, height and bytesperline (stride).
> > >
> > > Add these fields to struct v4l2_meta_format and document them.
> > >
> > > Also add V4L2_FMT_FLAG_META_LINE_BASED to tell a given format is
> > > line-based i.e. these fields of struct v4l2_meta_format are valid for it.
> > >
> > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > ---
> > >  .../userspace-api/media/v4l/dev-meta.rst          | 15 +++++++++++++++
> > >  .../userspace-api/media/v4l/vidioc-enum-fmt.rst   |  7 +++++++
> > >  .../media/videodev2.h.rst.exceptions              |  1 +
> > >  drivers/media/v4l2-core/v4l2-ioctl.c              |  5 +++--
> > >  include/uapi/linux/videodev2.h                    | 10 ++++++++++
> > >  5 files changed, 36 insertions(+), 2 deletions(-)
> > >
> > > diff --git a/Documentation/userspace-api/media/v4l/dev-meta.rst b/Documentation/userspace-api/media/v4l/dev-meta.rst
> > > index 0e7e1ee1471a..4b24bae6e171 100644
> > > --- a/Documentation/userspace-api/media/v4l/dev-meta.rst
> > > +++ b/Documentation/userspace-api/media/v4l/dev-meta.rst
> > > @@ -65,3 +65,18 @@ to 0.
> > >        - ``buffersize``
> > >        - Maximum buffer size in bytes required for data. The value is set by the
> > >          driver.
> > > +    * - __u32
> > > +      - ``width``
> > > +      - Width of a line of metadata in samples. Valid when :c:type`v4l2_fmtdesc`
> > > +	flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
> > > +	:c:func:`VIDIOC_ENUM_FMT`.
> > > +    * - __u32
> > > +      - ``height``
> > > +      - Number of rows of metadata. Valid when :c:type`v4l2_fmtdesc` flag
> > > +	``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
> > > +	:c:func:`VIDIOC_ENUM_FMT`.
> > > +    * - __u32
> > > +      - ``bytesperline``
> > > +      - Offset in bytes between the beginning of two consecutive lines. Valid
> > > +	when :c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is
> > > +	set, otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
> > > diff --git a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> > > index 000c154b0f98..6d7664345a4e 100644
> > > --- a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> > > +++ b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> > > @@ -227,6 +227,13 @@ the ``mbus_code`` field is handled differently:
> > >  	The application can ask to configure the quantization of the capture
> > >  	device when calling the :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl with
> > >  	:ref:`V4L2_PIX_FMT_FLAG_SET_CSC <v4l2-pix-fmt-flag-set-csc>` set.
> > > +    * - ``V4L2_FMT_FLAG_META_LINE_BASED``
> > > +      - 0x0200
> > > +      - The metadata format is line-based. In this case the ``width``,
> > > +	``height`` and ``bytesperline`` fields of :c:type:`v4l2_meta_format` are
> > > +	valid. The buffer consists of ``height`` lines, each having ``width``
> > > +	bytes of data and offset between the beginning of each two consecutive
> > 
> > Isn't ``width`` in samples ?
> 
> Indeed, it's better to refer to samples for clarity. I'll fix for v4.

How do you define a "sample" in this case ? I wonder if it wouldn't be
simpler for both userspace and kernel drivers if the width was specified
in bytes, including the padding bytes.

We need an implementation here to test it out. The good news is that I'm
working on it, the bad news is that I'm also busy with other things.

> I'll also add bytesperline is in bytes (and not in samples).

Thanks for not messing up (more than needed) with my mental health by
not specifying bytesperline in something else than bytes :-)

> > > +	lines is ``bytesperline``.
> > >
> > >  Return Value
> > >  ============
> > > diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b/Documentation/userspace-api/media/videodev2.h.rst.exceptions
> > > index 3e58aac4ef0b..bdc628e8c1d6 100644
> > > --- a/Documentation/userspace-api/media/videodev2.h.rst.exceptions
> > > +++ b/Documentation/userspace-api/media/videodev2.h.rst.exceptions
> > > @@ -215,6 +215,7 @@ replace define V4L2_FMT_FLAG_CSC_XFER_FUNC fmtdesc-flags
> > >  replace define V4L2_FMT_FLAG_CSC_YCBCR_ENC fmtdesc-flags
> > >  replace define V4L2_FMT_FLAG_CSC_HSV_ENC fmtdesc-flags
> > >  replace define V4L2_FMT_FLAG_CSC_QUANTIZATION fmtdesc-flags
> > > +replace define V4L2_FMT_FLAG_META_LINE_BASED fmtdesc-flags
> > >
> > >  # V4L2 timecode types
> > >  replace define V4L2_TC_TYPE_24FPS timecode-type
> > > diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> > > index fbbddc333a30..971d784e7429 100644
> > > --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> > > +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> > > @@ -343,8 +343,9 @@ static void v4l_print_format(const void *arg, bool write_only)
> > >  	case V4L2_BUF_TYPE_META_OUTPUT:
> > >  		meta = &p->fmt.meta;
> > >  		pixelformat = meta->dataformat;
> > > -		pr_cont(", dataformat=%p4cc, buffersize=%u\n",
> > > -			&pixelformat, meta->buffersize);
> > > +		pr_cont(", dataformat=%p4cc, buffersize=%u, width=%u, height=%u, bytesperline=%u\n",
> > > +			&pixelformat, meta->buffersize, meta->width,
> > > +			meta->height, meta->bytesperline);
> > >  		break;
> > >  	}
> > >  }
> > > diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> > > index b4284a564025..d26c0650c6a7 100644
> > > --- a/include/uapi/linux/videodev2.h
> > > +++ b/include/uapi/linux/videodev2.h
> > > @@ -877,6 +877,7 @@ struct v4l2_fmtdesc {
> > >  #define V4L2_FMT_FLAG_CSC_YCBCR_ENC		0x0080
> > >  #define V4L2_FMT_FLAG_CSC_HSV_ENC		V4L2_FMT_FLAG_CSC_YCBCR_ENC
> > >  #define V4L2_FMT_FLAG_CSC_QUANTIZATION		0x0100
> > > +#define V4L2_FMT_FLAG_META_LINE_BASED		0x0200
> > >
> > >  	/* Frame Size and frame rate enumeration */
> > >  /*
> > > @@ -2420,10 +2421,19 @@ struct v4l2_sdr_format {
> > >   * struct v4l2_meta_format - metadata format definition
> > >   * @dataformat:		little endian four character code (fourcc)
> > >   * @buffersize:		maximum size in bytes required for data
> > > + * @width:		number of bytes of data per line (valid for line based
> > 
> > I'm a bit confused here as well, isn't width in samples ?
> 
> I'll change this one as well.
> 
> > > + *			formats only, see format documentation)
> > > + * @height:		number of lines of data per buffer (valid for line based
> > > + *			formats only)
> > > + * @bytesperline:	offset between the beginnings of two adjacent lines in
> > > + *			bytes (valid for line based formats only)
> > >   */
> > >  struct v4l2_meta_format {
> > >  	__u32				dataformat;
> > >  	__u32				buffersize;
> > > +	__u32				width;
> > > +	__u32				height;
> > > +	__u32				bytesperline;
> > >  } __attribute__ ((packed));
> > >
> > >  /**

-- 
Regards,

Laurent Pinchart

Re: [PATCH v3 09/10] media: Add media bus codes for MIPI CCS embedded data

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Tue, Aug 08, 2023 at 10:55:37AM +0300, Sakari Ailus wrote:
> Add new MIPI CCS embedded data media bus formats.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../media/v4l/subdev-formats.rst              | 32 +++++++++++++++++++
>  include/uapi/linux/media-bus-format.h         | 10 +++++-
>  2 files changed, 41 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> index c615da08502d..5d5407738af9 100644
> --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
> +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> @@ -8491,3 +8491,35 @@ and finally the bit number in subscript. "p" indicates a padding bit.
>        - p
>        - p
>        - p
> +
> +MIPI CCS Embedded Data Formats
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +`MIPI CCS <https://www.mipi.org/specifications/camera-command-set>`_ defines an

s/an$/a/

> +metadata format for sensor embedded data, which is used to store the register
> +configuration used for capturing a given frame. The format is defined in the CCS
> +specification.

Strictly speaking, the MIPI CCS embedded data format specifies not just
the data packing (insertion of padding bytes) and the data encoding (the
data format byte and the tag codes), but also the register addresses and
values that are reported in the embedded data. Do you envision the media
bus formats defined here as being applicable to sensors that use the
same packing and encoding as CCS, but different registers, or only to
fully compliant CCS sensors ?

> +
> +The bit depth of the CCS embedded data matches the pixel data bit depth
> +configured on the sensor. The formats used and their corresponding generic
> +formats are listed in the table below.
> +
> +.. flat-table: CCS embedded data mbus formats and corresponding generic formats
> +    :header-rows: 1
> +
> +    * - CCS embedded data mbus format
> +      - Generic metadata format
> +    * - MEDIA_BUS_FMT_CCS_EMBEDDED_8
> +      - MEDIA_BUS_FMT_META_8
> +    * - MEDIA_BUS_FMT_CCS_EMBEDDED_10
> +      - MEDIA_BUS_FMT_META_10
> +    * - MEDIA_BUS_FMT_CCS_EMBEDDED_12
> +      - MEDIA_BUS_FMT_META_12
> +    * - MEDIA_BUS_FMT_CCS_EMBEDDED_14
> +      - MEDIA_BUS_FMT_META_14
> +    * - MEDIA_BUS_FMT_CCS_EMBEDDED_16
> +      - MEDIA_BUS_FMT_META_16
> +    * - MEDIA_BUS_FMT_CCS_EMBEDDED_20
> +      - MEDIA_BUS_FMT_META_20
> +    * - MEDIA_BUS_FMT_CCS_EMBEDDED_24
> +      - MEDIA_BUS_FMT_META_24
> diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
> index 2486b4178c5f..31d532446b23 100644
> --- a/include/uapi/linux/media-bus-format.h
> +++ b/include/uapi/linux/media-bus-format.h
> @@ -182,7 +182,15 @@
>  #define MEDIA_BUS_FMT_META_20			0x8006
>  #define MEDIA_BUS_FMT_META_24			0x8007
>  
> +/* Specific metadata formats. Next is 0x9008. */
> +#define MEDIA_BUS_FMT_CCS_EMBEDDED_8		0x9001
> +#define MEDIA_BUS_FMT_CCS_EMBEDDED_10		0x9002
> +#define MEDIA_BUS_FMT_CCS_EMBEDDED_12		0x9003
> +#define MEDIA_BUS_FMT_CCS_EMBEDDED_14		0x9004
> +#define MEDIA_BUS_FMT_CCS_EMBEDDED_16		0x9005
> +#define MEDIA_BUS_FMT_CCS_EMBEDDED_20		0x9006
> +#define MEDIA_BUS_FMT_CCS_EMBEDDED_24		0x9007
> +
>  #define MEDIA_BUS_FMT_IS_META(code)		\
>  	((code) & 0xf000 == 0x7000 || (code) & 0xf000 == 0x8000)
> -
>  #endif /* __LINUX_MEDIA_BUS_FORMAT_H */

-- 
Regards,

Laurent Pinchart

Re: [PATCH v3 10/10] media: uapi: v4l: Document source routes

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Fri, Aug 11, 2023 at 10:44:40AM +0000, Sakari Ailus wrote:
> On Tue, Aug 08, 2023 at 10:55:24AM +0200, Hans Verkuil wrote:
> > On 08/08/2023 09:55, Sakari Ailus wrote:
> > > Document how internal pads are used on source routes.
> > > 
> > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > ---
> > >  .../userspace-api/media/v4l/dev-subdev.rst    | 179 ++++++++++++++++++
> > >  1 file changed, 179 insertions(+)
> > > 
> > > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > index b023918177b5..27b0fe2dc83a 100644
> > > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > @@ -551,6 +551,27 @@ A stream at a specific point in the media pipeline is identified by the
> > >  sub-device and a (pad, stream) pair. For sub-devices that do not support
> > >  multiplexed streams the 'stream' field is always 0.
> > >  
> > > +.. _v4l2-subdev-source-routes:
> > > +
> > > +Internal pads and source routes
> > > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > +
> > > +Cases where a single sub-device source pad is traversed by multiple streams one
> > > +or more of which originate from within the sub-device itself are special as
> > > +there is no external sink pad for such routes. In those cases, the sources of
> > > +the internally generated streams are represented by internal sink pads, which
> > > +are sink pads that have the :ref:`MEDIA_PAD_FL_INTERNAL <MEDIA-PAD-FL-INTERNAL>`
> > > +pad flag set.
> > > +
> > > +Internal pads have all the properties of an external pad, including formats and
> > > +selections. The format in this case is the source format of the stream. An
> > > +internal pad always has a single stream only (0).
> > > +
> > > +/Source routes/ are routes from an internal sink pad to a(n external) source

"a(n external)" looks weird.

> > '/Source routes/' appears just like that in the generated documentation, that's
> > probably not what you intended.
> 
> This was intended to be *Source routes*. I'll address it in v4.
> 
> > > +pad. Generally source routes are not modifiable but they can be activated and
> > > +deactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > > +<v4l2-subdev-routing-flags>` flag, depending on driver capabilities.

We need to update the documentation of the routing API to clearly
explain the different models (i.e. specifying all routes, active and
inactive, vs. including only active routes). That's out of scope for
this patch, but it could be done in the same series.

> > > +
> > >  Interaction between routes, streams, formats and selections
> > >  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > >  
> > > @@ -666,3 +687,161 @@ A common way to accomplish this is to start from the sensors and propagate the
> > >  configurations along the stream towards the receiver,
> > >  using :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each
> > >  stream endpoint in each sub-device.
> > > +
> > > +Internal pads setup example
> > > +^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > +
> > > +A simple example of a multiplexed stream setup might be as follows:
> > > +
> > > +- A CCS camera sensor source sub-device, with one sink pad (0), one source pad
> > 
> > Explain what 'CCS' means: probably a link to MIPI CCS is sufficient, but for newbies
> > 'CCS' doesn't provide any information.
> > 
> > > +  (1), an internal sink pad (2) that represents the source of embedded
> > 
> > So since the INTERNAL flag is associated with a sink pad, this is now called an
> > 'internal sink' instead of 'internal source' as it was before. I agree with that,
> > but note that the phrase 'internal source' is still used in several places,
> > including the cover letter. For the next version you post, please check for that
> > and change the terminology since mixing the two is very confusing :-)
> 
> Are you referring to this patch? There don't seem to be any references to
> "intenal source pads" left.
> 
> > > +  data. There are two routes, one from the sink pad to the source, and another
> > > +  from the internal sink pad to the source pad. The embedded data stream needs
> > > +  to be enabled by activating the related route. The configuration of the rest
> > > +  of the CCS sub-devices is omitted from this example.

CCS is complex, with its multiple subdevs. I think a simpler raw sensor
without a scaler, using a single subdev, would be a better example.

> > > +
> > > +- Multiplexer bridge (Bridge). The bridge has one sink pad, connected to the
> > > +  sensor (pad 0), and one source pad (pad 1), which outputs two streams.
> > > +
> > > +- Receiver in the SoC (Receiver). The receiver has a single sink pad (pad 0),
> > > +  connected to the bridge, and two source pads (pads 1-2), going to the DMA
> > > +  engine. The receiver demultiplexes the incoming streams to the source pads.
> > > +
> > > +- DMA Engines in the SoC (DMA Engine), one for each stream. Each DMA engine is
> > > +  connected to a single source pad in the receiver.
> > > +
> > > +The sensor, the bridge and the receiver are modeled as V4L2 sub-devices,
> > > +exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are
> > > +modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
> > > +
> > > +To configure this pipeline, the userspace must take the following steps:
> > > +
> > > +1) Set up media links between entities: connect the sensors to the bridge,
> > > +   bridge to the receiver, and the receiver to the DMA engines. This step does
> > > +   not differ from normal non-multiplexed media controller setup.
> > > +
> > > +2) Configure routing
> > > +
> > > +.. flat-table:: Camera sensor
> > > +    :header-rows: 1
> > > +
> > > +    * - Sink Pad/Stream
> > > +      - Source Pad/Stream
> > > +      - Routing Flags
> > > +      - Comments
> > > +    * - 0/0
> > > +      - 1/0
> > > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > > +      - Pixel data stream from the sink pad
> > 
> > Huh? It is a sensor subdev, so aren't both pixel data and metadata
> > coming from an internal sink?
> 
> The CCS driver exposes three sub-devices, so in this case the sink pad
> isn't internal. Almost (?) all other sensor drivers have a single
> sub-device only, perhaps I'll change the example to align with one of
> those.

Ah, it looks like we have thought about the same thing :-)

> > I think you are actually describing the internal subdev of a CCS
> > sensor here, but that is not clear from the description at all.
> > 
> > Since CCS sensors are relatively common (I think), it might be
> > good to give a fully fledged example.
> > 
> > But suppose you have a non-CCS sensor that generates the pixel data
> > and metadata, then you would have two internal sink pads and one
> > source pad for two streams, right?
> > 
> > And very simple sensors that currently just have a single source pad
> > could be describes as having an internal sink pad connected to the
> > source pad by a fixed route. But we don't do that, because it is
> > overkill. Correct? If so, then this might be useful information to add
> > to answer the question why some sensors have internal sinks and others
> > do not.
> > 
> > > +    * - 2/0
> > > +      - 1/1
> > > +      - **V4L2_SUBDEV_ROUTE_FL_ACTIVE**
> > > +      - Metadata stream from the internal sink pad
> > > +
> > > +.. flat-table:: Bridge routing table
> > > +    :header-rows: 1
> > > +
> > > +    * - Sink Pad/Stream
> > > +      - Source Pad/Stream
> > > +      - Routing Flags
> > > +      - Comments
> > > +    * - 0/0
> > > +      - 1/0
> > > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > > +      - Pixel data stream from camera sensor
> > > +    * - 0/1
> > > +      - 1/1
> > > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > > +      - Metadata stream from camera sensor
> > > +
> > > +.. flat-table:: Receiver routing table
> > > +    :header-rows:  1
> > > +
> > > +    * - Sink Pad/Stream
> > > +      - Source Pad/Stream
> > > +      - Routing Flags
> > > +      - Comments
> > > +    * - 0/0
> > > +      - 1/0
> > > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > > +      - Pixel data stream from camera sensor
> > > +    * - 0/1
> > > +      - 2/0
> > > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > > +      - Metadata stream from camera sensor
> > > +
> > > +The options available in sensor's routing configuration are dictated by hardware
> > > +capabilities: typically camera sensors always produce image data stream while
> > > +the embedded data stream typically can be either enabled or disabled.
> > > +
> > > +3) Configure formats and selections
> > > +
> > > +This example assumes that the formats are propagated from sink pad to the source
> > > +pad as-is. The tables contain fields of both struct v4l2_subdev_format and
> > > +struct v4l2_mbus_framefmt. The full configuration of CCS camera sensor is out of
> > > +scope of this example.
> > > +
> > > +.. flat-table:: Formats set on the sub-devices. Bold values are set, others are
> > > +                static or propagated.
> > > +    :header-rows: 1
> > > +    :fill-cells:
> > > +
> > > +    * - Sub-device
> > > +      - Pad/Stream
> > > +      - Width
> > > +      - Height
> > > +      - Code
> > > +    * - :rspan:`3` Camera sensor sub-device (CCS source sub-device)
> > > +      - 0/0
> > > +      - 640
> > > +      - 480
> > > +      - MEDIA_BUS_FMT_SGRBG10
> > > +    * - 1/0
> > > +      - 640
> > > +      - 480
> > > +      - **MEDIA_BUS_FMT_SGRBG10**
> > > +    * - 2/0
> > > +      - 640
> > > +      - 2
> > > +      - MEDIA_BUS_FMT_CCS_EMBEDDED_10
> > > +    * - 1/1
> > > +      - 640
> > > +      - 2
> > > +      - MEDIA_BUS_FMT_META_10
> > > +    * - :rspan:`3` Bridge
> > > +      - 0/0
> > > +      - **640**
> > > +      - **480**
> > > +      - **MEDIA_BUS_FMT_SGRBG10**
> > > +    * - 1/0
> > > +      - 640
> > > +      - 480
> > > +      - MEDIA_BUS_FMT_SGRBG10
> > > +    * - 0/1
> > > +      - **640**
> > > +      - **2**
> > > +      - **MEDIA_BUS_FMT_META_10**
> > > +    * - 1/1
> > > +      - 640
> > > +      - 2
> > > +      - MEDIA_BUS_FMT_META_10
> > > +    * - :rspan:`3` Receiver
> > > +      - 0/0
> > > +      - **640**
> > > +      - **480**
> > > +      - **MEDIA_BUS_FMT_SGRBG10**
> > > +    * - 1/0
> > > +      - 640
> > > +      - 480
> > > +      - MEDIA_BUS_FMT_SGRBG10
> > > +    * - 0/1
> > > +      - **640**
> > > +      - **2**
> > > +      - **MEDIA_BUS_FMT_META_10**
> > > +    * - 2/0
> > > +      - 640
> > > +      - 2
> > > +      - MEDIA_BUS_FMT_META_10
> > > +
> > > +The embedded data format does not need to be configured as the format is
> > > +dictated by the pixel data format in this case.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v3 08/10] media: v4l: Support line-based metadata capture

[Jacopo Mondi]

Hi Laurent

On Tue, Sep 05, 2023 at 08:15:33PM +0300, Laurent Pinchart wrote:
> Hi Sakari,
>
> On Mon, Aug 14, 2023 at 11:02:40AM +0000, Sakari Ailus wrote:
> > On Thu, Aug 10, 2023 at 05:24:14PM +0200, Jacopo Mondi wrote:
> > > On Tue, Aug 08, 2023 at 10:55:36AM +0300, Sakari Ailus wrote:
> > > > many camera sensors, among other devices, transmit embedded data and image
>
> s/many/Many/
>
> > > > data for each CSI-2 frame. This embedded data typically contains register
> > > > configuration of the sensor that has been used to capture the image data
> > > > of the same frame.
> > > >
> > > > The embedded data is received by the CSI-2 receiver and has the same
> > > > properties as the image data, including that it is line based: it has
> > > > width, height and bytesperline (stride).
> > > >
> > > > Add these fields to struct v4l2_meta_format and document them.
> > > >
> > > > Also add V4L2_FMT_FLAG_META_LINE_BASED to tell a given format is
> > > > line-based i.e. these fields of struct v4l2_meta_format are valid for it.
> > > >
> > > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > > ---
> > > >  .../userspace-api/media/v4l/dev-meta.rst          | 15 +++++++++++++++
> > > >  .../userspace-api/media/v4l/vidioc-enum-fmt.rst   |  7 +++++++
> > > >  .../media/videodev2.h.rst.exceptions              |  1 +
> > > >  drivers/media/v4l2-core/v4l2-ioctl.c              |  5 +++--
> > > >  include/uapi/linux/videodev2.h                    | 10 ++++++++++
> > > >  5 files changed, 36 insertions(+), 2 deletions(-)
> > > >
> > > > diff --git a/Documentation/userspace-api/media/v4l/dev-meta.rst b/Documentation/userspace-api/media/v4l/dev-meta.rst
> > > > index 0e7e1ee1471a..4b24bae6e171 100644
> > > > --- a/Documentation/userspace-api/media/v4l/dev-meta.rst
> > > > +++ b/Documentation/userspace-api/media/v4l/dev-meta.rst
> > > > @@ -65,3 +65,18 @@ to 0.
> > > >        - ``buffersize``
> > > >        - Maximum buffer size in bytes required for data. The value is set by the
> > > >          driver.
> > > > +    * - __u32
> > > > +      - ``width``
> > > > +      - Width of a line of metadata in samples. Valid when :c:type`v4l2_fmtdesc`
> > > > +	flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
> > > > +	:c:func:`VIDIOC_ENUM_FMT`.
> > > > +    * - __u32
> > > > +      - ``height``
> > > > +      - Number of rows of metadata. Valid when :c:type`v4l2_fmtdesc` flag
> > > > +	``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
> > > > +	:c:func:`VIDIOC_ENUM_FMT`.
> > > > +    * - __u32
> > > > +      - ``bytesperline``
> > > > +      - Offset in bytes between the beginning of two consecutive lines. Valid
> > > > +	when :c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is
> > > > +	set, otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
> > > > diff --git a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> > > > index 000c154b0f98..6d7664345a4e 100644
> > > > --- a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> > > > +++ b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> > > > @@ -227,6 +227,13 @@ the ``mbus_code`` field is handled differently:
> > > >  	The application can ask to configure the quantization of the capture
> > > >  	device when calling the :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl with
> > > >  	:ref:`V4L2_PIX_FMT_FLAG_SET_CSC <v4l2-pix-fmt-flag-set-csc>` set.
> > > > +    * - ``V4L2_FMT_FLAG_META_LINE_BASED``
> > > > +      - 0x0200
> > > > +      - The metadata format is line-based. In this case the ``width``,
> > > > +	``height`` and ``bytesperline`` fields of :c:type:`v4l2_meta_format` are
> > > > +	valid. The buffer consists of ``height`` lines, each having ``width``
> > > > +	bytes of data and offset between the beginning of each two consecutive
> > >
> > > Isn't ``width`` in samples ?
> >
> > Indeed, it's better to refer to samples for clarity. I'll fix for v4.
>
> How do you define a "sample" in this case ? I wonder if it wouldn't be
> simpler for both userspace and kernel drivers if the width was specified
> in bytes, including the padding bytes.

Wouldn't this make the image line length (expressed in 'samples')
different than the embedded data line length ? Would this confuse
userspace or is it fine ?

>
> We need an implementation here to test it out. The good news is that I'm
> working on it, the bad news is that I'm also busy with other things.
>
> > I'll also add bytesperline is in bytes (and not in samples).
>
> Thanks for not messing up (more than needed) with my mental health by
> not specifying bytesperline in something else than bytes :-)
>

:)

> > > > +	lines is ``bytesperline``.
> > > >
> > > >  Return Value
> > > >  ============
> > > > diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b/Documentation/userspace-api/media/videodev2.h.rst.exceptions
> > > > index 3e58aac4ef0b..bdc628e8c1d6 100644
> > > > --- a/Documentation/userspace-api/media/videodev2.h.rst.exceptions
> > > > +++ b/Documentation/userspace-api/media/videodev2.h.rst.exceptions
> > > > @@ -215,6 +215,7 @@ replace define V4L2_FMT_FLAG_CSC_XFER_FUNC fmtdesc-flags
> > > >  replace define V4L2_FMT_FLAG_CSC_YCBCR_ENC fmtdesc-flags
> > > >  replace define V4L2_FMT_FLAG_CSC_HSV_ENC fmtdesc-flags
> > > >  replace define V4L2_FMT_FLAG_CSC_QUANTIZATION fmtdesc-flags
> > > > +replace define V4L2_FMT_FLAG_META_LINE_BASED fmtdesc-flags
> > > >
> > > >  # V4L2 timecode types
> > > >  replace define V4L2_TC_TYPE_24FPS timecode-type
> > > > diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> > > > index fbbddc333a30..971d784e7429 100644
> > > > --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> > > > +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> > > > @@ -343,8 +343,9 @@ static void v4l_print_format(const void *arg, bool write_only)
> > > >  	case V4L2_BUF_TYPE_META_OUTPUT:
> > > >  		meta = &p->fmt.meta;
> > > >  		pixelformat = meta->dataformat;
> > > > -		pr_cont(", dataformat=%p4cc, buffersize=%u\n",
> > > > -			&pixelformat, meta->buffersize);
> > > > +		pr_cont(", dataformat=%p4cc, buffersize=%u, width=%u, height=%u, bytesperline=%u\n",
> > > > +			&pixelformat, meta->buffersize, meta->width,
> > > > +			meta->height, meta->bytesperline);
> > > >  		break;
> > > >  	}
> > > >  }
> > > > diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> > > > index b4284a564025..d26c0650c6a7 100644
> > > > --- a/include/uapi/linux/videodev2.h
> > > > +++ b/include/uapi/linux/videodev2.h
> > > > @@ -877,6 +877,7 @@ struct v4l2_fmtdesc {
> > > >  #define V4L2_FMT_FLAG_CSC_YCBCR_ENC		0x0080
> > > >  #define V4L2_FMT_FLAG_CSC_HSV_ENC		V4L2_FMT_FLAG_CSC_YCBCR_ENC
> > > >  #define V4L2_FMT_FLAG_CSC_QUANTIZATION		0x0100
> > > > +#define V4L2_FMT_FLAG_META_LINE_BASED		0x0200
> > > >
> > > >  	/* Frame Size and frame rate enumeration */
> > > >  /*
> > > > @@ -2420,10 +2421,19 @@ struct v4l2_sdr_format {
> > > >   * struct v4l2_meta_format - metadata format definition
> > > >   * @dataformat:		little endian four character code (fourcc)
> > > >   * @buffersize:		maximum size in bytes required for data
> > > > + * @width:		number of bytes of data per line (valid for line based
> > >
> > > I'm a bit confused here as well, isn't width in samples ?
> >
> > I'll change this one as well.
> >
> > > > + *			formats only, see format documentation)
> > > > + * @height:		number of lines of data per buffer (valid for line based
> > > > + *			formats only)
> > > > + * @bytesperline:	offset between the beginnings of two adjacent lines in
> > > > + *			bytes (valid for line based formats only)
> > > >   */
> > > >  struct v4l2_meta_format {
> > > >  	__u32				dataformat;
> > > >  	__u32				buffersize;
> > > > +	__u32				width;
> > > > +	__u32				height;
> > > > +	__u32				bytesperline;
> > > >  } __attribute__ ((packed));
> > > >
> > > >  /**
>
> --
> Regards,
>
> Laurent Pinchart

Re: [PATCH v3 04/10] media: uapi: Add generic serial metadata mbus formats

[Tomi Valkeinen]

Hi,

On 05/09/2023 19:38, Laurent Pinchart wrote:
> On Thu, Aug 24, 2023 at 11:26:56AM +0300, Tomi Valkeinen wrote:
>> On 24/08/2023 10:24, Sakari Ailus wrote:
>>> On Wed, Aug 23, 2023 at 04:16:13PM +0300, Tomi Valkeinen wrote:
>>>> On 08/08/2023 10:55, Sakari Ailus wrote:
>>>>> Add generic serial metadata mbus formats. These formats describe data
>>>>> width and packing but not the content itself. The reason for specifying
>>>>> such formats is that the formats as such are fairly device specific but
>>>>> they are still handled by CSI-2 receiver drivers that should not be aware
> 
> Do we want these formats to be CSI-2-specific ?
 >
>>>>> of device specific formats. What makes generic metadata formats possible
>>>>> is that these formats are parsed by software only, after capturing the
>>>>> data to system memory.
>>>>
>>>> If I'm not mistaken, the CSI-2 spec doesn't say much about embedded data,
>>>> except that it may exist. Afaics, in CSI-2, the embedded data is split into
>>>> "lines", although the amount of data can be different than in the video
>>>> lines.
>>>>
>>>> The CCS specs talks more about embedded data. Some of it is quite odd, like:
>>>> "The length of the embedded data line shall not exceed the length of the
>>>> image data line. The embedded data line should have the same length as the
>>>> image data line.". I think it means the embedded line can be shorter than
>>>> image line, but not longer.
>>>
>>> That's what it means, yes. The CCS also has means to obtain the actual
>>> length --- frame format descriptors.
>>>
>>>> CCS also says that an embedded line should use 0x07 as padding at the end of
>>>> the line, if there's less data than the image line.
>>>>
>>>> CCS also talks about how the embedded data would be packed, and in some
>>>> cases that packing would be the same as for pixel data.
>>>
>>> In fact the packing is the same as for pixel data: the CSI-2 does not
>>> really differentiate there.
>>
>> If I understand the CCS spec right, the packing is not the same as for
>> the pixel data. You can have RAW12 pixel data but 8-byte embedded data.
>> But if you meant that the different packing style options are the same
>> for pixel and embedded data, yes.
> 
> The idea is to allow CSI-2 receivers to unpack embedded data the same
> way as pixel data at the hardware level, and guarantee that no embedded

Yes, but my comment was to Sakari's "the packing is the same as for 
pixel data" comment. Afaics, there's nothing in CSI-2 or CCS that says 
the packing is the same for pixel data and embedded data.

In fact, the CSS says "By default, top-embedded data shall use the same 
data packing as the Visible pixel data with one embedded data byte per 
pixel", implying that it would not always be the case. It continues that 
for RAW16/RAW20/RAW24 pixels the embedded data could be sent with 
RAW8/RAW10/RAW12.

That was just for top-embedded data, and I couldn't right away see 
mention of other embedded data.

> data significant byte gets split across multiple samples. When unpacking
> RAW10 pixel data, the CSI-2 receiver will take 5 bytes and output 4
> samples on a 10-bit wide bus. For embedded data, it can conceptually do
> that same, and output embedded data with one byte per sample, aligned on
> the MSB of the 10-bit bus. However, in practice, I think the CSI-2
> receiver will simply receive the bytes and send them to the DMA engine,
> without unpacking and then repacking.

At least TI CAL CSI-2 RX can extract the data from the CSI-2 stream 
using different unpacking for pixel and embedded data, and then 
(optionally) pack it in different ways when writing to memory.

> This can also be viewed as away to allow CSI-2 transmitters to serialize
> pixel data and embedded data the same way. For a RAW10 sensor, for
> instance, the embedded data would be generated on an internal 10-bit
> wide bus, aligned to the MSB, and would go through CSI-2 packing the
> same way that pixels do. That's nice, but it's internal to the sensor,
> so we don't need to expose it to userspace on that side.
> 
>>>> But I don't think these formats are generic. They're defined in CCS, so
>>>> shouldn't the format be, e.g., MEDIA_BUS_FMT_META_CCS_RAW_12 or such?
>>>
>>> The reason for having generic definitions is that we do not need receiver
>>> drivers to be aware of formats that are specific to another driver.
>>
>> Yes, I agree with that, and that's not my point here. But "generic"
>> doesn't mean the definitions are not for some particular bus or
>> standard, "generic" just means that it doesn't specify the content, only
>> the packing.
>>
>> My point is that these packings seem to be specific to CCS. While
>> non-CCS compliant sensors may use the same packing, the packing itself
>> is still a "CCS packing".
> 
> Agreed.
> 
>> So why not call them that? The 8-bit format is
>> fully generic, whereas the rest are CCS packings, as defined in the CCS
>> spec (the CCS spec also specifies the content, but here were only using
>> the CCS packing).
> 
> I think an additional question is whether we actually need multiple
> media bus codes here. Are there CSI-2 receivers, or other downstream
> blocks, that will need to be configured differently to receive embedded
> data from a CCS (in the sense of using CCS packing, not being fully
> CCS-compliant) RAW10 sensor compared to a RAW12 sensor ? Or will the
> CSI-2 receiver just send bytes to memory without caring ? In the latter
> case, it will be the application that will need to know how the data is
> packed to ignore one byte out of three for RAW12 and out of five for
> RAW10.

I would presume all receivers can operate in "just write what you 
receive" mode, in which case the packing doesn't matter as you say, but 
then again, I think it's safer to have mbus codes for the different 
packings.

>> Maybe they shouldn't be called "generic", but... umm... Content unaware
>> metadata formats... doesn't sound very good =).
>>
>> Also, I'm a bit confused about CSI-2 pixel and embedded data formats and
>> how we handle them.
>>
>> For MEDIA_BUS_FMT_SBGGR10_1X10, we define that the data contains 10 bits
>> per pixel, from bit 0 to 9. But CSI-2, for RAW10, actually sends it
>> differently, with the higher bits first, and the lowest bits in the
>> fourth byte. So that CSI-2 packing is implicit, kind of hidden here.
> 
> Usage of MEDIA_BUS_FMT_SBGGR10_1X10 for CSI-2 is a historical decision
> (or mistake, depending on how you look at it). It doesn't match what
> actually happens on the bus.
> 
>> Which is probably fine, as we're really only interested in the unpacked
>> data, not the CSI-2 packed. And when writing this data to memory, the
>> DMA engine can write it either as is, or unpack each pixel to a 16-bit
>> container.
>>
>> For MEDIA_BUS_FMT_META_10, we define it similarly to
>> MEDIA_BUS_FMT_SBGGR10_1X10, except the lowest 2 bits are marked as
>> padding bits. And, I think, MEDIA_BUS_FMT_META_10 implies RAW10 CSI-2
>> format.
> 
> That's how I understand it too
> 
>> However, when writing the data to memory, we don't want either
>> of the modes used in the above pixel data case, but rather we want to
>> write the data as it is in the CSI-2 bus. So, the DMA engine can either
>> reverse the RAW10 unpacking to get the wanted output format, or
>> alternatively the CSI-2 receiver could instead use RAW8 mode to avoid
>> any unpacking.
> 
> I don't think the DMA engine would "reverse the RAW10 packing" here. It
> will write bytes as they arrive (or more likely, a set of bytes at a
> time to operate on bursts). The other option would be for the DMA engine
> to drop the padding bytes, but I don't know of any CSI-2 receiver that
> can do that.

Well, for pixel data, TI CAL has the following options when writing to 
memory for, e.g. RAW12:
- RAW12 MIPI (a separate byte for lsbs)
- RAW12 Linear (12 consecutive bits per pixel)
- RAW16 (12 consecutive bits per pixel in a 16bit container)

Of these, only the RAW12 MIPI makes sense for embedded data (actually 
the whole processing block can be and should be skipped for embedded 
data, so it's more like CAL is receiving RAW8 and writing out RAW8).

Or, if it would be possible, skipping the padding bytes. Which looks 
like it is almost possible with TI CAL, but not quite: it can write with 
RAW8, but afaics that always takes the lowest bits of the 12-bit pixel, 
but here we'd like it to take the highest bits.

Anyway, I don't think there's any issue here, I was just writing down my 
thoughts. As long as we have good mbus formats which describe the 
packing fully, and V4L2 formats which describe the memory formats, we 
should be fine.

  Tomi

>> Does the above make any sense? I'm a bit confused about all the packings
>> and unpackings =). Does MEDIA_BUS_FMT_META_10 mean that the CSI-2 TX
>> uses RAW10 CSI-2 packing, but the receiver uses RAW8 CSI-2 packing?
>>
>>> If there is a need for new generic formats that do not match this, we can
>>> always add more. But the point is: drivers for devices that do not produce
>>> the data should never deal with (device) specific formats.
>>>
>>> What comes to CSI-2 and these formats --- on parallel buses you might have
>>> the data aligned to the least significant bits instead. But is there a need
>>> to transport such data on parallel buses, at least so it would be expressed
>>> in mbus formats?
>>
>> I have a parallel sensor that sends embedded data. I'll have a look how
>> it organizes the data.
> 

Re: [PATCH v3 04/10] media: uapi: Add generic serial metadata mbus formats

[Laurent Pinchart]

Hi Tomi,

On Wed, Sep 06, 2023 at 11:28:37AM +0300, Tomi Valkeinen wrote:
> On 05/09/2023 19:38, Laurent Pinchart wrote:
> > On Thu, Aug 24, 2023 at 11:26:56AM +0300, Tomi Valkeinen wrote:
> >> On 24/08/2023 10:24, Sakari Ailus wrote:
> >>> On Wed, Aug 23, 2023 at 04:16:13PM +0300, Tomi Valkeinen wrote:
> >>>> On 08/08/2023 10:55, Sakari Ailus wrote:
> >>>>> Add generic serial metadata mbus formats. These formats describe data
> >>>>> width and packing but not the content itself. The reason for specifying
> >>>>> such formats is that the formats as such are fairly device specific but
> >>>>> they are still handled by CSI-2 receiver drivers that should not be aware
> > 
> > Do we want these formats to be CSI-2-specific ?
> >
> >>>>> of device specific formats. What makes generic metadata formats possible
> >>>>> is that these formats are parsed by software only, after capturing the
> >>>>> data to system memory.
> >>>>
> >>>> If I'm not mistaken, the CSI-2 spec doesn't say much about embedded data,
> >>>> except that it may exist. Afaics, in CSI-2, the embedded data is split into
> >>>> "lines", although the amount of data can be different than in the video
> >>>> lines.
> >>>>
> >>>> The CCS specs talks more about embedded data. Some of it is quite odd, like:
> >>>> "The length of the embedded data line shall not exceed the length of the
> >>>> image data line. The embedded data line should have the same length as the
> >>>> image data line.". I think it means the embedded line can be shorter than
> >>>> image line, but not longer.
> >>>
> >>> That's what it means, yes. The CCS also has means to obtain the actual
> >>> length --- frame format descriptors.
> >>>
> >>>> CCS also says that an embedded line should use 0x07 as padding at the end of
> >>>> the line, if there's less data than the image line.
> >>>>
> >>>> CCS also talks about how the embedded data would be packed, and in some
> >>>> cases that packing would be the same as for pixel data.
> >>>
> >>> In fact the packing is the same as for pixel data: the CSI-2 does not
> >>> really differentiate there.
> >>
> >> If I understand the CCS spec right, the packing is not the same as for
> >> the pixel data. You can have RAW12 pixel data but 8-byte embedded data.
> >> But if you meant that the different packing style options are the same
> >> for pixel and embedded data, yes.
> > 
> > The idea is to allow CSI-2 receivers to unpack embedded data the same
> > way as pixel data at the hardware level, and guarantee that no embedded
> 
> Yes, but my comment was to Sakari's "the packing is the same as for 
> pixel data" comment. Afaics, there's nothing in CSI-2 or CCS that says 
> the packing is the same for pixel data and embedded data.
> 
> In fact, the CSS says "By default, top-embedded data shall use the same 
> data packing as the Visible pixel data with one embedded data byte per 
> pixel", implying that it would not always be the case. It continues that 
> for RAW16/RAW20/RAW24 pixels the embedded data could be sent with 
> RAW8/RAW10/RAW12.

The spec if a bit ambiguous :-S "by default" implies that something else
could be done, but it's not clear what the other options are.

> That was just for top-embedded data, and I couldn't right away see 
> mention of other embedded data.

The specification also mentions PDAF data for instance.

> > data significant byte gets split across multiple samples. When unpacking
> > RAW10 pixel data, the CSI-2 receiver will take 5 bytes and output 4
> > samples on a 10-bit wide bus. For embedded data, it can conceptually do
> > that same, and output embedded data with one byte per sample, aligned on
> > the MSB of the 10-bit bus. However, in practice, I think the CSI-2
> > receiver will simply receive the bytes and send them to the DMA engine,
> > without unpacking and then repacking.
> 
> At least TI CAL CSI-2 RX can extract the data from the CSI-2 stream 
> using different unpacking for pixel and embedded data, and then 
> (optionally) pack it in different ways when writing to memory.

Does this mean that the CAL could take RAW10 embedded data and drop the
padding bytes when writing to memory ?

> > This can also be viewed as away to allow CSI-2 transmitters to serialize
> > pixel data and embedded data the same way. For a RAW10 sensor, for
> > instance, the embedded data would be generated on an internal 10-bit
> > wide bus, aligned to the MSB, and would go through CSI-2 packing the
> > same way that pixels do. That's nice, but it's internal to the sensor,
> > so we don't need to expose it to userspace on that side.
> > 
> >>>> But I don't think these formats are generic. They're defined in CCS, so
> >>>> shouldn't the format be, e.g., MEDIA_BUS_FMT_META_CCS_RAW_12 or such?
> >>>
> >>> The reason for having generic definitions is that we do not need receiver
> >>> drivers to be aware of formats that are specific to another driver.
> >>
> >> Yes, I agree with that, and that's not my point here. But "generic"
> >> doesn't mean the definitions are not for some particular bus or
> >> standard, "generic" just means that it doesn't specify the content, only
> >> the packing.
> >>
> >> My point is that these packings seem to be specific to CCS. While
> >> non-CCS compliant sensors may use the same packing, the packing itself
> >> is still a "CCS packing".
> > 
> > Agreed.
> > 
> >> So why not call them that? The 8-bit format is
> >> fully generic, whereas the rest are CCS packings, as defined in the CCS
> >> spec (the CCS spec also specifies the content, but here were only using
> >> the CCS packing).
> > 
> > I think an additional question is whether we actually need multiple
> > media bus codes here. Are there CSI-2 receivers, or other downstream
> > blocks, that will need to be configured differently to receive embedded
> > data from a CCS (in the sense of using CCS packing, not being fully
> > CCS-compliant) RAW10 sensor compared to a RAW12 sensor ? Or will the
> > CSI-2 receiver just send bytes to memory without caring ? In the latter
> > case, it will be the application that will need to know how the data is
> > packed to ignore one byte out of three for RAW12 and out of five for
> > RAW10.
> 
> I would presume all receivers can operate in "just write what you 
> receive" mode, in which case the packing doesn't matter as you say, but 
> then again, I think it's safer to have mbus codes for the different 
> packings.
> 
> >> Maybe they shouldn't be called "generic", but... umm... Content unaware
> >> metadata formats... doesn't sound very good =).
> >>
> >> Also, I'm a bit confused about CSI-2 pixel and embedded data formats and
> >> how we handle them.
> >>
> >> For MEDIA_BUS_FMT_SBGGR10_1X10, we define that the data contains 10 bits
> >> per pixel, from bit 0 to 9. But CSI-2, for RAW10, actually sends it
> >> differently, with the higher bits first, and the lowest bits in the
> >> fourth byte. So that CSI-2 packing is implicit, kind of hidden here.
> > 
> > Usage of MEDIA_BUS_FMT_SBGGR10_1X10 for CSI-2 is a historical decision
> > (or mistake, depending on how you look at it). It doesn't match what
> > actually happens on the bus.
> > 
> >> Which is probably fine, as we're really only interested in the unpacked
> >> data, not the CSI-2 packed. And when writing this data to memory, the
> >> DMA engine can write it either as is, or unpack each pixel to a 16-bit
> >> container.
> >>
> >> For MEDIA_BUS_FMT_META_10, we define it similarly to
> >> MEDIA_BUS_FMT_SBGGR10_1X10, except the lowest 2 bits are marked as
> >> padding bits. And, I think, MEDIA_BUS_FMT_META_10 implies RAW10 CSI-2
> >> format.
> > 
> > That's how I understand it too
> > 
> >> However, when writing the data to memory, we don't want either
> >> of the modes used in the above pixel data case, but rather we want to
> >> write the data as it is in the CSI-2 bus. So, the DMA engine can either
> >> reverse the RAW10 unpacking to get the wanted output format, or
> >> alternatively the CSI-2 receiver could instead use RAW8 mode to avoid
> >> any unpacking.
> > 
> > I don't think the DMA engine would "reverse the RAW10 packing" here. It
> > will write bytes as they arrive (or more likely, a set of bytes at a
> > time to operate on bursts). The other option would be for the DMA engine
> > to drop the padding bytes, but I don't know of any CSI-2 receiver that
> > can do that.
> 
> Well, for pixel data, TI CAL has the following options when writing to 
> memory for, e.g. RAW12:
> - RAW12 MIPI (a separate byte for lsbs)
> - RAW12 Linear (12 consecutive bits per pixel)
> - RAW16 (12 consecutive bits per pixel in a 16bit container)
> 
> Of these, only the RAW12 MIPI makes sense for embedded data (actually 
> the whole processing block can be and should be skipped for embedded 
> data, so it's more like CAL is receiving RAW8 and writing out RAW8).
> 
> Or, if it would be possible, skipping the padding bytes. Which looks 
> like it is almost possible with TI CAL, but not quite: it can write with 
> RAW8, but afaics that always takes the lowest bits of the 12-bit pixel, 
> but here we'd like it to take the highest bits.

That's a weird one, even for image data, dropping the MSBs doesn't make
much sense, while writing the RAW8 MSB when receiving RAW10 or RAW12
would be useful.

> Anyway, I don't think there's any issue here, I was just writing down my 
> thoughts. As long as we have good mbus formats which describe the 
> packing fully, and V4L2 formats which describe the memory formats, we 
> should be fine.
> 
> >> Does the above make any sense? I'm a bit confused about all the packings
> >> and unpackings =). Does MEDIA_BUS_FMT_META_10 mean that the CSI-2 TX
> >> uses RAW10 CSI-2 packing, but the receiver uses RAW8 CSI-2 packing?
> >>
> >>> If there is a need for new generic formats that do not match this, we can
> >>> always add more. But the point is: drivers for devices that do not produce
> >>> the data should never deal with (device) specific formats.
> >>>
> >>> What comes to CSI-2 and these formats --- on parallel buses you might have
> >>> the data aligned to the least significant bits instead. But is there a need
> >>> to transport such data on parallel buses, at least so it would be expressed
> >>> in mbus formats?
> >>
> >> I have a parallel sensor that sends embedded data. I'll have a look how
> >> it organizes the data.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v3 06/10] media: uapi: Add a macro to tell whether an mbus code is metadata

[Sakari Ailus]

Hi Laurent,

On Tue, Sep 05, 2023 at 08:06:04PM +0300, Laurent Pinchart wrote:
> On Tue, Sep 05, 2023 at 10:37:45AM +0000, Sakari Ailus wrote:
> > On Tue, Sep 05, 2023 at 12:47:21PM +0300, Tomi Valkeinen wrote:
> > > On 08/08/2023 10:55, Sakari Ailus wrote:
> > > > Add a macro to tell whether a given mbus code is metadata.
> > > > 
> > > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > > ---
> > > >   include/uapi/linux/media-bus-format.h | 3 +++
> > > >   1 file changed, 3 insertions(+)
> > > > 
> > > > diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
> > > > index 9ee031397372..2486b4178c5f 100644
> > > > --- a/include/uapi/linux/media-bus-format.h
> > > > +++ b/include/uapi/linux/media-bus-format.h
> > > > @@ -182,4 +182,7 @@
> > > >   #define MEDIA_BUS_FMT_META_20			0x8006
> > > >   #define MEDIA_BUS_FMT_META_24			0x8007
> > > > +#define MEDIA_BUS_FMT_IS_META(code)		\
> > > > +	((code) & 0xf000 == 0x7000 || (code) & 0xf000 == 0x8000)
> > > > +
> > > >   #endif /* __LINUX_MEDIA_BUS_FORMAT_H */
> > > 
> > > mbus code seems to be u32, so the above won't work. Maybe:
> > > 
> > > (((code) & ~0xfffULL) == 0x7000 || ((code) & ~0xfffULL) == 0x8000)
> > > 
> > > Also, embedded formats with 0x9nnn codes are added later in the series.
> > 
> > Thanks, I'll address these for v4.
> 
> It would be nice to make this an inline function, to avoid evaluating
> the code twice. I think you can move it to an internal kernel header, it
> doesn't need to be exposed to userspace.

This was meant for user space consumption for setting the buffer type to
the video node accordingly. It's certainly possible to do that without this
macro, too.

-- 
Regards,

Sakari Ailus

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

[Sakari Ailus]

Hi Laurent,

On Tue, Sep 05, 2023 at 07:47:20PM +0300, Laurent Pinchart wrote:
> Hi Sakari,
> 
> On Fri, Aug 11, 2023 at 09:11:39AM +0000, Sakari Ailus wrote:
> > On Fri, Aug 11, 2023 at 08:31:16AM +0200, Jacopo Mondi wrote:
> > > > > +V4L2_META_FMT_GENERIC_CSI2_10
> > > > > +-----------------------------
> > > > > +
> > > > > +V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
> > > > > +for each 8 bits of data. Every four bytes of metadata is followed by a single
> > > > > +byte of padding. The way the data is stored follows the CSI-2 specification.
> > > > > +
> > > > > +This format is also used on CSI-2 on 20 bits per sample format that packs two
> > > > > +bytes of metadata into one sample.
> > > > > +
> > > > > +This format is little endian.
> > > > > +
> > > > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
> > > > > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > > >
> > > > I think you should document whether the padding is always 0 or can be any value.
> > > > Perhaps 'X' is a better 'name' for the padding byte in the latter case.
> > > 
> > > Did I get this right that this format is supposed to work as the RAW10
> > > CSI-2 packed image format, where 4 bytes contain the higher 8 bits of
> > > the 10 bits sample and the 5th byte every 4 contains the lower 2 bits of
> > > the previous 4 sample ?
> > > 
> > > If that's the case, is 'padding' the correct term here ?
> > 
> > What else would you call it? It'll be zeros that exist just due to the bit
> > depth used and as such not interesting at all.
> 
> It's actually not 0, CCS requires the padding bytes to be 0x55.
> 
> I wonder if the conformance test suite tests the contents of the padding
> bytes.

I don't know. I could add the value is unspecified but as it has not been
specified, there's no change in meaning (just size).

-- 
Regards,

Sakari Ailus

Re: [PATCH v3 04/10] media: uapi: Add generic serial metadata mbus formats

[Tomi Valkeinen]

On 06/09/2023 14:31, Laurent Pinchart wrote:
> Hi Tomi,
> 
> On Wed, Sep 06, 2023 at 11:28:37AM +0300, Tomi Valkeinen wrote:
>> On 05/09/2023 19:38, Laurent Pinchart wrote:
>>> On Thu, Aug 24, 2023 at 11:26:56AM +0300, Tomi Valkeinen wrote:
>>>> On 24/08/2023 10:24, Sakari Ailus wrote:
>>>>> On Wed, Aug 23, 2023 at 04:16:13PM +0300, Tomi Valkeinen wrote:
>>>>>> On 08/08/2023 10:55, Sakari Ailus wrote:
>>>>>>> Add generic serial metadata mbus formats. These formats describe data
>>>>>>> width and packing but not the content itself. The reason for specifying
>>>>>>> such formats is that the formats as such are fairly device specific but
>>>>>>> they are still handled by CSI-2 receiver drivers that should not be aware
>>>
>>> Do we want these formats to be CSI-2-specific ?
>>>
>>>>>>> of device specific formats. What makes generic metadata formats possible
>>>>>>> is that these formats are parsed by software only, after capturing the
>>>>>>> data to system memory.
>>>>>>
>>>>>> If I'm not mistaken, the CSI-2 spec doesn't say much about embedded data,
>>>>>> except that it may exist. Afaics, in CSI-2, the embedded data is split into
>>>>>> "lines", although the amount of data can be different than in the video
>>>>>> lines.
>>>>>>
>>>>>> The CCS specs talks more about embedded data. Some of it is quite odd, like:
>>>>>> "The length of the embedded data line shall not exceed the length of the
>>>>>> image data line. The embedded data line should have the same length as the
>>>>>> image data line.". I think it means the embedded line can be shorter than
>>>>>> image line, but not longer.
>>>>>
>>>>> That's what it means, yes. The CCS also has means to obtain the actual
>>>>> length --- frame format descriptors.
>>>>>
>>>>>> CCS also says that an embedded line should use 0x07 as padding at the end of
>>>>>> the line, if there's less data than the image line.
>>>>>>
>>>>>> CCS also talks about how the embedded data would be packed, and in some
>>>>>> cases that packing would be the same as for pixel data.
>>>>>
>>>>> In fact the packing is the same as for pixel data: the CSI-2 does not
>>>>> really differentiate there.
>>>>
>>>> If I understand the CCS spec right, the packing is not the same as for
>>>> the pixel data. You can have RAW12 pixel data but 8-byte embedded data.
>>>> But if you meant that the different packing style options are the same
>>>> for pixel and embedded data, yes.
>>>
>>> The idea is to allow CSI-2 receivers to unpack embedded data the same
>>> way as pixel data at the hardware level, and guarantee that no embedded
>>
>> Yes, but my comment was to Sakari's "the packing is the same as for
>> pixel data" comment. Afaics, there's nothing in CSI-2 or CCS that says
>> the packing is the same for pixel data and embedded data.
>>
>> In fact, the CSS says "By default, top-embedded data shall use the same
>> data packing as the Visible pixel data with one embedded data byte per
>> pixel", implying that it would not always be the case. It continues that
>> for RAW16/RAW20/RAW24 pixels the embedded data could be sent with
>> RAW8/RAW10/RAW12.
> 
> The spec if a bit ambiguous :-S "by default" implies that something else
> could be done, but it's not clear what the other options are.
> 
>> That was just for top-embedded data, and I couldn't right away see
>> mention of other embedded data.
> 
> The specification also mentions PDAF data for instance.
> 
>>> data significant byte gets split across multiple samples. When unpacking
>>> RAW10 pixel data, the CSI-2 receiver will take 5 bytes and output 4
>>> samples on a 10-bit wide bus. For embedded data, it can conceptually do
>>> that same, and output embedded data with one byte per sample, aligned on
>>> the MSB of the 10-bit bus. However, in practice, I think the CSI-2
>>> receiver will simply receive the bytes and send them to the DMA engine,
>>> without unpacking and then repacking.
>>
>> At least TI CAL CSI-2 RX can extract the data from the CSI-2 stream
>> using different unpacking for pixel and embedded data, and then
>> (optionally) pack it in different ways when writing to memory.
> 
> Does this mean that the CAL could take RAW10 embedded data and drop the
> padding bytes when writing to memory ?
> 
>>> This can also be viewed as away to allow CSI-2 transmitters to serialize
>>> pixel data and embedded data the same way. For a RAW10 sensor, for
>>> instance, the embedded data would be generated on an internal 10-bit
>>> wide bus, aligned to the MSB, and would go through CSI-2 packing the
>>> same way that pixels do. That's nice, but it's internal to the sensor,
>>> so we don't need to expose it to userspace on that side.
>>>
>>>>>> But I don't think these formats are generic. They're defined in CCS, so
>>>>>> shouldn't the format be, e.g., MEDIA_BUS_FMT_META_CCS_RAW_12 or such?
>>>>>
>>>>> The reason for having generic definitions is that we do not need receiver
>>>>> drivers to be aware of formats that are specific to another driver.
>>>>
>>>> Yes, I agree with that, and that's not my point here. But "generic"
>>>> doesn't mean the definitions are not for some particular bus or
>>>> standard, "generic" just means that it doesn't specify the content, only
>>>> the packing.
>>>>
>>>> My point is that these packings seem to be specific to CCS. While
>>>> non-CCS compliant sensors may use the same packing, the packing itself
>>>> is still a "CCS packing".
>>>
>>> Agreed.
>>>
>>>> So why not call them that? The 8-bit format is
>>>> fully generic, whereas the rest are CCS packings, as defined in the CCS
>>>> spec (the CCS spec also specifies the content, but here were only using
>>>> the CCS packing).
>>>
>>> I think an additional question is whether we actually need multiple
>>> media bus codes here. Are there CSI-2 receivers, or other downstream
>>> blocks, that will need to be configured differently to receive embedded
>>> data from a CCS (in the sense of using CCS packing, not being fully
>>> CCS-compliant) RAW10 sensor compared to a RAW12 sensor ? Or will the
>>> CSI-2 receiver just send bytes to memory without caring ? In the latter
>>> case, it will be the application that will need to know how the data is
>>> packed to ignore one byte out of three for RAW12 and out of five for
>>> RAW10.
>>
>> I would presume all receivers can operate in "just write what you
>> receive" mode, in which case the packing doesn't matter as you say, but
>> then again, I think it's safer to have mbus codes for the different
>> packings.
>>
>>>> Maybe they shouldn't be called "generic", but... umm... Content unaware
>>>> metadata formats... doesn't sound very good =).
>>>>
>>>> Also, I'm a bit confused about CSI-2 pixel and embedded data formats and
>>>> how we handle them.
>>>>
>>>> For MEDIA_BUS_FMT_SBGGR10_1X10, we define that the data contains 10 bits
>>>> per pixel, from bit 0 to 9. But CSI-2, for RAW10, actually sends it
>>>> differently, with the higher bits first, and the lowest bits in the
>>>> fourth byte. So that CSI-2 packing is implicit, kind of hidden here.
>>>
>>> Usage of MEDIA_BUS_FMT_SBGGR10_1X10 for CSI-2 is a historical decision
>>> (or mistake, depending on how you look at it). It doesn't match what
>>> actually happens on the bus.
>>>
>>>> Which is probably fine, as we're really only interested in the unpacked
>>>> data, not the CSI-2 packed. And when writing this data to memory, the
>>>> DMA engine can write it either as is, or unpack each pixel to a 16-bit
>>>> container.
>>>>
>>>> For MEDIA_BUS_FMT_META_10, we define it similarly to
>>>> MEDIA_BUS_FMT_SBGGR10_1X10, except the lowest 2 bits are marked as
>>>> padding bits. And, I think, MEDIA_BUS_FMT_META_10 implies RAW10 CSI-2
>>>> format.
>>>
>>> That's how I understand it too
>>>
>>>> However, when writing the data to memory, we don't want either
>>>> of the modes used in the above pixel data case, but rather we want to
>>>> write the data as it is in the CSI-2 bus. So, the DMA engine can either
>>>> reverse the RAW10 unpacking to get the wanted output format, or
>>>> alternatively the CSI-2 receiver could instead use RAW8 mode to avoid
>>>> any unpacking.
>>>
>>> I don't think the DMA engine would "reverse the RAW10 packing" here. It
>>> will write bytes as they arrive (or more likely, a set of bytes at a
>>> time to operate on bursts). The other option would be for the DMA engine
>>> to drop the padding bytes, but I don't know of any CSI-2 receiver that
>>> can do that.
>>
>> Well, for pixel data, TI CAL has the following options when writing to
>> memory for, e.g. RAW12:
>> - RAW12 MIPI (a separate byte for lsbs)
>> - RAW12 Linear (12 consecutive bits per pixel)
>> - RAW16 (12 consecutive bits per pixel in a 16bit container)
>>
>> Of these, only the RAW12 MIPI makes sense for embedded data (actually
>> the whole processing block can be and should be skipped for embedded
>> data, so it's more like CAL is receiving RAW8 and writing out RAW8).
>>
>> Or, if it would be possible, skipping the padding bytes. Which looks
>> like it is almost possible with TI CAL, but not quite: it can write with
>> RAW8, but afaics that always takes the lowest bits of the 12-bit pixel,
>> but here we'd like it to take the highest bits.
> 
> That's a weird one, even for image data, dropping the MSBs doesn't make
> much sense, while writing the RAW8 MSB when receiving RAW10 or RAW12
> would be useful.

I think the RAW8 output is designed to be used only for RAW8 input. I 
haven't confirmed this behavior, but the doc says that RAW8 output 
writes bits 0-7. And if the input is RAW10/12, then that would imply 
that the MSBs are dropped.

I could try this at some point, as it's not quite clear from the docs.

  Tomi

>> Anyway, I don't think there's any issue here, I was just writing down my
>> thoughts. As long as we have good mbus formats which describe the
>> packing fully, and V4L2 formats which describe the memory formats, we
>> should be fine.
>>
>>>> Does the above make any sense? I'm a bit confused about all the packings
>>>> and unpackings =). Does MEDIA_BUS_FMT_META_10 mean that the CSI-2 TX
>>>> uses RAW10 CSI-2 packing, but the receiver uses RAW8 CSI-2 packing?
>>>>
>>>>> If there is a need for new generic formats that do not match this, we can
>>>>> always add more. But the point is: drivers for devices that do not produce
>>>>> the data should never deal with (device) specific formats.
>>>>>
>>>>> What comes to CSI-2 and these formats --- on parallel buses you might have
>>>>> the data aligned to the least significant bits instead. But is there a need
>>>>> to transport such data on parallel buses, at least so it would be expressed
>>>>> in mbus formats?
>>>>
>>>> I have a parallel sensor that sends embedded data. I'll have a look how
>>>> it organizes the data.
> 

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

[Sakari Ailus]

Hi Laurent,

On Tue, Sep 05, 2023 at 07:55:58PM +0300, Laurent Pinchart wrote:
> On Tue, Aug 08, 2023 at 10:55:35AM +0300, Sakari Ailus wrote:
> > Generic 8-bit metadata formats define the in-memory data layout but not
> > the format of the data itself. The reasoning for having such formats is to
> > allow CSI-2 receiver drivers to receive and DMA drivers to write the data
> > to memory without knowing a large number of device specific formats.
> > 
> > These formats may be used only in conjunction of a Media controller
> > pipeline where the internal pad of the source sub-device defines the
> > specific format of the data (using an mbus code).
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  .../userspace-api/media/v4l/meta-formats.rst  |   1 +
> >  .../media/v4l/metafmt-generic.rst             | 331 ++++++++++++++++++
> >  drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
> >  include/uapi/linux/videodev2.h                |   9 +
> >  4 files changed, 349 insertions(+)
> >  create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
> > index 0bb61fc5bc00..919f595576b9 100644
> > --- a/Documentation/userspace-api/media/v4l/meta-formats.rst
> > +++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
> > @@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only.
> >      metafmt-vsp1-hgo
> >      metafmt-vsp1-hgt
> >      metafmt-vivid
> > +    metafmt-generic
> > diff --git a/Documentation/userspace-api/media/v4l/metafmt-generic.rst b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > new file mode 100644
> > index 000000000000..a27bfc721edf
> > --- /dev/null
> > +++ b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > @@ -0,0 +1,331 @@
> > +.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
> > +
> > +**************************************************************************************************************************************************************************************************************************************************************************************************************************
> > +V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O'), V4L2_META_FMT_GENERIC_CSI2_2_24 ('MC2O')
> > +**************************************************************************************************************************************************************************************************************************************************************************************************************************
> > +
> > +
> > +Generic line-based metadata formats
> > +
> > +
> > +Description
> > +===========
> > +
> > +These generic line-based metadata formats define the memory layout of the data
> > +without defining the format or meaning of the metadata itself. These formats may
> > +only be used with a Media controller pipeline where the more specific format is
> > +defined in an :ref:`internal source pad <MEDIA-PAD-FL-INTERNAL>` of the source
> > +sub-device. See also :ref:`source routes <v4l2-subdev-source-routes>`.
> > +
> > +.. _v4l2-meta-fmt-generic-8:
> > +
> > +V4L2_META_FMT_GENERIC_8
> > +-----------------------
> > +
> > +The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format.
> > +
> > +This format is also used on CSI-2 on both 8 bits per sample as well as on
> 
> s/also on/by/
> 
> I would also mention "MIPI CCS" instead of "CSI-2".

If CCS were to be mentioned here, then all uses of this format should be
included as well.

> 
> > +16 bits per sample when two bytes of metadata are packed into one sample.
> 
> "bits per sample" is very ill-defined for metadata, as there's no
> sample. I would write "for both the RAW8 packing and the 2 bytes RAW16
> packing" or something similar.
> 
> Similar comments for below.

From CSI-2 bus point of view there's no difference between pixel and
embedded data when it comes to encoding that data. "Sample" is the next
best term beyond "pixel", as the bus can carry samples that may or may not
be pixel data. But I'm fine with changing the wording if you think it makes
it more understandable.

> 
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_8.**
> > +Each cell is one byte. "M" denotes a byte of metadata.
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|
> > +
> > +.. flat-table::
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - M\ :sub:`10`
> > +      - M\ :sub:`20`
> > +      - M\ :sub:`30`
> > +    * - start + 4:
> > +      - M\ :sub:`01`
> > +      - M\ :sub:`11`
> > +      - M\ :sub:`21`
> > +      - M\ :sub:`31`
> > +
> > +.. _v4l2-meta-fmt-generic-csi2-10:
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_10
> > +-----------------------------
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
> > +for each 8 bits of data. Every four bytes of metadata is followed by a single
> > +byte of padding.
> 
> It sounds really weird to write that this format writes 10 bits for each
> 8 bits of data, when essentially it adds a packing byte every four
> bytes.

That's how the hardware has been implemented and probably there is a
hardware implementation related reason for this.

> 
> > The way the data is stored follows the CSI-2 specification.
> 
> Again, it's CCS, not CSI-2.

The packing comes from CSI-2 but CCS adds padding. I'll add this
information.

> 
> > +
> > +This format is also used on CSI-2 on 20 bits per sample format that packs two
> > +bytes of metadata into one sample.
> > +
> > +This format is little endian.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
> > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
> > +
> > +.. flat-table::
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - M\ :sub:`10`
> > +      - M\ :sub:`20`
> > +      - M\ :sub:`30`
> > +      - p
> > +    * - start + 5:
> > +      - M\ :sub:`01`
> > +      - M\ :sub:`11`
> > +      - M\ :sub:`21`
> > +      - M\ :sub:`31`
> > +      - p
> > +
> > +.. _v4l2-meta-fmt-generic-csi2-12:
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_12
> > +-----------------------------
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_12 contains packed 8-bit generic metadata, 12 bits
> > +for each 8 bits of data. Every four bytes of metadata is followed by two bytes
> > +of padding. The way the data is stored follows the CSI-2 specification.
> > +
> > +This format is little endian.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_12.**
> > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> > +
> > +.. flat-table::
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - M\ :sub:`10`
> > +      - M\ :sub:`20`
> > +      - M\ :sub:`30`
> > +      - p
> > +      - p
> > +    * - start + 6:
> > +      - M\ :sub:`01`
> > +      - M\ :sub:`11`
> > +      - M\ :sub:`21`
> > +      - M\ :sub:`31`
> > +      - p
> > +      - p
> > +
> > +.. _v4l2-meta-fmt-generic-csi2-14:
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_14
> > +-----------------------------
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_14 contains packed 8-bit generic metadata, 14 bits
> > +for each 8 bits of data. Every four bytes of metadata is followed by three
> > +bytes of padding. The way the data is stored follows the CSI-2 specification.
> > +
> > +This format is little endian.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_14.**
> > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{.8cm}|
> > +
> > +.. flat-table::
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - M\ :sub:`10`
> > +      - M\ :sub:`20`
> > +      - M\ :sub:`30`
> > +      - p
> > +      - p
> > +      - p
> > +    * - start + 7:
> > +      - M\ :sub:`01`
> > +      - M\ :sub:`11`
> > +      - M\ :sub:`21`
> > +      - M\ :sub:`31`
> > +      - p
> > +      - p
> > +      - p
> > +
> > +.. _v4l2-meta-fmt-generic-csi2-16:
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_16
> > +-----------------------------
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_16 contains packed 8-bit generic metadata, 16 bits
> > +for each 8 bits of data. Every byte of metadata is followed by one byte of
> > +padding. The way the data is stored follows the CSI-2 specification.
> > +
> > +This format is little endian.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_16.**
> > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|
> > +
> > +.. flat-table::
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - p
> > +      - M\ :sub:`10`
> > +      - p
> > +      - M\ :sub:`20`
> > +      - p
> > +      - M\ :sub:`30`
> > +      - p
> > +    * - start + 8:
> > +      - M\ :sub:`01`
> > +      - p
> > +      - M\ :sub:`11`
> > +      - p
> > +      - M\ :sub:`21`
> > +      - p
> > +      - M\ :sub:`31`
> > +      - p
> > +
> > +.. _v4l2-meta-fmt-generic-csi2-20:
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_20
> > +-----------------------------
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_20 contains packed 8-bit generic metadata, 20 bits
> > +for each 8 bits of data. Every byte of metadata is followed by alternating one
> > +and two bytes of padding. The way the data is stored follows the CSI-2
> > +specification.
> > +
> > +This format is little endian.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_20.**
> > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> > +
> > +.. flat-table::
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8 8 8 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - p
> > +      - M\ :sub:`10`
> > +      - p
> > +      - p
> > +      - M\ :sub:`20`
> > +      - p
> > +      - M\ :sub:`30`
> > +      - p
> > +      - p
> > +    * - start + 10:
> > +      - M\ :sub:`01`
> > +      - p
> > +      - M\ :sub:`11`
> > +      - p
> > +      - p
> > +      - M\ :sub:`21`
> > +      - p
> > +      - M\ :sub:`31`
> > +      - p
> > +      - p
> > +
> > +.. _v4l2-meta-fmt-generic-csi2-24:
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_24
> > +-----------------------------
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_24 contains packed 8-bit generic metadata, 24 bits
> > +for each 8 bits of data. Every byte of metadata is followed by two bytes of
> > +padding. The way the data is stored follows the CSI-2 specification.
> > +
> > +This format is little endian.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_24.**
> > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> > +
> > +.. flat-table::
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8 8 8 8 8 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - p
> > +      - p
> > +      - M\ :sub:`10`
> > +      - p
> > +      - p
> > +      - M\ :sub:`20`
> > +      - p
> > +      - p
> > +      - M\ :sub:`30`
> > +      - p
> > +      - p
> > +    * - start + 12:
> > +      - M\ :sub:`01`
> > +      - p
> > +      - p
> > +      - M\ :sub:`11`
> > +      - p
> > +      - p
> > +      - M\ :sub:`21`
> > +      - p
> > +      - p
> > +      - M\ :sub:`31`
> > +      - p
> > +      - p
> > +
> > +.. _v4l2-meta-fmt-generic-csi2-2-24:
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_2_24
> > +-------------------------------
> > +
> > +V4L2_META_FMT_GENERIC_CSI2_2_24 contains packed 8-bit generic metadata, 24 bits
> > +for each two times 8 bits of data. Every two bytes of metadata are followed by
> > +one byte of padding. The way the data is stored follows the CSI-2
> > +specification.
> > +
> > +This format is little endian.
> > +
> > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_2_24.**
> > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > +
> > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
> > +
> > +.. flat-table::
> > +    :header-rows:  0
> > +    :stub-columns: 0
> > +    :widths: 12 8 8 8 8 8 8
> > +
> > +    * - start + 0:
> > +      - M\ :sub:`00`
> > +      - M\ :sub:`10`
> > +      - p
> > +      - M\ :sub:`20`
> > +      - M\ :sub:`30`
> > +      - p
> > +    * - start + 6:
> > +      - M\ :sub:`01`
> > +      - M\ :sub:`11`
> > +      - p
> > +      - M\ :sub:`21`
> > +      - M\ :sub:`31`
> > +      - p
> > +
> > diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> > index f4d9d6279094..fbbddc333a30 100644
> > --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> > +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> > @@ -1452,6 +1452,14 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
> >  	case V4L2_PIX_FMT_Y210:		descr = "10-bit YUYV Packed"; break;
> >  	case V4L2_PIX_FMT_Y212:		descr = "12-bit YUYV Packed"; break;
> >  	case V4L2_PIX_FMT_Y216:		descr = "16-bit YUYV Packed"; break;
> > +	case V4L2_META_FMT_GENERIC_8:	descr = "8-bit Generic Metadata"; break;
> > +	case V4L2_META_FMT_GENERIC_CSI2_10:	descr = "8b Generic Meta, 10b CSI-2"; break;
> > +	case V4L2_META_FMT_GENERIC_CSI2_12:	descr = "8b Generic Meta, 12b CSI-2"; break;
> > +	case V4L2_META_FMT_GENERIC_CSI2_14:	descr = "8b Generic Meta, 14b CSI-2"; break;
> > +	case V4L2_META_FMT_GENERIC_CSI2_16:	descr = "8b Generic Meta, 16b CSI-2"; break;
> > +	case V4L2_META_FMT_GENERIC_CSI2_20:	descr = "8b Generic Meta, 20b CSI-2"; break;
> > +	case V4L2_META_FMT_GENERIC_CSI2_24:	descr = "8b Generic Meta, 24b CSI-2"; break;
> > +	case V4L2_META_FMT_GENERIC_CSI2_2_24:	descr = "2x8b Generic Meta, 24b CSI-2"; break;
> >  
> >  	default:
> >  		/* Compressed formats */
> > diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> > index 78260e5d9985..b4284a564025 100644
> > --- a/include/uapi/linux/videodev2.h
> > +++ b/include/uapi/linux/videodev2.h
> > @@ -838,6 +838,15 @@ struct v4l2_pix_format {
> >  #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
> >  #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
> >  
> > +#define V4L2_META_FMT_GENERIC_8		v4l2_fourcc('M', 'E', 'T', '8') /* Generic 8-bit metadata */
> > +#define V4L2_META_FMT_GENERIC_CSI2_10	v4l2_fourcc('M', 'C', '1', 'A') /* 10-bit CSI-2 packed 8-bit metadata */
> > +#define V4L2_META_FMT_GENERIC_CSI2_12	v4l2_fourcc('M', 'C', '1', 'C') /* 12-bit CSI-2 packed 8-bit metadata */
> > +#define V4L2_META_FMT_GENERIC_CSI2_14	v4l2_fourcc('M', 'C', '1', 'E') /* 14-bit CSI-2 packed 8-bit metadata */
> > +#define V4L2_META_FMT_GENERIC_CSI2_16	v4l2_fourcc('M', 'C', '1', 'G') /* 16-bit CSI-2 packed 8-bit metadata */
> > +#define V4L2_META_FMT_GENERIC_CSI2_20	v4l2_fourcc('M', 'C', '1', 'K') /* 20-bit CSI-2 packed 8-bit metadata */
> > +#define V4L2_META_FMT_GENERIC_CSI2_24	v4l2_fourcc('M', 'C', '1', 'O') /* 24-bit CSI-2 packed 8-bit metadata */
> > +#define V4L2_META_FMT_GENERIC_CSI2_2_24	v4l2_fourcc('M', 'C', '2', 'O') /* 2 bytes of 8-bit metadata, 24-bit CSI-2 packed */
> > +
> >  /* priv field value to indicates that subsequent fields are valid. */
> >  #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
> >  
> 

-- 
Regards,

Sakari Ailus

Re: [PATCH v3 10/10] media: uapi: v4l: Document source routes

[Sakari Ailus]

Hi Laurent,

Thank you for the review.

On Wed, Sep 06, 2023 at 02:17:02AM +0300, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.
> 
> On Fri, Aug 11, 2023 at 10:44:40AM +0000, Sakari Ailus wrote:
> > On Tue, Aug 08, 2023 at 10:55:24AM +0200, Hans Verkuil wrote:
> > > On 08/08/2023 09:55, Sakari Ailus wrote:
> > > > Document how internal pads are used on source routes.
> > > > 
> > > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > > ---
> > > >  .../userspace-api/media/v4l/dev-subdev.rst    | 179 ++++++++++++++++++
> > > >  1 file changed, 179 insertions(+)
> > > > 
> > > > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > > index b023918177b5..27b0fe2dc83a 100644
> > > > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > > @@ -551,6 +551,27 @@ A stream at a specific point in the media pipeline is identified by the
> > > >  sub-device and a (pad, stream) pair. For sub-devices that do not support
> > > >  multiplexed streams the 'stream' field is always 0.
> > > >  
> > > > +.. _v4l2-subdev-source-routes:
> > > > +
> > > > +Internal pads and source routes
> > > > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > > +
> > > > +Cases where a single sub-device source pad is traversed by multiple streams one
> > > > +or more of which originate from within the sub-device itself are special as
> > > > +there is no external sink pad for such routes. In those cases, the sources of
> > > > +the internally generated streams are represented by internal sink pads, which
> > > > +are sink pads that have the :ref:`MEDIA_PAD_FL_INTERNAL <MEDIA-PAD-FL-INTERNAL>`
> > > > +pad flag set.
> > > > +
> > > > +Internal pads have all the properties of an external pad, including formats and
> > > > +selections. The format in this case is the source format of the stream. An
> > > > +internal pad always has a single stream only (0).
> > > > +
> > > > +/Source routes/ are routes from an internal sink pad to a(n external) source
> 
> "a(n external)" looks weird.

I'll remove the parentheses. Both are applicable. We don't generally call
pads external but saying that here is relevant.

> 
> > > '/Source routes/' appears just like that in the generated documentation, that's
> > > probably not what you intended.
> > 
> > This was intended to be *Source routes*. I'll address it in v4.
> > 
> > > > +pad. Generally source routes are not modifiable but they can be activated and
> > > > +deactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > > > +<v4l2-subdev-routing-flags>` flag, depending on driver capabilities.
> 
> We need to update the documentation of the routing API to clearly
> explain the different models (i.e. specifying all routes, active and
> inactive, vs. including only active routes). That's out of scope for
> this patch, but it could be done in the same series.

I agree.

With this series eventually merged, I'm also hoping we could enable the
streams API.

> 
> > > > +
> > > >  Interaction between routes, streams, formats and selections
> > > >  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > >  
> > > > @@ -666,3 +687,161 @@ A common way to accomplish this is to start from the sensors and propagate the
> > > >  configurations along the stream towards the receiver,
> > > >  using :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each
> > > >  stream endpoint in each sub-device.
> > > > +
> > > > +Internal pads setup example
> > > > +^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > > +
> > > > +A simple example of a multiplexed stream setup might be as follows:
> > > > +
> > > > +- A CCS camera sensor source sub-device, with one sink pad (0), one source pad
> > > 
> > > Explain what 'CCS' means: probably a link to MIPI CCS is sufficient, but for newbies
> > > 'CCS' doesn't provide any information.
> > > 
> > > > +  (1), an internal sink pad (2) that represents the source of embedded
> > > 
> > > So since the INTERNAL flag is associated with a sink pad, this is now called an
> > > 'internal sink' instead of 'internal source' as it was before. I agree with that,
> > > but note that the phrase 'internal source' is still used in several places,
> > > including the cover letter. For the next version you post, please check for that
> > > and change the terminology since mixing the two is very confusing :-)
> > 
> > Are you referring to this patch? There don't seem to be any references to
> > "intenal source pads" left.
> > 
> > > > +  data. There are two routes, one from the sink pad to the source, and another
> > > > +  from the internal sink pad to the source pad. The embedded data stream needs
> > > > +  to be enabled by activating the related route. The configuration of the rest
> > > > +  of the CCS sub-devices is omitted from this example.
> 
> CCS is complex, with its multiple subdevs. I think a simpler raw sensor
> without a scaler, using a single subdev, would be a better example.
> 
> > > > +
> > > > +- Multiplexer bridge (Bridge). The bridge has one sink pad, connected to the
> > > > +  sensor (pad 0), and one source pad (pad 1), which outputs two streams.
> > > > +
> > > > +- Receiver in the SoC (Receiver). The receiver has a single sink pad (pad 0),
> > > > +  connected to the bridge, and two source pads (pads 1-2), going to the DMA
> > > > +  engine. The receiver demultiplexes the incoming streams to the source pads.
> > > > +
> > > > +- DMA Engines in the SoC (DMA Engine), one for each stream. Each DMA engine is
> > > > +  connected to a single source pad in the receiver.
> > > > +
> > > > +The sensor, the bridge and the receiver are modeled as V4L2 sub-devices,
> > > > +exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are
> > > > +modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
> > > > +
> > > > +To configure this pipeline, the userspace must take the following steps:
> > > > +
> > > > +1) Set up media links between entities: connect the sensors to the bridge,
> > > > +   bridge to the receiver, and the receiver to the DMA engines. This step does
> > > > +   not differ from normal non-multiplexed media controller setup.
> > > > +
> > > > +2) Configure routing
> > > > +
> > > > +.. flat-table:: Camera sensor
> > > > +    :header-rows: 1
> > > > +
> > > > +    * - Sink Pad/Stream
> > > > +      - Source Pad/Stream
> > > > +      - Routing Flags
> > > > +      - Comments
> > > > +    * - 0/0
> > > > +      - 1/0
> > > > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > > > +      - Pixel data stream from the sink pad
> > > 
> > > Huh? It is a sensor subdev, so aren't both pixel data and metadata
> > > coming from an internal sink?
> > 
> > The CCS driver exposes three sub-devices, so in this case the sink pad
> > isn't internal. Almost (?) all other sensor drivers have a single
> > sub-device only, perhaps I'll change the example to align with one of
> > those.
> 
> Ah, it looks like we have thought about the same thing :-)
> 
> > > I think you are actually describing the internal subdev of a CCS
> > > sensor here, but that is not clear from the description at all.
> > > 
> > > Since CCS sensors are relatively common (I think), it might be
> > > good to give a fully fledged example.
> > > 
> > > But suppose you have a non-CCS sensor that generates the pixel data
> > > and metadata, then you would have two internal sink pads and one
> > > source pad for two streams, right?
> > > 
> > > And very simple sensors that currently just have a single source pad
> > > could be describes as having an internal sink pad connected to the
> > > source pad by a fixed route. But we don't do that, because it is
> > > overkill. Correct? If so, then this might be useful information to add
> > > to answer the question why some sensors have internal sinks and others
> > > do not.
> > > 
> > > > +    * - 2/0
> > > > +      - 1/1
> > > > +      - **V4L2_SUBDEV_ROUTE_FL_ACTIVE**
> > > > +      - Metadata stream from the internal sink pad
> > > > +
> > > > +.. flat-table:: Bridge routing table
> > > > +    :header-rows: 1
> > > > +
> > > > +    * - Sink Pad/Stream
> > > > +      - Source Pad/Stream
> > > > +      - Routing Flags
> > > > +      - Comments
> > > > +    * - 0/0
> > > > +      - 1/0
> > > > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > > > +      - Pixel data stream from camera sensor
> > > > +    * - 0/1
> > > > +      - 1/1
> > > > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > > > +      - Metadata stream from camera sensor
> > > > +
> > > > +.. flat-table:: Receiver routing table
> > > > +    :header-rows:  1
> > > > +
> > > > +    * - Sink Pad/Stream
> > > > +      - Source Pad/Stream
> > > > +      - Routing Flags
> > > > +      - Comments
> > > > +    * - 0/0
> > > > +      - 1/0
> > > > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > > > +      - Pixel data stream from camera sensor
> > > > +    * - 0/1
> > > > +      - 2/0
> > > > +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > > > +      - Metadata stream from camera sensor
> > > > +
> > > > +The options available in sensor's routing configuration are dictated by hardware
> > > > +capabilities: typically camera sensors always produce image data stream while
> > > > +the embedded data stream typically can be either enabled or disabled.
> > > > +
> > > > +3) Configure formats and selections
> > > > +
> > > > +This example assumes that the formats are propagated from sink pad to the source
> > > > +pad as-is. The tables contain fields of both struct v4l2_subdev_format and
> > > > +struct v4l2_mbus_framefmt. The full configuration of CCS camera sensor is out of
> > > > +scope of this example.
> > > > +
> > > > +.. flat-table:: Formats set on the sub-devices. Bold values are set, others are
> > > > +                static or propagated.
> > > > +    :header-rows: 1
> > > > +    :fill-cells:
> > > > +
> > > > +    * - Sub-device
> > > > +      - Pad/Stream
> > > > +      - Width
> > > > +      - Height
> > > > +      - Code
> > > > +    * - :rspan:`3` Camera sensor sub-device (CCS source sub-device)
> > > > +      - 0/0
> > > > +      - 640
> > > > +      - 480
> > > > +      - MEDIA_BUS_FMT_SGRBG10
> > > > +    * - 1/0
> > > > +      - 640
> > > > +      - 480
> > > > +      - **MEDIA_BUS_FMT_SGRBG10**
> > > > +    * - 2/0
> > > > +      - 640
> > > > +      - 2
> > > > +      - MEDIA_BUS_FMT_CCS_EMBEDDED_10
> > > > +    * - 1/1
> > > > +      - 640
> > > > +      - 2
> > > > +      - MEDIA_BUS_FMT_META_10
> > > > +    * - :rspan:`3` Bridge
> > > > +      - 0/0
> > > > +      - **640**
> > > > +      - **480**
> > > > +      - **MEDIA_BUS_FMT_SGRBG10**
> > > > +    * - 1/0
> > > > +      - 640
> > > > +      - 480
> > > > +      - MEDIA_BUS_FMT_SGRBG10
> > > > +    * - 0/1
> > > > +      - **640**
> > > > +      - **2**
> > > > +      - **MEDIA_BUS_FMT_META_10**
> > > > +    * - 1/1
> > > > +      - 640
> > > > +      - 2
> > > > +      - MEDIA_BUS_FMT_META_10
> > > > +    * - :rspan:`3` Receiver
> > > > +      - 0/0
> > > > +      - **640**
> > > > +      - **480**
> > > > +      - **MEDIA_BUS_FMT_SGRBG10**
> > > > +    * - 1/0
> > > > +      - 640
> > > > +      - 480
> > > > +      - MEDIA_BUS_FMT_SGRBG10
> > > > +    * - 0/1
> > > > +      - **640**
> > > > +      - **2**
> > > > +      - **MEDIA_BUS_FMT_META_10**
> > > > +    * - 2/0
> > > > +      - 640
> > > > +      - 2
> > > > +      - MEDIA_BUS_FMT_META_10
> > > > +
> > > > +The embedded data format does not need to be configured as the format is
> > > > +dictated by the pixel data format in this case.

-- 
Regards,

Sakari Ailus

Re: [PATCH v3 06/10] media: uapi: Add a macro to tell whether an mbus code is metadata

[Laurent Pinchart]

On Wed, Sep 06, 2023 at 11:33:30AM +0000, Sakari Ailus wrote:
> On Tue, Sep 05, 2023 at 08:06:04PM +0300, Laurent Pinchart wrote:
> > On Tue, Sep 05, 2023 at 10:37:45AM +0000, Sakari Ailus wrote:
> > > On Tue, Sep 05, 2023 at 12:47:21PM +0300, Tomi Valkeinen wrote:
> > > > On 08/08/2023 10:55, Sakari Ailus wrote:
> > > > > Add a macro to tell whether a given mbus code is metadata.
> > > > > 
> > > > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > > > ---
> > > > >   include/uapi/linux/media-bus-format.h | 3 +++
> > > > >   1 file changed, 3 insertions(+)
> > > > > 
> > > > > diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
> > > > > index 9ee031397372..2486b4178c5f 100644
> > > > > --- a/include/uapi/linux/media-bus-format.h
> > > > > +++ b/include/uapi/linux/media-bus-format.h
> > > > > @@ -182,4 +182,7 @@
> > > > >   #define MEDIA_BUS_FMT_META_20			0x8006
> > > > >   #define MEDIA_BUS_FMT_META_24			0x8007
> > > > > +#define MEDIA_BUS_FMT_IS_META(code)		\
> > > > > +	((code) & 0xf000 == 0x7000 || (code) & 0xf000 == 0x8000)
> > > > > +
> > > > >   #endif /* __LINUX_MEDIA_BUS_FORMAT_H */
> > > > 
> > > > mbus code seems to be u32, so the above won't work. Maybe:
> > > > 
> > > > (((code) & ~0xfffULL) == 0x7000 || ((code) & ~0xfffULL) == 0x8000)
> > > > 
> > > > Also, embedded formats with 0x9nnn codes are added later in the series.
> > > 
> > > Thanks, I'll address these for v4.
> > 
> > It would be nice to make this an inline function, to avoid evaluating
> > the code twice. I think you can move it to an internal kernel header, it
> > doesn't need to be exposed to userspace.
> 
> This was meant for user space consumption for setting the buffer type to
> the video node accordingly. It's certainly possible to do that without this
> macro, too.

I think userspace would be fine without it :-)

-- 
Regards,

Laurent Pinchart

Re: [PATCH v3 08/10] media: v4l: Support line-based metadata capture

[Sakari Ailus]

Hi Laurent,

On Wed, Sep 06, 2023 at 09:21:42AM +0200, Jacopo Mondi wrote:
> Hi Laurent
> 
> On Tue, Sep 05, 2023 at 08:15:33PM +0300, Laurent Pinchart wrote:
> > Hi Sakari,
> >
> > On Mon, Aug 14, 2023 at 11:02:40AM +0000, Sakari Ailus wrote:
> > > On Thu, Aug 10, 2023 at 05:24:14PM +0200, Jacopo Mondi wrote:
> > > > On Tue, Aug 08, 2023 at 10:55:36AM +0300, Sakari Ailus wrote:
> > > > > many camera sensors, among other devices, transmit embedded data and image
> >
> > s/many/Many/
> >
> > > > > data for each CSI-2 frame. This embedded data typically contains register
> > > > > configuration of the sensor that has been used to capture the image data
> > > > > of the same frame.
> > > > >
> > > > > The embedded data is received by the CSI-2 receiver and has the same
> > > > > properties as the image data, including that it is line based: it has
> > > > > width, height and bytesperline (stride).
> > > > >
> > > > > Add these fields to struct v4l2_meta_format and document them.
> > > > >
> > > > > Also add V4L2_FMT_FLAG_META_LINE_BASED to tell a given format is
> > > > > line-based i.e. these fields of struct v4l2_meta_format are valid for it.
> > > > >
> > > > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > > > ---
> > > > >  .../userspace-api/media/v4l/dev-meta.rst          | 15 +++++++++++++++
> > > > >  .../userspace-api/media/v4l/vidioc-enum-fmt.rst   |  7 +++++++
> > > > >  .../media/videodev2.h.rst.exceptions              |  1 +
> > > > >  drivers/media/v4l2-core/v4l2-ioctl.c              |  5 +++--
> > > > >  include/uapi/linux/videodev2.h                    | 10 ++++++++++
> > > > >  5 files changed, 36 insertions(+), 2 deletions(-)
> > > > >
> > > > > diff --git a/Documentation/userspace-api/media/v4l/dev-meta.rst b/Documentation/userspace-api/media/v4l/dev-meta.rst
> > > > > index 0e7e1ee1471a..4b24bae6e171 100644
> > > > > --- a/Documentation/userspace-api/media/v4l/dev-meta.rst
> > > > > +++ b/Documentation/userspace-api/media/v4l/dev-meta.rst
> > > > > @@ -65,3 +65,18 @@ to 0.
> > > > >        - ``buffersize``
> > > > >        - Maximum buffer size in bytes required for data. The value is set by the
> > > > >          driver.
> > > > > +    * - __u32
> > > > > +      - ``width``
> > > > > +      - Width of a line of metadata in samples. Valid when :c:type`v4l2_fmtdesc`
> > > > > +	flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
> > > > > +	:c:func:`VIDIOC_ENUM_FMT`.
> > > > > +    * - __u32
> > > > > +      - ``height``
> > > > > +      - Number of rows of metadata. Valid when :c:type`v4l2_fmtdesc` flag
> > > > > +	``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
> > > > > +	:c:func:`VIDIOC_ENUM_FMT`.
> > > > > +    * - __u32
> > > > > +      - ``bytesperline``
> > > > > +      - Offset in bytes between the beginning of two consecutive lines. Valid
> > > > > +	when :c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is
> > > > > +	set, otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
> > > > > diff --git a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> > > > > index 000c154b0f98..6d7664345a4e 100644
> > > > > --- a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> > > > > +++ b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> > > > > @@ -227,6 +227,13 @@ the ``mbus_code`` field is handled differently:
> > > > >  	The application can ask to configure the quantization of the capture
> > > > >  	device when calling the :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl with
> > > > >  	:ref:`V4L2_PIX_FMT_FLAG_SET_CSC <v4l2-pix-fmt-flag-set-csc>` set.
> > > > > +    * - ``V4L2_FMT_FLAG_META_LINE_BASED``
> > > > > +      - 0x0200
> > > > > +      - The metadata format is line-based. In this case the ``width``,
> > > > > +	``height`` and ``bytesperline`` fields of :c:type:`v4l2_meta_format` are
> > > > > +	valid. The buffer consists of ``height`` lines, each having ``width``
> > > > > +	bytes of data and offset between the beginning of each two consecutive
> > > >
> > > > Isn't ``width`` in samples ?
> > >
> > > Indeed, it's better to refer to samples for clarity. I'll fix for v4.
> >
> > How do you define a "sample" in this case ? I wonder if it wouldn't be
> > simpler for both userspace and kernel drivers if the width was specified
> > in bytes, including the padding bytes.
> 
> Wouldn't this make the image line length (expressed in 'samples')
> different than the embedded data line length ? Would this confuse
> userspace or is it fine ?

If padding is included to width, then the user needs to calculate how many
bytes of metadata there is, apart from the padding (which is redundant).
That value is provided to the user for this purpose --- just like for image
data.

> 
> >
> > We need an implementation here to test it out. The good news is that I'm
> > working on it, the bad news is that I'm also busy with other things.
> >
> > > I'll also add bytesperline is in bytes (and not in samples).
> >
> > Thanks for not messing up (more than needed) with my mental health by
> > not specifying bytesperline in something else than bytes :-)
> >
> 
> :)

I remember back in the Amiga days when memory was sometimes scarce, nibbles
were used. That works when bytes have an even number of bits.

-- 
Regards,

Sakari Ailus

Re: [PATCH v3 04/10] media: uapi: Add generic serial metadata mbus formats

[Sakari Ailus]

Hi Laurent, Tomi,

On Wed, Sep 06, 2023 at 02:31:17PM +0300, Laurent Pinchart wrote:
> Hi Tomi,
> 
> On Wed, Sep 06, 2023 at 11:28:37AM +0300, Tomi Valkeinen wrote:
> > On 05/09/2023 19:38, Laurent Pinchart wrote:
> > > On Thu, Aug 24, 2023 at 11:26:56AM +0300, Tomi Valkeinen wrote:
> > >> On 24/08/2023 10:24, Sakari Ailus wrote:
> > >>> On Wed, Aug 23, 2023 at 04:16:13PM +0300, Tomi Valkeinen wrote:
> > >>>> On 08/08/2023 10:55, Sakari Ailus wrote:
> > >>>>> Add generic serial metadata mbus formats. These formats describe data
> > >>>>> width and packing but not the content itself. The reason for specifying
> > >>>>> such formats is that the formats as such are fairly device specific but
> > >>>>> they are still handled by CSI-2 receiver drivers that should not be aware
> > > 
> > > Do we want these formats to be CSI-2-specific ?
> > >
> > >>>>> of device specific formats. What makes generic metadata formats possible
> > >>>>> is that these formats are parsed by software only, after capturing the
> > >>>>> data to system memory.
> > >>>>
> > >>>> If I'm not mistaken, the CSI-2 spec doesn't say much about embedded data,
> > >>>> except that it may exist. Afaics, in CSI-2, the embedded data is split into
> > >>>> "lines", although the amount of data can be different than in the video
> > >>>> lines.
> > >>>>
> > >>>> The CCS specs talks more about embedded data. Some of it is quite odd, like:
> > >>>> "The length of the embedded data line shall not exceed the length of the
> > >>>> image data line. The embedded data line should have the same length as the
> > >>>> image data line.". I think it means the embedded line can be shorter than
> > >>>> image line, but not longer.
> > >>>
> > >>> That's what it means, yes. The CCS also has means to obtain the actual
> > >>> length --- frame format descriptors.
> > >>>
> > >>>> CCS also says that an embedded line should use 0x07 as padding at the end of
> > >>>> the line, if there's less data than the image line.
> > >>>>
> > >>>> CCS also talks about how the embedded data would be packed, and in some
> > >>>> cases that packing would be the same as for pixel data.
> > >>>
> > >>> In fact the packing is the same as for pixel data: the CSI-2 does not
> > >>> really differentiate there.
> > >>
> > >> If I understand the CCS spec right, the packing is not the same as for
> > >> the pixel data. You can have RAW12 pixel data but 8-byte embedded data.
> > >> But if you meant that the different packing style options are the same
> > >> for pixel and embedded data, yes.
> > > 
> > > The idea is to allow CSI-2 receivers to unpack embedded data the same
> > > way as pixel data at the hardware level, and guarantee that no embedded
> > 
> > Yes, but my comment was to Sakari's "the packing is the same as for 
> > pixel data" comment. Afaics, there's nothing in CSI-2 or CCS that says 
> > the packing is the same for pixel data and embedded data.
> > 
> > In fact, the CSS says "By default, top-embedded data shall use the same 
> > data packing as the Visible pixel data with one embedded data byte per 
> > pixel", implying that it would not always be the case. It continues that 
> > for RAW16/RAW20/RAW24 pixels the embedded data could be sent with 
> > RAW8/RAW10/RAW12.
> 
> The spec if a bit ambiguous :-S "by default" implies that something else
> could be done, but it's not clear what the other options are.

The options for packing are detailed in the spec. For higher bit depths
there are options (as detailed by Tomi above).

> 
> > That was just for top-embedded data, and I couldn't right away see 
> > mention of other embedded data.
> 
> The specification also mentions PDAF data for instance.

The bottom PDAF data is vendor defined (as it's processed) so the vendor is
free to do whatever there. Note that the better way to support PDAF is via
embedded PDAF pixels, not via separate bottom embedded (PDAF) data. The
bottom embedded data could also be different kinds of statistics.

> 
> > > data significant byte gets split across multiple samples. When unpacking
> > > RAW10 pixel data, the CSI-2 receiver will take 5 bytes and output 4
> > > samples on a 10-bit wide bus. For embedded data, it can conceptually do
> > > that same, and output embedded data with one byte per sample, aligned on
> > > the MSB of the 10-bit bus. However, in practice, I think the CSI-2
> > > receiver will simply receive the bytes and send them to the DMA engine,
> > > without unpacking and then repacking.
> > 
> > At least TI CAL CSI-2 RX can extract the data from the CSI-2 stream 
> > using different unpacking for pixel and embedded data, and then 
> > (optionally) pack it in different ways when writing to memory.
> 
> Does this mean that the CAL could take RAW10 embedded data and drop the
> padding bytes when writing to memory ?
> 
> > > This can also be viewed as away to allow CSI-2 transmitters to serialize
> > > pixel data and embedded data the same way. For a RAW10 sensor, for
> > > instance, the embedded data would be generated on an internal 10-bit
> > > wide bus, aligned to the MSB, and would go through CSI-2 packing the
> > > same way that pixels do. That's nice, but it's internal to the sensor,
> > > so we don't need to expose it to userspace on that side.
> > > 
> > >>>> But I don't think these formats are generic. They're defined in CCS, so
> > >>>> shouldn't the format be, e.g., MEDIA_BUS_FMT_META_CCS_RAW_12 or such?
> > >>>
> > >>> The reason for having generic definitions is that we do not need receiver
> > >>> drivers to be aware of formats that are specific to another driver.
> > >>
> > >> Yes, I agree with that, and that's not my point here. But "generic"
> > >> doesn't mean the definitions are not for some particular bus or
> > >> standard, "generic" just means that it doesn't specify the content, only
> > >> the packing.
> > >>
> > >> My point is that these packings seem to be specific to CCS. While
> > >> non-CCS compliant sensors may use the same packing, the packing itself
> > >> is still a "CCS packing".
> > > 
> > > Agreed.
> > > 
> > >> So why not call them that? The 8-bit format is
> > >> fully generic, whereas the rest are CCS packings, as defined in the CCS
> > >> spec (the CCS spec also specifies the content, but here were only using
> > >> the CCS packing).
> > > 
> > > I think an additional question is whether we actually need multiple
> > > media bus codes here. Are there CSI-2 receivers, or other downstream
> > > blocks, that will need to be configured differently to receive embedded
> > > data from a CCS (in the sense of using CCS packing, not being fully
> > > CCS-compliant) RAW10 sensor compared to a RAW12 sensor ? Or will the
> > > CSI-2 receiver just send bytes to memory without caring ? In the latter
> > > case, it will be the application that will need to know how the data is
> > > packed to ignore one byte out of three for RAW12 and out of five for
> > > RAW10.
> > 
> > I would presume all receivers can operate in "just write what you 
> > receive" mode, in which case the packing doesn't matter as you say, but 
> > then again, I think it's safer to have mbus codes for the different 
> > packings.
> > 
> > >> Maybe they shouldn't be called "generic", but... umm... Content unaware
> > >> metadata formats... doesn't sound very good =).
> > >>
> > >> Also, I'm a bit confused about CSI-2 pixel and embedded data formats and
> > >> how we handle them.
> > >>
> > >> For MEDIA_BUS_FMT_SBGGR10_1X10, we define that the data contains 10 bits
> > >> per pixel, from bit 0 to 9. But CSI-2, for RAW10, actually sends it
> > >> differently, with the higher bits first, and the lowest bits in the
> > >> fourth byte. So that CSI-2 packing is implicit, kind of hidden here.
> > > 
> > > Usage of MEDIA_BUS_FMT_SBGGR10_1X10 for CSI-2 is a historical decision
> > > (or mistake, depending on how you look at it). It doesn't match what
> > > actually happens on the bus.
> > > 
> > >> Which is probably fine, as we're really only interested in the unpacked
> > >> data, not the CSI-2 packed. And when writing this data to memory, the
> > >> DMA engine can write it either as is, or unpack each pixel to a 16-bit
> > >> container.
> > >>
> > >> For MEDIA_BUS_FMT_META_10, we define it similarly to
> > >> MEDIA_BUS_FMT_SBGGR10_1X10, except the lowest 2 bits are marked as
> > >> padding bits. And, I think, MEDIA_BUS_FMT_META_10 implies RAW10 CSI-2
> > >> format.
> > > 
> > > That's how I understand it too
> > > 
> > >> However, when writing the data to memory, we don't want either
> > >> of the modes used in the above pixel data case, but rather we want to
> > >> write the data as it is in the CSI-2 bus. So, the DMA engine can either
> > >> reverse the RAW10 unpacking to get the wanted output format, or
> > >> alternatively the CSI-2 receiver could instead use RAW8 mode to avoid
> > >> any unpacking.
> > > 
> > > I don't think the DMA engine would "reverse the RAW10 packing" here. It
> > > will write bytes as they arrive (or more likely, a set of bytes at a
> > > time to operate on bursts). The other option would be for the DMA engine
> > > to drop the padding bytes, but I don't know of any CSI-2 receiver that
> > > can do that.
> > 
> > Well, for pixel data, TI CAL has the following options when writing to 
> > memory for, e.g. RAW12:
> > - RAW12 MIPI (a separate byte for lsbs)
> > - RAW12 Linear (12 consecutive bits per pixel)
> > - RAW16 (12 consecutive bits per pixel in a 16bit container)
> > 
> > Of these, only the RAW12 MIPI makes sense for embedded data (actually 
> > the whole processing block can be and should be skipped for embedded 
> > data, so it's more like CAL is receiving RAW8 and writing out RAW8).
> > 
> > Or, if it would be possible, skipping the padding bytes. Which looks 
> > like it is almost possible with TI CAL, but not quite: it can write with 
> > RAW8, but afaics that always takes the lowest bits of the 12-bit pixel, 
> > but here we'd like it to take the highest bits.
> 
> That's a weird one, even for image data, dropping the MSBs doesn't make
> much sense, while writing the RAW8 MSB when receiving RAW10 or RAW12
> would be useful.

I'd suppose it would drop the four most significant bits if it was a
parallel receiver. Does CAL have 8-bit input support? That should just work
in this case.

-- 
Regards,

Sakari Ailus

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

[Laurent Pinchart]

CC'ing Naush and David

On Wed, Sep 06, 2023 at 11:36:45AM +0000, Sakari Ailus wrote:
> On Tue, Sep 05, 2023 at 07:47:20PM +0300, Laurent Pinchart wrote:
> > On Fri, Aug 11, 2023 at 09:11:39AM +0000, Sakari Ailus wrote:
> > > On Fri, Aug 11, 2023 at 08:31:16AM +0200, Jacopo Mondi wrote:
> > > > > > +V4L2_META_FMT_GENERIC_CSI2_10
> > > > > > +-----------------------------
> > > > > > +
> > > > > > +V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
> > > > > > +for each 8 bits of data. Every four bytes of metadata is followed by a single
> > > > > > +byte of padding. The way the data is stored follows the CSI-2 specification.
> > > > > > +
> > > > > > +This format is also used on CSI-2 on 20 bits per sample format that packs two
> > > > > > +bytes of metadata into one sample.
> > > > > > +
> > > > > > +This format is little endian.
> > > > > > +
> > > > > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
> > > > > > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > > > >
> > > > > I think you should document whether the padding is always 0 or can be any value.
> > > > > Perhaps 'X' is a better 'name' for the padding byte in the latter case.
> > > > 
> > > > Did I get this right that this format is supposed to work as the RAW10
> > > > CSI-2 packed image format, where 4 bytes contain the higher 8 bits of
> > > > the 10 bits sample and the 5th byte every 4 contains the lower 2 bits of
> > > > the previous 4 sample ?
> > > > 
> > > > If that's the case, is 'padding' the correct term here ?
> > > 
> > > What else would you call it? It'll be zeros that exist just due to the bit
> > > depth used and as such not interesting at all.
> > 
> > It's actually not 0, CCS requires the padding bytes to be 0x55.
> > 
> > I wonder if the conformance test suite tests the contents of the padding
> > bytes.
> 
> I don't know. I could add the value is unspecified but as it has not been
> specified, there's no change in meaning (just size).

I started writing that I don't see how it could help applications to
know that the padding byte is 0x55, but the SMIA++ embedded data parser
in libcamera actually checks for it, and considers the embedded data to
be erroneous if it has a different value.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v3 01/10] media: Documentation: Align numbered list

[Sakari Ailus]

Hi Laurent,

On Tue, Sep 05, 2023 at 04:06:40PM +0300, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.
> 
> On Tue, Aug 08, 2023 at 10:55:29AM +0300, Sakari Ailus wrote:
> > Align lines for numbered list so that Sphinx produces an uniform output
> > for all list entries.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  .../userspace-api/media/v4l/dev-subdev.rst    | 27 +++++++++----------
> >  1 file changed, 13 insertions(+), 14 deletions(-)
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > index a4f1df7093e8..b023918177b5 100644
> > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > @@ -579,20 +579,19 @@ is started.
> >  
> >  There are three steps in configuring the streams:
> >  
> > -1) Set up links. Connect the pads between sub-devices using the :ref:`Media
> > -Controller API <media_controller>`
> > +1) Set up links. Connect the pads between sub-devices using the
> > +   :ref:`Media Controller API <media_controller>`
> 
> While at it, could we standardize on 1. vs. 1) ? Both are used in the
> same file. After this patch they both render to "1.", but it would be
> nice to be consistent in the source too.

I'll check the rest of the file. The number followed by a trailing dot is a
numbered list in ReST, we should align with that.

> 
> >  
> > -2) Streams. Streams are declared and their routing is configured by
> > -setting the routing table for the sub-device using
> > -:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that
> > -setting the routing table will reset formats and selections in the
> > -sub-device to default values.
> > +2) Streams. Streams are declared and their routing is configured by setting the
> > +   routing table for the sub-device using :ref:`VIDIOC_SUBDEV_S_ROUTING
> > +   <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that setting the routing table will
> > +   reset formats and selections in the sub-device to default values.
> >  
> > -3) Configure formats and selections. Formats and selections of each stream
> > -are configured separately as documented for plain sub-devices in
> > -:ref:`format-propagation`. The stream ID is set to the same stream ID
> > -associated with either sink or source pads of routes configured using the
> > -:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl.
> > +3) Configure formats and selections. Formats and selections of each stream are
> > +   configured separately as documented for plain sub-devices in
> > +   :ref:`format-propagation`. The stream ID is set to the same stream ID
> > +   associated with either sink or source pads of routes configured using the
> > +   :ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl.
> >  
> >  Multiplexed streams setup example
> >  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > @@ -619,8 +618,8 @@ modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
> >  To configure this pipeline, the userspace must take the following steps:
> >  
> >  1) Set up media links between entities: connect the sensors to the bridge,
> > -bridge to the receiver, and the receiver to the DMA engines. This step does
> > -not differ from normal non-multiplexed media controller setup.
> > +   bridge to the receiver, and the receiver to the DMA engines. This step does
> > +   not differ from normal non-multiplexed media controller setup.
> >  
> >  2) Configure routing
> >  
> 
> 3) below contains two paragraphs that are not indented.

From the documentation output this does not matter. Or was your comment
about aligning the formatting in the source? (I prefer it as-is but no
strong opinion about it either. Just slightly easier to edit that way. :-))

-- 
Regards,

Sakari Ailus

Re: [PATCH v3 04/10] media: uapi: Add generic serial metadata mbus formats

[Laurent Pinchart]

Hi Sakari,

On Wed, Sep 06, 2023 at 12:34:10PM +0000, Sakari Ailus wrote:
> On Wed, Sep 06, 2023 at 02:31:17PM +0300, Laurent Pinchart wrote:
> > On Wed, Sep 06, 2023 at 11:28:37AM +0300, Tomi Valkeinen wrote:
> > > On 05/09/2023 19:38, Laurent Pinchart wrote:
> > > > On Thu, Aug 24, 2023 at 11:26:56AM +0300, Tomi Valkeinen wrote:
> > > >> On 24/08/2023 10:24, Sakari Ailus wrote:
> > > >>> On Wed, Aug 23, 2023 at 04:16:13PM +0300, Tomi Valkeinen wrote:
> > > >>>> On 08/08/2023 10:55, Sakari Ailus wrote:
> > > >>>>> Add generic serial metadata mbus formats. These formats describe data
> > > >>>>> width and packing but not the content itself. The reason for specifying
> > > >>>>> such formats is that the formats as such are fairly device specific but
> > > >>>>> they are still handled by CSI-2 receiver drivers that should not be aware
> > > > 
> > > > Do we want these formats to be CSI-2-specific ?
> > > >
> > > >>>>> of device specific formats. What makes generic metadata formats possible
> > > >>>>> is that these formats are parsed by software only, after capturing the
> > > >>>>> data to system memory.
> > > >>>>
> > > >>>> If I'm not mistaken, the CSI-2 spec doesn't say much about embedded data,
> > > >>>> except that it may exist. Afaics, in CSI-2, the embedded data is split into
> > > >>>> "lines", although the amount of data can be different than in the video
> > > >>>> lines.
> > > >>>>
> > > >>>> The CCS specs talks more about embedded data. Some of it is quite odd, like:
> > > >>>> "The length of the embedded data line shall not exceed the length of the
> > > >>>> image data line. The embedded data line should have the same length as the
> > > >>>> image data line.". I think it means the embedded line can be shorter than
> > > >>>> image line, but not longer.
> > > >>>
> > > >>> That's what it means, yes. The CCS also has means to obtain the actual
> > > >>> length --- frame format descriptors.
> > > >>>
> > > >>>> CCS also says that an embedded line should use 0x07 as padding at the end of
> > > >>>> the line, if there's less data than the image line.
> > > >>>>
> > > >>>> CCS also talks about how the embedded data would be packed, and in some
> > > >>>> cases that packing would be the same as for pixel data.
> > > >>>
> > > >>> In fact the packing is the same as for pixel data: the CSI-2 does not
> > > >>> really differentiate there.
> > > >>
> > > >> If I understand the CCS spec right, the packing is not the same as for
> > > >> the pixel data. You can have RAW12 pixel data but 8-byte embedded data.
> > > >> But if you meant that the different packing style options are the same
> > > >> for pixel and embedded data, yes.
> > > > 
> > > > The idea is to allow CSI-2 receivers to unpack embedded data the same
> > > > way as pixel data at the hardware level, and guarantee that no embedded
> > > 
> > > Yes, but my comment was to Sakari's "the packing is the same as for 
> > > pixel data" comment. Afaics, there's nothing in CSI-2 or CCS that says 
> > > the packing is the same for pixel data and embedded data.
> > > 
> > > In fact, the CSS says "By default, top-embedded data shall use the same 
> > > data packing as the Visible pixel data with one embedded data byte per 
> > > pixel", implying that it would not always be the case. It continues that 
> > > for RAW16/RAW20/RAW24 pixels the embedded data could be sent with 
> > > RAW8/RAW10/RAW12.
> > 
> > The spec if a bit ambiguous :-S "by default" implies that something else
> > could be done, but it's not clear what the other options are.
> 
> The options for packing are detailed in the spec. For higher bit depths
> there are options (as detailed by Tomi above).

The "by default" wording could be construed as allowing any packing. I
think the intent was to only allow an alternative packing for RAW16 and
higher, but it's not very clear.

> > > That was just for top-embedded data, and I couldn't right away see 
> > > mention of other embedded data.
> > 
> > The specification also mentions PDAF data for instance.
> 
> The bottom PDAF data is vendor defined (as it's processed) so the vendor is
> free to do whatever there. Note that the better way to support PDAF is via
> embedded PDAF pixels, not via separate bottom embedded (PDAF) data. The
> bottom embedded data could also be different kinds of statistics.

Lots of ISPs don't support hardware processing of embedded PDAF pixels,
so sensor-side processing is often nice to have. I wouldn't consider the
embedded PDAF lines to be always worse than embedded PDAF pixels.

> > > > data significant byte gets split across multiple samples. When unpacking
> > > > RAW10 pixel data, the CSI-2 receiver will take 5 bytes and output 4
> > > > samples on a 10-bit wide bus. For embedded data, it can conceptually do
> > > > that same, and output embedded data with one byte per sample, aligned on
> > > > the MSB of the 10-bit bus. However, in practice, I think the CSI-2
> > > > receiver will simply receive the bytes and send them to the DMA engine,
> > > > without unpacking and then repacking.
> > > 
> > > At least TI CAL CSI-2 RX can extract the data from the CSI-2 stream 
> > > using different unpacking for pixel and embedded data, and then 
> > > (optionally) pack it in different ways when writing to memory.
> > 
> > Does this mean that the CAL could take RAW10 embedded data and drop the
> > padding bytes when writing to memory ?
> > 
> > > > This can also be viewed as away to allow CSI-2 transmitters to serialize
> > > > pixel data and embedded data the same way. For a RAW10 sensor, for
> > > > instance, the embedded data would be generated on an internal 10-bit
> > > > wide bus, aligned to the MSB, and would go through CSI-2 packing the
> > > > same way that pixels do. That's nice, but it's internal to the sensor,
> > > > so we don't need to expose it to userspace on that side.
> > > > 
> > > >>>> But I don't think these formats are generic. They're defined in CCS, so
> > > >>>> shouldn't the format be, e.g., MEDIA_BUS_FMT_META_CCS_RAW_12 or such?
> > > >>>
> > > >>> The reason for having generic definitions is that we do not need receiver
> > > >>> drivers to be aware of formats that are specific to another driver.
> > > >>
> > > >> Yes, I agree with that, and that's not my point here. But "generic"
> > > >> doesn't mean the definitions are not for some particular bus or
> > > >> standard, "generic" just means that it doesn't specify the content, only
> > > >> the packing.
> > > >>
> > > >> My point is that these packings seem to be specific to CCS. While
> > > >> non-CCS compliant sensors may use the same packing, the packing itself
> > > >> is still a "CCS packing".
> > > > 
> > > > Agreed.
> > > > 
> > > >> So why not call them that? The 8-bit format is
> > > >> fully generic, whereas the rest are CCS packings, as defined in the CCS
> > > >> spec (the CCS spec also specifies the content, but here were only using
> > > >> the CCS packing).
> > > > 
> > > > I think an additional question is whether we actually need multiple
> > > > media bus codes here. Are there CSI-2 receivers, or other downstream
> > > > blocks, that will need to be configured differently to receive embedded
> > > > data from a CCS (in the sense of using CCS packing, not being fully
> > > > CCS-compliant) RAW10 sensor compared to a RAW12 sensor ? Or will the
> > > > CSI-2 receiver just send bytes to memory without caring ? In the latter
> > > > case, it will be the application that will need to know how the data is
> > > > packed to ignore one byte out of three for RAW12 and out of five for
> > > > RAW10.
> > > 
> > > I would presume all receivers can operate in "just write what you 
> > > receive" mode, in which case the packing doesn't matter as you say, but 
> > > then again, I think it's safer to have mbus codes for the different 
> > > packings.
> > > 
> > > >> Maybe they shouldn't be called "generic", but... umm... Content unaware
> > > >> metadata formats... doesn't sound very good =).
> > > >>
> > > >> Also, I'm a bit confused about CSI-2 pixel and embedded data formats and
> > > >> how we handle them.
> > > >>
> > > >> For MEDIA_BUS_FMT_SBGGR10_1X10, we define that the data contains 10 bits
> > > >> per pixel, from bit 0 to 9. But CSI-2, for RAW10, actually sends it
> > > >> differently, with the higher bits first, and the lowest bits in the
> > > >> fourth byte. So that CSI-2 packing is implicit, kind of hidden here.
> > > > 
> > > > Usage of MEDIA_BUS_FMT_SBGGR10_1X10 for CSI-2 is a historical decision
> > > > (or mistake, depending on how you look at it). It doesn't match what
> > > > actually happens on the bus.
> > > > 
> > > >> Which is probably fine, as we're really only interested in the unpacked
> > > >> data, not the CSI-2 packed. And when writing this data to memory, the
> > > >> DMA engine can write it either as is, or unpack each pixel to a 16-bit
> > > >> container.
> > > >>
> > > >> For MEDIA_BUS_FMT_META_10, we define it similarly to
> > > >> MEDIA_BUS_FMT_SBGGR10_1X10, except the lowest 2 bits are marked as
> > > >> padding bits. And, I think, MEDIA_BUS_FMT_META_10 implies RAW10 CSI-2
> > > >> format.
> > > > 
> > > > That's how I understand it too
> > > > 
> > > >> However, when writing the data to memory, we don't want either
> > > >> of the modes used in the above pixel data case, but rather we want to
> > > >> write the data as it is in the CSI-2 bus. So, the DMA engine can either
> > > >> reverse the RAW10 unpacking to get the wanted output format, or
> > > >> alternatively the CSI-2 receiver could instead use RAW8 mode to avoid
> > > >> any unpacking.
> > > > 
> > > > I don't think the DMA engine would "reverse the RAW10 packing" here. It
> > > > will write bytes as they arrive (or more likely, a set of bytes at a
> > > > time to operate on bursts). The other option would be for the DMA engine
> > > > to drop the padding bytes, but I don't know of any CSI-2 receiver that
> > > > can do that.
> > > 
> > > Well, for pixel data, TI CAL has the following options when writing to 
> > > memory for, e.g. RAW12:
> > > - RAW12 MIPI (a separate byte for lsbs)
> > > - RAW12 Linear (12 consecutive bits per pixel)
> > > - RAW16 (12 consecutive bits per pixel in a 16bit container)
> > > 
> > > Of these, only the RAW12 MIPI makes sense for embedded data (actually 
> > > the whole processing block can be and should be skipped for embedded 
> > > data, so it's more like CAL is receiving RAW8 and writing out RAW8).
> > > 
> > > Or, if it would be possible, skipping the padding bytes. Which looks 
> > > like it is almost possible with TI CAL, but not quite: it can write with 
> > > RAW8, but afaics that always takes the lowest bits of the 12-bit pixel, 
> > > but here we'd like it to take the highest bits.
> > 
> > That's a weird one, even for image data, dropping the MSBs doesn't make
> > much sense, while writing the RAW8 MSB when receiving RAW10 or RAW12
> > would be useful.
> 
> I'd suppose it would drop the four most significant bits if it was a
> parallel receiver. Does CAL have 8-bit input support? That should just work
> in this case.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v3 01/10] media: Documentation: Align numbered list

[Laurent Pinchart]

On Wed, Sep 06, 2023 at 12:43:44PM +0000, Sakari Ailus wrote:
> On Tue, Sep 05, 2023 at 04:06:40PM +0300, Laurent Pinchart wrote:
> > On Tue, Aug 08, 2023 at 10:55:29AM +0300, Sakari Ailus wrote:
> > > Align lines for numbered list so that Sphinx produces an uniform output
> > > for all list entries.
> > > 
> > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > ---
> > >  .../userspace-api/media/v4l/dev-subdev.rst    | 27 +++++++++----------
> > >  1 file changed, 13 insertions(+), 14 deletions(-)
> > > 
> > > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > index a4f1df7093e8..b023918177b5 100644
> > > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > @@ -579,20 +579,19 @@ is started.
> > >  
> > >  There are three steps in configuring the streams:
> > >  
> > > -1) Set up links. Connect the pads between sub-devices using the :ref:`Media
> > > -Controller API <media_controller>`
> > > +1) Set up links. Connect the pads between sub-devices using the
> > > +   :ref:`Media Controller API <media_controller>`
> > 
> > While at it, could we standardize on 1. vs. 1) ? Both are used in the
> > same file. After this patch they both render to "1.", but it would be
> > nice to be consistent in the source too.
> 
> I'll check the rest of the file. The number followed by a trailing dot is a
> numbered list in ReST, we should align with that.
> 
> > 
> > >  
> > > -2) Streams. Streams are declared and their routing is configured by
> > > -setting the routing table for the sub-device using
> > > -:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that
> > > -setting the routing table will reset formats and selections in the
> > > -sub-device to default values.
> > > +2) Streams. Streams are declared and their routing is configured by setting the
> > > +   routing table for the sub-device using :ref:`VIDIOC_SUBDEV_S_ROUTING
> > > +   <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that setting the routing table will
> > > +   reset formats and selections in the sub-device to default values.
> > >  
> > > -3) Configure formats and selections. Formats and selections of each stream
> > > -are configured separately as documented for plain sub-devices in
> > > -:ref:`format-propagation`. The stream ID is set to the same stream ID
> > > -associated with either sink or source pads of routes configured using the
> > > -:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl.
> > > +3) Configure formats and selections. Formats and selections of each stream are
> > > +   configured separately as documented for plain sub-devices in
> > > +   :ref:`format-propagation`. The stream ID is set to the same stream ID
> > > +   associated with either sink or source pads of routes configured using the
> > > +   :ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl.
> > >  
> > >  Multiplexed streams setup example
> > >  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > @@ -619,8 +618,8 @@ modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
> > >  To configure this pipeline, the userspace must take the following steps:
> > >  
> > >  1) Set up media links between entities: connect the sensors to the bridge,
> > > -bridge to the receiver, and the receiver to the DMA engines. This step does
> > > -not differ from normal non-multiplexed media controller setup.
> > > +   bridge to the receiver, and the receiver to the DMA engines. This step does
> > > +   not differ from normal non-multiplexed media controller setup.
> > >  
> > >  2) Configure routing
> > >  
> > 
> > 3) below contains two paragraphs that are not indented.
> 
> From the documentation output this does not matter. Or was your comment
> about aligning the formatting in the source? (I prefer it as-is but no
> strong opinion about it either. Just slightly easier to edit that way. :-))

I like consistency, both in the source and the output :-)

-- 
Regards,

Laurent Pinchart

Re: [PATCH v3 09/10] media: Add media bus codes for MIPI CCS embedded data

[Sakari Ailus]

Hi Laurent,

On Tue, Sep 05, 2023 at 08:25:35PM +0300, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.
> 
> On Tue, Aug 08, 2023 at 10:55:37AM +0300, Sakari Ailus wrote:
> > Add new MIPI CCS embedded data media bus formats.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  .../media/v4l/subdev-formats.rst              | 32 +++++++++++++++++++
> >  include/uapi/linux/media-bus-format.h         | 10 +++++-
> >  2 files changed, 41 insertions(+), 1 deletion(-)
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > index c615da08502d..5d5407738af9 100644
> > --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > @@ -8491,3 +8491,35 @@ and finally the bit number in subscript. "p" indicates a padding bit.
> >        - p
> >        - p
> >        - p
> > +
> > +MIPI CCS Embedded Data Formats
> > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > +
> > +`MIPI CCS <https://www.mipi.org/specifications/camera-command-set>`_ defines an
> 
> s/an$/a/

Yes. I think I had "embedded" there in the past...

> 
> > +metadata format for sensor embedded data, which is used to store the register
> > +configuration used for capturing a given frame. The format is defined in the CCS
> > +specification.
> 
> Strictly speaking, the MIPI CCS embedded data format specifies not just
> the data packing (insertion of padding bytes) and the data encoding (the
> data format byte and the tag codes), but also the register addresses and
> values that are reported in the embedded data. Do you envision the media
> bus formats defined here as being applicable to sensors that use the
> same packing and encoding as CCS, but different registers, or only to
> fully compliant CCS sensors ?

There are sensors that aren't fully compatible with CCS (including those
compatible with SMIA and SMIA++) but I wouldn't expect the format to be
used by devices that are entirely incompatible with CCS.

-- 
Regards,

Sakari Ailus

Re: [PATCH v3 06/10] media: uapi: Add a macro to tell whether an mbus code is metadata

[Sakari Ailus]

On Wed, Sep 06, 2023 at 03:23:08PM +0300, Laurent Pinchart wrote:
> On Wed, Sep 06, 2023 at 11:33:30AM +0000, Sakari Ailus wrote:
> > On Tue, Sep 05, 2023 at 08:06:04PM +0300, Laurent Pinchart wrote:
> > > On Tue, Sep 05, 2023 at 10:37:45AM +0000, Sakari Ailus wrote:
> > > > On Tue, Sep 05, 2023 at 12:47:21PM +0300, Tomi Valkeinen wrote:
> > > > > On 08/08/2023 10:55, Sakari Ailus wrote:
> > > > > > Add a macro to tell whether a given mbus code is metadata.
> > > > > > 
> > > > > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > > > > ---
> > > > > >   include/uapi/linux/media-bus-format.h | 3 +++
> > > > > >   1 file changed, 3 insertions(+)
> > > > > > 
> > > > > > diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
> > > > > > index 9ee031397372..2486b4178c5f 100644
> > > > > > --- a/include/uapi/linux/media-bus-format.h
> > > > > > +++ b/include/uapi/linux/media-bus-format.h
> > > > > > @@ -182,4 +182,7 @@
> > > > > >   #define MEDIA_BUS_FMT_META_20			0x8006
> > > > > >   #define MEDIA_BUS_FMT_META_24			0x8007
> > > > > > +#define MEDIA_BUS_FMT_IS_META(code)		\
> > > > > > +	((code) & 0xf000 == 0x7000 || (code) & 0xf000 == 0x8000)
> > > > > > +
> > > > > >   #endif /* __LINUX_MEDIA_BUS_FORMAT_H */
> > > > > 
> > > > > mbus code seems to be u32, so the above won't work. Maybe:
> > > > > 
> > > > > (((code) & ~0xfffULL) == 0x7000 || ((code) & ~0xfffULL) == 0x8000)
> > > > > 
> > > > > Also, embedded formats with 0x9nnn codes are added later in the series.
> > > > 
> > > > Thanks, I'll address these for v4.
> > > 
> > > It would be nice to make this an inline function, to avoid evaluating
> > > the code twice. I think you can move it to an internal kernel header, it
> > > doesn't need to be exposed to userspace.
> > 
> > This was meant for user space consumption for setting the buffer type to
> > the video node accordingly. It's certainly possible to do that without this
> > macro, too.
> 
> I think userspace would be fine without it :-)

I'll see if this would seem useful in yavta support before dropping it: the
buffer type needs to be specified correctly and knowing the format is a
metadata format maybe helpful. Let's see.

-- 
Sakari Ailus

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

[Laurent Pinchart]

On Wed, Sep 06, 2023 at 11:56:47AM +0000, Sakari Ailus wrote:
> Hi Laurent,
> 
> On Tue, Sep 05, 2023 at 07:55:58PM +0300, Laurent Pinchart wrote:
> > On Tue, Aug 08, 2023 at 10:55:35AM +0300, Sakari Ailus wrote:
> > > Generic 8-bit metadata formats define the in-memory data layout but not
> > > the format of the data itself. The reasoning for having such formats is to
> > > allow CSI-2 receiver drivers to receive and DMA drivers to write the data
> > > to memory without knowing a large number of device specific formats.
> > > 
> > > These formats may be used only in conjunction of a Media controller
> > > pipeline where the internal pad of the source sub-device defines the
> > > specific format of the data (using an mbus code).
> > > 
> > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > ---
> > >  .../userspace-api/media/v4l/meta-formats.rst  |   1 +
> > >  .../media/v4l/metafmt-generic.rst             | 331 ++++++++++++++++++
> > >  drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
> > >  include/uapi/linux/videodev2.h                |   9 +
> > >  4 files changed, 349 insertions(+)
> > >  create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > > 
> > > diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
> > > index 0bb61fc5bc00..919f595576b9 100644
> > > --- a/Documentation/userspace-api/media/v4l/meta-formats.rst
> > > +++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
> > > @@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only.
> > >      metafmt-vsp1-hgo
> > >      metafmt-vsp1-hgt
> > >      metafmt-vivid
> > > +    metafmt-generic
> > > diff --git a/Documentation/userspace-api/media/v4l/metafmt-generic.rst b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > > new file mode 100644
> > > index 000000000000..a27bfc721edf
> > > --- /dev/null
> > > +++ b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > > @@ -0,0 +1,331 @@
> > > +.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
> > > +
> > > +**************************************************************************************************************************************************************************************************************************************************************************************************************************
> > > +V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O'), V4L2_META_FMT_GENERIC_CSI2_2_24 ('MC2O')
> > > +**************************************************************************************************************************************************************************************************************************************************************************************************************************
> > > +
> > > +
> > > +Generic line-based metadata formats
> > > +
> > > +
> > > +Description
> > > +===========
> > > +
> > > +These generic line-based metadata formats define the memory layout of the data
> > > +without defining the format or meaning of the metadata itself. These formats may
> > > +only be used with a Media controller pipeline where the more specific format is
> > > +defined in an :ref:`internal source pad <MEDIA-PAD-FL-INTERNAL>` of the source
> > > +sub-device. See also :ref:`source routes <v4l2-subdev-source-routes>`.
> > > +
> > > +.. _v4l2-meta-fmt-generic-8:
> > > +
> > > +V4L2_META_FMT_GENERIC_8
> > > +-----------------------
> > > +
> > > +The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format.
> > > +
> > > +This format is also used on CSI-2 on both 8 bits per sample as well as on
> > 
> > s/also on/by/
> > 
> > I would also mention "MIPI CCS" instead of "CSI-2".
> 
> If CCS were to be mentioned here, then all uses of this format should be
> included as well.
> 
> > > +16 bits per sample when two bytes of metadata are packed into one sample.
> > 
> > "bits per sample" is very ill-defined for metadata, as there's no
> > sample. I would write "for both the RAW8 packing and the 2 bytes RAW16
> > packing" or something similar.
> > 
> > Similar comments for below.
> 
> From CSI-2 bus point of view there's no difference between pixel and
> embedded data when it comes to encoding that data. "Sample" is the next
> best term beyond "pixel", as the bus can carry samples that may or may not
> be pixel data. But I'm fine with changing the wording if you think it makes
> it more understandable.

The CSI-2 specification doesn't seem to define any "sample" concept.

> > > +
> > > +**Byte Order Of V4L2_META_FMT_GENERIC_8.**
> > > +Each cell is one byte. "M" denotes a byte of metadata.
> > > +
> > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|
> > > +
> > > +.. flat-table::
> > > +    :header-rows:  0
> > > +    :stub-columns: 0
> > > +    :widths: 12 8 8 8 8
> > > +
> > > +    * - start + 0:
> > > +      - M\ :sub:`00`
> > > +      - M\ :sub:`10`
> > > +      - M\ :sub:`20`
> > > +      - M\ :sub:`30`
> > > +    * - start + 4:
> > > +      - M\ :sub:`01`
> > > +      - M\ :sub:`11`
> > > +      - M\ :sub:`21`
> > > +      - M\ :sub:`31`
> > > +
> > > +.. _v4l2-meta-fmt-generic-csi2-10:
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_10
> > > +-----------------------------
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
> > > +for each 8 bits of data. Every four bytes of metadata is followed by a single
> > > +byte of padding.
> > 
> > It sounds really weird to write that this format writes 10 bits for each
> > 8 bits of data, when essentially it adds a packing byte every four
> > bytes.
> 
> That's how the hardware has been implemented and probably there is a
> hardware implementation related reason for this.

I can imagine the specification has been designed to make it possible to
push embedded data and pixel data through the same serialization
hardware, but I can't tell if that's how hardware has been implemented.
Still, from an application point of view, when documenting pixel
formats, it sounds confusing. Can't we instead say that there's a 0x55
padding after every four bytes of data ?

> > > The way the data is stored follows the CSI-2 specification.
> > 
> > Again, it's CCS, not CSI-2.
> 
> The packing comes from CSI-2 but CCS adds padding. I'll add this
> information.
> 
> > > +
> > > +This format is also used on CSI-2 on 20 bits per sample format that packs two
> > > +bytes of metadata into one sample.
> > > +
> > > +This format is little endian.
> > > +
> > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
> > > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > > +
> > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
> > > +
> > > +.. flat-table::
> > > +    :header-rows:  0
> > > +    :stub-columns: 0
> > > +    :widths: 12 8 8 8 8 8
> > > +
> > > +    * - start + 0:
> > > +      - M\ :sub:`00`
> > > +      - M\ :sub:`10`
> > > +      - M\ :sub:`20`
> > > +      - M\ :sub:`30`
> > > +      - p
> > > +    * - start + 5:
> > > +      - M\ :sub:`01`
> > > +      - M\ :sub:`11`
> > > +      - M\ :sub:`21`
> > > +      - M\ :sub:`31`
> > > +      - p
> > > +
> > > +.. _v4l2-meta-fmt-generic-csi2-12:
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_12
> > > +-----------------------------
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_12 contains packed 8-bit generic metadata, 12 bits
> > > +for each 8 bits of data. Every four bytes of metadata is followed by two bytes
> > > +of padding. The way the data is stored follows the CSI-2 specification.
> > > +
> > > +This format is little endian.
> > > +
> > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_12.**
> > > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > > +
> > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> > > +
> > > +.. flat-table::
> > > +    :header-rows:  0
> > > +    :stub-columns: 0
> > > +    :widths: 12 8 8 8 8 8 8
> > > +
> > > +    * - start + 0:
> > > +      - M\ :sub:`00`
> > > +      - M\ :sub:`10`
> > > +      - M\ :sub:`20`
> > > +      - M\ :sub:`30`
> > > +      - p
> > > +      - p
> > > +    * - start + 6:
> > > +      - M\ :sub:`01`
> > > +      - M\ :sub:`11`
> > > +      - M\ :sub:`21`
> > > +      - M\ :sub:`31`
> > > +      - p
> > > +      - p
> > > +
> > > +.. _v4l2-meta-fmt-generic-csi2-14:
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_14
> > > +-----------------------------
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_14 contains packed 8-bit generic metadata, 14 bits
> > > +for each 8 bits of data. Every four bytes of metadata is followed by three
> > > +bytes of padding. The way the data is stored follows the CSI-2 specification.
> > > +
> > > +This format is little endian.
> > > +
> > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_14.**
> > > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > > +
> > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{.8cm}|
> > > +
> > > +.. flat-table::
> > > +    :header-rows:  0
> > > +    :stub-columns: 0
> > > +    :widths: 12 8 8 8 8 8 8 8
> > > +
> > > +    * - start + 0:
> > > +      - M\ :sub:`00`
> > > +      - M\ :sub:`10`
> > > +      - M\ :sub:`20`
> > > +      - M\ :sub:`30`
> > > +      - p
> > > +      - p
> > > +      - p
> > > +    * - start + 7:
> > > +      - M\ :sub:`01`
> > > +      - M\ :sub:`11`
> > > +      - M\ :sub:`21`
> > > +      - M\ :sub:`31`
> > > +      - p
> > > +      - p
> > > +      - p
> > > +
> > > +.. _v4l2-meta-fmt-generic-csi2-16:
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_16
> > > +-----------------------------
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_16 contains packed 8-bit generic metadata, 16 bits
> > > +for each 8 bits of data. Every byte of metadata is followed by one byte of
> > > +padding. The way the data is stored follows the CSI-2 specification.
> > > +
> > > +This format is little endian.
> > > +
> > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_16.**
> > > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > > +
> > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|
> > > +
> > > +.. flat-table::
> > > +    :header-rows:  0
> > > +    :stub-columns: 0
> > > +    :widths: 12 8 8 8 8 8 8 8 8
> > > +
> > > +    * - start + 0:
> > > +      - M\ :sub:`00`
> > > +      - p
> > > +      - M\ :sub:`10`
> > > +      - p
> > > +      - M\ :sub:`20`
> > > +      - p
> > > +      - M\ :sub:`30`
> > > +      - p
> > > +    * - start + 8:
> > > +      - M\ :sub:`01`
> > > +      - p
> > > +      - M\ :sub:`11`
> > > +      - p
> > > +      - M\ :sub:`21`
> > > +      - p
> > > +      - M\ :sub:`31`
> > > +      - p
> > > +
> > > +.. _v4l2-meta-fmt-generic-csi2-20:
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_20
> > > +-----------------------------
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_20 contains packed 8-bit generic metadata, 20 bits
> > > +for each 8 bits of data. Every byte of metadata is followed by alternating one
> > > +and two bytes of padding. The way the data is stored follows the CSI-2
> > > +specification.
> > > +
> > > +This format is little endian.
> > > +
> > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_20.**
> > > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > > +
> > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> > > +
> > > +.. flat-table::
> > > +    :header-rows:  0
> > > +    :stub-columns: 0
> > > +    :widths: 12 8 8 8 8 8 8 8 8 8 8
> > > +
> > > +    * - start + 0:
> > > +      - M\ :sub:`00`
> > > +      - p
> > > +      - M\ :sub:`10`
> > > +      - p
> > > +      - p
> > > +      - M\ :sub:`20`
> > > +      - p
> > > +      - M\ :sub:`30`
> > > +      - p
> > > +      - p
> > > +    * - start + 10:
> > > +      - M\ :sub:`01`
> > > +      - p
> > > +      - M\ :sub:`11`
> > > +      - p
> > > +      - p
> > > +      - M\ :sub:`21`
> > > +      - p
> > > +      - M\ :sub:`31`
> > > +      - p
> > > +      - p
> > > +
> > > +.. _v4l2-meta-fmt-generic-csi2-24:
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_24
> > > +-----------------------------
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_24 contains packed 8-bit generic metadata, 24 bits
> > > +for each 8 bits of data. Every byte of metadata is followed by two bytes of
> > > +padding. The way the data is stored follows the CSI-2 specification.
> > > +
> > > +This format is little endian.
> > > +
> > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_24.**
> > > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > > +
> > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> > > +
> > > +.. flat-table::
> > > +    :header-rows:  0
> > > +    :stub-columns: 0
> > > +    :widths: 12 8 8 8 8 8 8 8 8 8 8 8 8
> > > +
> > > +    * - start + 0:
> > > +      - M\ :sub:`00`
> > > +      - p
> > > +      - p
> > > +      - M\ :sub:`10`
> > > +      - p
> > > +      - p
> > > +      - M\ :sub:`20`
> > > +      - p
> > > +      - p
> > > +      - M\ :sub:`30`
> > > +      - p
> > > +      - p
> > > +    * - start + 12:
> > > +      - M\ :sub:`01`
> > > +      - p
> > > +      - p
> > > +      - M\ :sub:`11`
> > > +      - p
> > > +      - p
> > > +      - M\ :sub:`21`
> > > +      - p
> > > +      - p
> > > +      - M\ :sub:`31`
> > > +      - p
> > > +      - p
> > > +
> > > +.. _v4l2-meta-fmt-generic-csi2-2-24:
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_2_24
> > > +-------------------------------
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_2_24 contains packed 8-bit generic metadata, 24 bits
> > > +for each two times 8 bits of data. Every two bytes of metadata are followed by
> > > +one byte of padding. The way the data is stored follows the CSI-2
> > > +specification.
> > > +
> > > +This format is little endian.
> > > +
> > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_2_24.**
> > > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > > +
> > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
> > > +
> > > +.. flat-table::
> > > +    :header-rows:  0
> > > +    :stub-columns: 0
> > > +    :widths: 12 8 8 8 8 8 8
> > > +
> > > +    * - start + 0:
> > > +      - M\ :sub:`00`
> > > +      - M\ :sub:`10`
> > > +      - p
> > > +      - M\ :sub:`20`
> > > +      - M\ :sub:`30`
> > > +      - p
> > > +    * - start + 6:
> > > +      - M\ :sub:`01`
> > > +      - M\ :sub:`11`
> > > +      - p
> > > +      - M\ :sub:`21`
> > > +      - M\ :sub:`31`
> > > +      - p
> > > +
> > > diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> > > index f4d9d6279094..fbbddc333a30 100644
> > > --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> > > +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> > > @@ -1452,6 +1452,14 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
> > >  	case V4L2_PIX_FMT_Y210:		descr = "10-bit YUYV Packed"; break;
> > >  	case V4L2_PIX_FMT_Y212:		descr = "12-bit YUYV Packed"; break;
> > >  	case V4L2_PIX_FMT_Y216:		descr = "16-bit YUYV Packed"; break;
> > > +	case V4L2_META_FMT_GENERIC_8:	descr = "8-bit Generic Metadata"; break;
> > > +	case V4L2_META_FMT_GENERIC_CSI2_10:	descr = "8b Generic Meta, 10b CSI-2"; break;
> > > +	case V4L2_META_FMT_GENERIC_CSI2_12:	descr = "8b Generic Meta, 12b CSI-2"; break;
> > > +	case V4L2_META_FMT_GENERIC_CSI2_14:	descr = "8b Generic Meta, 14b CSI-2"; break;
> > > +	case V4L2_META_FMT_GENERIC_CSI2_16:	descr = "8b Generic Meta, 16b CSI-2"; break;
> > > +	case V4L2_META_FMT_GENERIC_CSI2_20:	descr = "8b Generic Meta, 20b CSI-2"; break;
> > > +	case V4L2_META_FMT_GENERIC_CSI2_24:	descr = "8b Generic Meta, 24b CSI-2"; break;
> > > +	case V4L2_META_FMT_GENERIC_CSI2_2_24:	descr = "2x8b Generic Meta, 24b CSI-2"; break;
> > >  
> > >  	default:
> > >  		/* Compressed formats */
> > > diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> > > index 78260e5d9985..b4284a564025 100644
> > > --- a/include/uapi/linux/videodev2.h
> > > +++ b/include/uapi/linux/videodev2.h
> > > @@ -838,6 +838,15 @@ struct v4l2_pix_format {
> > >  #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
> > >  #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
> > >  
> > > +#define V4L2_META_FMT_GENERIC_8		v4l2_fourcc('M', 'E', 'T', '8') /* Generic 8-bit metadata */
> > > +#define V4L2_META_FMT_GENERIC_CSI2_10	v4l2_fourcc('M', 'C', '1', 'A') /* 10-bit CSI-2 packed 8-bit metadata */
> > > +#define V4L2_META_FMT_GENERIC_CSI2_12	v4l2_fourcc('M', 'C', '1', 'C') /* 12-bit CSI-2 packed 8-bit metadata */
> > > +#define V4L2_META_FMT_GENERIC_CSI2_14	v4l2_fourcc('M', 'C', '1', 'E') /* 14-bit CSI-2 packed 8-bit metadata */
> > > +#define V4L2_META_FMT_GENERIC_CSI2_16	v4l2_fourcc('M', 'C', '1', 'G') /* 16-bit CSI-2 packed 8-bit metadata */
> > > +#define V4L2_META_FMT_GENERIC_CSI2_20	v4l2_fourcc('M', 'C', '1', 'K') /* 20-bit CSI-2 packed 8-bit metadata */
> > > +#define V4L2_META_FMT_GENERIC_CSI2_24	v4l2_fourcc('M', 'C', '1', 'O') /* 24-bit CSI-2 packed 8-bit metadata */
> > > +#define V4L2_META_FMT_GENERIC_CSI2_2_24	v4l2_fourcc('M', 'C', '2', 'O') /* 2 bytes of 8-bit metadata, 24-bit CSI-2 packed */
> > > +
> > >  /* priv field value to indicates that subsequent fields are valid. */
> > >  #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
> > >  

-- 
Regards,

Laurent Pinchart

Re: [PATCH v3 09/10] media: Add media bus codes for MIPI CCS embedded data

[Laurent Pinchart]

On Wed, Sep 06, 2023 at 01:03:18PM +0000, Sakari Ailus wrote:
> On Tue, Sep 05, 2023 at 08:25:35PM +0300, Laurent Pinchart wrote:
> > On Tue, Aug 08, 2023 at 10:55:37AM +0300, Sakari Ailus wrote:
> > > Add new MIPI CCS embedded data media bus formats.
> > > 
> > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > ---
> > >  .../media/v4l/subdev-formats.rst              | 32 +++++++++++++++++++
> > >  include/uapi/linux/media-bus-format.h         | 10 +++++-
> > >  2 files changed, 41 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > > index c615da08502d..5d5407738af9 100644
> > > --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > > +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > > @@ -8491,3 +8491,35 @@ and finally the bit number in subscript. "p" indicates a padding bit.
> > >        - p
> > >        - p
> > >        - p
> > > +
> > > +MIPI CCS Embedded Data Formats
> > > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > +
> > > +`MIPI CCS <https://www.mipi.org/specifications/camera-command-set>`_ defines an
> > 
> > s/an$/a/
> 
> Yes. I think I had "embedded" there in the past...
> 
> > > +metadata format for sensor embedded data, which is used to store the register
> > > +configuration used for capturing a given frame. The format is defined in the CCS
> > > +specification.
> > 
> > Strictly speaking, the MIPI CCS embedded data format specifies not just
> > the data packing (insertion of padding bytes) and the data encoding (the
> > data format byte and the tag codes), but also the register addresses and
> > values that are reported in the embedded data. Do you envision the media
> > bus formats defined here as being applicable to sensors that use the
> > same packing and encoding as CCS, but different registers, or only to
> > fully compliant CCS sensors ?
> 
> There are sensors that aren't fully compatible with CCS (including those
> compatible with SMIA and SMIA++) but I wouldn't expect the format to be
> used by devices that are entirely incompatible with CCS.

So if a sensor uses the same packing and encoding as CCS, but a
different register set, you would require other media bus codes ? If so,
how would you require those media bus codes to be documented ? The
packing and encoding could be documented as being identical to CCS, but
what about the data itself ? Would you require the datasheet to be
public ? Or the entire register set being documented in the V4L2 spec ?

-- 
Regards,

Laurent Pinchart

Re: [PATCH v3 08/10] media: v4l: Support line-based metadata capture

[Laurent Pinchart]

On Wed, Sep 06, 2023 at 12:24:26PM +0000, Sakari Ailus wrote:
> On Wed, Sep 06, 2023 at 09:21:42AM +0200, Jacopo Mondi wrote:
> > On Tue, Sep 05, 2023 at 08:15:33PM +0300, Laurent Pinchart wrote:
> > > On Mon, Aug 14, 2023 at 11:02:40AM +0000, Sakari Ailus wrote:
> > > > On Thu, Aug 10, 2023 at 05:24:14PM +0200, Jacopo Mondi wrote:
> > > > > On Tue, Aug 08, 2023 at 10:55:36AM +0300, Sakari Ailus wrote:
> > > > > > many camera sensors, among other devices, transmit embedded data and image
> > >
> > > s/many/Many/
> > >
> > > > > > data for each CSI-2 frame. This embedded data typically contains register
> > > > > > configuration of the sensor that has been used to capture the image data
> > > > > > of the same frame.
> > > > > >
> > > > > > The embedded data is received by the CSI-2 receiver and has the same
> > > > > > properties as the image data, including that it is line based: it has
> > > > > > width, height and bytesperline (stride).
> > > > > >
> > > > > > Add these fields to struct v4l2_meta_format and document them.
> > > > > >
> > > > > > Also add V4L2_FMT_FLAG_META_LINE_BASED to tell a given format is
> > > > > > line-based i.e. these fields of struct v4l2_meta_format are valid for it.
> > > > > >
> > > > > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > > > > ---
> > > > > >  .../userspace-api/media/v4l/dev-meta.rst          | 15 +++++++++++++++
> > > > > >  .../userspace-api/media/v4l/vidioc-enum-fmt.rst   |  7 +++++++
> > > > > >  .../media/videodev2.h.rst.exceptions              |  1 +
> > > > > >  drivers/media/v4l2-core/v4l2-ioctl.c              |  5 +++--
> > > > > >  include/uapi/linux/videodev2.h                    | 10 ++++++++++
> > > > > >  5 files changed, 36 insertions(+), 2 deletions(-)
> > > > > >
> > > > > > diff --git a/Documentation/userspace-api/media/v4l/dev-meta.rst b/Documentation/userspace-api/media/v4l/dev-meta.rst
> > > > > > index 0e7e1ee1471a..4b24bae6e171 100644
> > > > > > --- a/Documentation/userspace-api/media/v4l/dev-meta.rst
> > > > > > +++ b/Documentation/userspace-api/media/v4l/dev-meta.rst
> > > > > > @@ -65,3 +65,18 @@ to 0.
> > > > > >        - ``buffersize``
> > > > > >        - Maximum buffer size in bytes required for data. The value is set by the
> > > > > >          driver.
> > > > > > +    * - __u32
> > > > > > +      - ``width``
> > > > > > +      - Width of a line of metadata in samples. Valid when :c:type`v4l2_fmtdesc`
> > > > > > +	flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
> > > > > > +	:c:func:`VIDIOC_ENUM_FMT`.
> > > > > > +    * - __u32
> > > > > > +      - ``height``
> > > > > > +      - Number of rows of metadata. Valid when :c:type`v4l2_fmtdesc` flag
> > > > > > +	``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
> > > > > > +	:c:func:`VIDIOC_ENUM_FMT`.
> > > > > > +    * - __u32
> > > > > > +      - ``bytesperline``
> > > > > > +      - Offset in bytes between the beginning of two consecutive lines. Valid
> > > > > > +	when :c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is
> > > > > > +	set, otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
> > > > > > diff --git a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> > > > > > index 000c154b0f98..6d7664345a4e 100644
> > > > > > --- a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> > > > > > +++ b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> > > > > > @@ -227,6 +227,13 @@ the ``mbus_code`` field is handled differently:
> > > > > >  	The application can ask to configure the quantization of the capture
> > > > > >  	device when calling the :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl with
> > > > > >  	:ref:`V4L2_PIX_FMT_FLAG_SET_CSC <v4l2-pix-fmt-flag-set-csc>` set.
> > > > > > +    * - ``V4L2_FMT_FLAG_META_LINE_BASED``
> > > > > > +      - 0x0200
> > > > > > +      - The metadata format is line-based. In this case the ``width``,
> > > > > > +	``height`` and ``bytesperline`` fields of :c:type:`v4l2_meta_format` are
> > > > > > +	valid. The buffer consists of ``height`` lines, each having ``width``
> > > > > > +	bytes of data and offset between the beginning of each two consecutive
> > > > >
> > > > > Isn't ``width`` in samples ?
> > > >
> > > > Indeed, it's better to refer to samples for clarity. I'll fix for v4.
> > >
> > > How do you define a "sample" in this case ? I wonder if it wouldn't be
> > > simpler for both userspace and kernel drivers if the width was specified
> > > in bytes, including the padding bytes.
> > 
> > Wouldn't this make the image line length (expressed in 'samples')
> > different than the embedded data line length ? Would this confuse
> > userspace or is it fine ?
> 
> If padding is included to width, then the user needs to calculate how many
> bytes of metadata there is, apart from the padding (which is redundant).
> That value is provided to the user for this purpose --- just like for image
> data.

The bytesperline value includes padding at the end of the line to
achieve a particular stride, so that doesn't tell how many bytes to
parse. If the width is specified in "samples", then the parser will need
to compute the number of bytes it spans, and then parse those bytes.

> > > We need an implementation here to test it out. The good news is that I'm
> > > working on it, the bad news is that I'm also busy with other things.
> > >
> > > > I'll also add bytesperline is in bytes (and not in samples).
> > >
> > > Thanks for not messing up (more than needed) with my mental health by
> > > not specifying bytesperline in something else than bytes :-)
> > 
> > :)
> 
> I remember back in the Amiga days when memory was sometimes scarce, nibbles
> were used. That works when bytes have an even number of bits.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

[Sakari Ailus]

Hi Laurent,

On Wed, Sep 06, 2023 at 03:36:58PM +0300, Laurent Pinchart wrote:
> CC'ing Naush and David
> 
> On Wed, Sep 06, 2023 at 11:36:45AM +0000, Sakari Ailus wrote:
> > On Tue, Sep 05, 2023 at 07:47:20PM +0300, Laurent Pinchart wrote:
> > > On Fri, Aug 11, 2023 at 09:11:39AM +0000, Sakari Ailus wrote:
> > > > On Fri, Aug 11, 2023 at 08:31:16AM +0200, Jacopo Mondi wrote:
> > > > > > > +V4L2_META_FMT_GENERIC_CSI2_10
> > > > > > > +-----------------------------
> > > > > > > +
> > > > > > > +V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
> > > > > > > +for each 8 bits of data. Every four bytes of metadata is followed by a single
> > > > > > > +byte of padding. The way the data is stored follows the CSI-2 specification.
> > > > > > > +
> > > > > > > +This format is also used on CSI-2 on 20 bits per sample format that packs two
> > > > > > > +bytes of metadata into one sample.
> > > > > > > +
> > > > > > > +This format is little endian.
> > > > > > > +
> > > > > > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
> > > > > > > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > > > > >
> > > > > > I think you should document whether the padding is always 0 or can be any value.
> > > > > > Perhaps 'X' is a better 'name' for the padding byte in the latter case.
> > > > > 
> > > > > Did I get this right that this format is supposed to work as the RAW10
> > > > > CSI-2 packed image format, where 4 bytes contain the higher 8 bits of
> > > > > the 10 bits sample and the 5th byte every 4 contains the lower 2 bits of
> > > > > the previous 4 sample ?
> > > > > 
> > > > > If that's the case, is 'padding' the correct term here ?
> > > > 
> > > > What else would you call it? It'll be zeros that exist just due to the bit
> > > > depth used and as such not interesting at all.
> > > 
> > > It's actually not 0, CCS requires the padding bytes to be 0x55.
> > > 
> > > I wonder if the conformance test suite tests the contents of the padding
> > > bytes.
> > 
> > I don't know. I could add the value is unspecified but as it has not been
> > specified, there's no change in meaning (just size).
> 
> I started writing that I don't see how it could help applications to
> know that the padding byte is 0x55, but the SMIA++ embedded data parser
> in libcamera actually checks for it, and considers the embedded data to
> be erroneous if it has a different value.

I think it's fine to check for it if you know it's CCS/SMIA++/SMIA embedded
data. But documenting it here isn't a great idea as then other uses of this
format definition would be excluded.

-- 
Regards,

Sakari Ailus

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

[Laurent Pinchart]

On Wed, Sep 06, 2023 at 01:25:53PM +0000, Sakari Ailus wrote:
> On Wed, Sep 06, 2023 at 03:36:58PM +0300, Laurent Pinchart wrote:
> > On Wed, Sep 06, 2023 at 11:36:45AM +0000, Sakari Ailus wrote:
> > > On Tue, Sep 05, 2023 at 07:47:20PM +0300, Laurent Pinchart wrote:
> > > > On Fri, Aug 11, 2023 at 09:11:39AM +0000, Sakari Ailus wrote:
> > > > > On Fri, Aug 11, 2023 at 08:31:16AM +0200, Jacopo Mondi wrote:
> > > > > > > > +V4L2_META_FMT_GENERIC_CSI2_10
> > > > > > > > +-----------------------------
> > > > > > > > +
> > > > > > > > +V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
> > > > > > > > +for each 8 bits of data. Every four bytes of metadata is followed by a single
> > > > > > > > +byte of padding. The way the data is stored follows the CSI-2 specification.
> > > > > > > > +
> > > > > > > > +This format is also used on CSI-2 on 20 bits per sample format that packs two
> > > > > > > > +bytes of metadata into one sample.
> > > > > > > > +
> > > > > > > > +This format is little endian.
> > > > > > > > +
> > > > > > > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
> > > > > > > > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > > > > > >
> > > > > > > I think you should document whether the padding is always 0 or can be any value.
> > > > > > > Perhaps 'X' is a better 'name' for the padding byte in the latter case.
> > > > > > 
> > > > > > Did I get this right that this format is supposed to work as the RAW10
> > > > > > CSI-2 packed image format, where 4 bytes contain the higher 8 bits of
> > > > > > the 10 bits sample and the 5th byte every 4 contains the lower 2 bits of
> > > > > > the previous 4 sample ?
> > > > > > 
> > > > > > If that's the case, is 'padding' the correct term here ?
> > > > > 
> > > > > What else would you call it? It'll be zeros that exist just due to the bit
> > > > > depth used and as such not interesting at all.
> > > > 
> > > > It's actually not 0, CCS requires the padding bytes to be 0x55.
> > > > 
> > > > I wonder if the conformance test suite tests the contents of the padding
> > > > bytes.
> > > 
> > > I don't know. I could add the value is unspecified but as it has not been
> > > specified, there's no change in meaning (just size).
> > 
> > I started writing that I don't see how it could help applications to
> > know that the padding byte is 0x55, but the SMIA++ embedded data parser
> > in libcamera actually checks for it, and considers the embedded data to
> > be erroneous if it has a different value.
> 
> I think it's fine to check for it if you know it's CCS/SMIA++/SMIA embedded
> data. But documenting it here isn't a great idea as then other uses of this
> format definition would be excluded.

I'm OK with that, but note that you've mentioned in a different patch in
the same series that you wouldn't use the CCS media bus code for sensors
that are compliant with the CCS packing and encoding but not the CCS
register set. That's not very consistent :-)

-- 
Regards,

Laurent Pinchart

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

[Sakari Ailus]

Hi Laurent,

On Wed, Sep 06, 2023 at 04:30:57PM +0300, Laurent Pinchart wrote:
> On Wed, Sep 06, 2023 at 01:25:53PM +0000, Sakari Ailus wrote:
> > On Wed, Sep 06, 2023 at 03:36:58PM +0300, Laurent Pinchart wrote:
> > > On Wed, Sep 06, 2023 at 11:36:45AM +0000, Sakari Ailus wrote:
> > > > On Tue, Sep 05, 2023 at 07:47:20PM +0300, Laurent Pinchart wrote:
> > > > > On Fri, Aug 11, 2023 at 09:11:39AM +0000, Sakari Ailus wrote:
> > > > > > On Fri, Aug 11, 2023 at 08:31:16AM +0200, Jacopo Mondi wrote:
> > > > > > > > > +V4L2_META_FMT_GENERIC_CSI2_10
> > > > > > > > > +-----------------------------
> > > > > > > > > +
> > > > > > > > > +V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
> > > > > > > > > +for each 8 bits of data. Every four bytes of metadata is followed by a single
> > > > > > > > > +byte of padding. The way the data is stored follows the CSI-2 specification.
> > > > > > > > > +
> > > > > > > > > +This format is also used on CSI-2 on 20 bits per sample format that packs two
> > > > > > > > > +bytes of metadata into one sample.
> > > > > > > > > +
> > > > > > > > > +This format is little endian.
> > > > > > > > > +
> > > > > > > > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
> > > > > > > > > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > > > > > > >
> > > > > > > > I think you should document whether the padding is always 0 or can be any value.
> > > > > > > > Perhaps 'X' is a better 'name' for the padding byte in the latter case.
> > > > > > > 
> > > > > > > Did I get this right that this format is supposed to work as the RAW10
> > > > > > > CSI-2 packed image format, where 4 bytes contain the higher 8 bits of
> > > > > > > the 10 bits sample and the 5th byte every 4 contains the lower 2 bits of
> > > > > > > the previous 4 sample ?
> > > > > > > 
> > > > > > > If that's the case, is 'padding' the correct term here ?
> > > > > > 
> > > > > > What else would you call it? It'll be zeros that exist just due to the bit
> > > > > > depth used and as such not interesting at all.
> > > > > 
> > > > > It's actually not 0, CCS requires the padding bytes to be 0x55.
> > > > > 
> > > > > I wonder if the conformance test suite tests the contents of the padding
> > > > > bytes.
> > > > 
> > > > I don't know. I could add the value is unspecified but as it has not been
> > > > specified, there's no change in meaning (just size).
> > > 
> > > I started writing that I don't see how it could help applications to
> > > know that the padding byte is 0x55, but the SMIA++ embedded data parser
> > > in libcamera actually checks for it, and considers the embedded data to
> > > be erroneous if it has a different value.
> > 
> > I think it's fine to check for it if you know it's CCS/SMIA++/SMIA embedded
> > data. But documenting it here isn't a great idea as then other uses of this
> > format definition would be excluded.
> 
> I'm OK with that, but note that you've mentioned in a different patch in
> the same series that you wouldn't use the CCS media bus code for sensors
> that are compliant with the CCS packing and encoding but not the CCS
> register set. That's not very consistent :-)

Which specific patch are you referring to?

-- 
Regards,

Sakari Ailus

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

[Laurent Pinchart]

On Wed, Sep 06, 2023 at 01:39:59PM +0000, Sakari Ailus wrote:
> Hi Laurent,
> 
> On Wed, Sep 06, 2023 at 04:30:57PM +0300, Laurent Pinchart wrote:
> > On Wed, Sep 06, 2023 at 01:25:53PM +0000, Sakari Ailus wrote:
> > > On Wed, Sep 06, 2023 at 03:36:58PM +0300, Laurent Pinchart wrote:
> > > > On Wed, Sep 06, 2023 at 11:36:45AM +0000, Sakari Ailus wrote:
> > > > > On Tue, Sep 05, 2023 at 07:47:20PM +0300, Laurent Pinchart wrote:
> > > > > > On Fri, Aug 11, 2023 at 09:11:39AM +0000, Sakari Ailus wrote:
> > > > > > > On Fri, Aug 11, 2023 at 08:31:16AM +0200, Jacopo Mondi wrote:
> > > > > > > > > > +V4L2_META_FMT_GENERIC_CSI2_10
> > > > > > > > > > +-----------------------------
> > > > > > > > > > +
> > > > > > > > > > +V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
> > > > > > > > > > +for each 8 bits of data. Every four bytes of metadata is followed by a single
> > > > > > > > > > +byte of padding. The way the data is stored follows the CSI-2 specification.
> > > > > > > > > > +
> > > > > > > > > > +This format is also used on CSI-2 on 20 bits per sample format that packs two
> > > > > > > > > > +bytes of metadata into one sample.
> > > > > > > > > > +
> > > > > > > > > > +This format is little endian.
> > > > > > > > > > +
> > > > > > > > > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
> > > > > > > > > > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > > > > > > > >
> > > > > > > > > I think you should document whether the padding is always 0 or can be any value.
> > > > > > > > > Perhaps 'X' is a better 'name' for the padding byte in the latter case.
> > > > > > > > 
> > > > > > > > Did I get this right that this format is supposed to work as the RAW10
> > > > > > > > CSI-2 packed image format, where 4 bytes contain the higher 8 bits of
> > > > > > > > the 10 bits sample and the 5th byte every 4 contains the lower 2 bits of
> > > > > > > > the previous 4 sample ?
> > > > > > > > 
> > > > > > > > If that's the case, is 'padding' the correct term here ?
> > > > > > > 
> > > > > > > What else would you call it? It'll be zeros that exist just due to the bit
> > > > > > > depth used and as such not interesting at all.
> > > > > > 
> > > > > > It's actually not 0, CCS requires the padding bytes to be 0x55.
> > > > > > 
> > > > > > I wonder if the conformance test suite tests the contents of the padding
> > > > > > bytes.
> > > > > 
> > > > > I don't know. I could add the value is unspecified but as it has not been
> > > > > specified, there's no change in meaning (just size).
> > > > 
> > > > I started writing that I don't see how it could help applications to
> > > > know that the padding byte is 0x55, but the SMIA++ embedded data parser
> > > > in libcamera actually checks for it, and considers the embedded data to
> > > > be erroneous if it has a different value.
> > > 
> > > I think it's fine to check for it if you know it's CCS/SMIA++/SMIA embedded
> > > data. But documenting it here isn't a great idea as then other uses of this
> > > format definition would be excluded.
> > 
> > I'm OK with that, but note that you've mentioned in a different patch in
> > the same series that you wouldn't use the CCS media bus code for sensors
> > that are compliant with the CCS packing and encoding but not the CCS
> > register set. That's not very consistent :-)
> 
> Which specific patch are you referring to?

"[PATCH v3 09/10] media: Add media bus codes for MIPI CCS embedded data"

You wrote

"There are sensors that aren't fully compatible with CCS (including
those compatible with SMIA and SMIA++) but I wouldn't expect the format
to be used by devices that are entirely incompatible with CCS."

-- 
Regards,

Laurent Pinchart

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

[Sakari Ailus]

Hi Laurent,

On Wed, Sep 06, 2023 at 04:47:37PM +0300, Laurent Pinchart wrote:
> On Wed, Sep 06, 2023 at 01:39:59PM +0000, Sakari Ailus wrote:
> > Hi Laurent,
> > 
> > On Wed, Sep 06, 2023 at 04:30:57PM +0300, Laurent Pinchart wrote:
> > > On Wed, Sep 06, 2023 at 01:25:53PM +0000, Sakari Ailus wrote:
> > > > On Wed, Sep 06, 2023 at 03:36:58PM +0300, Laurent Pinchart wrote:
> > > > > On Wed, Sep 06, 2023 at 11:36:45AM +0000, Sakari Ailus wrote:
> > > > > > On Tue, Sep 05, 2023 at 07:47:20PM +0300, Laurent Pinchart wrote:
> > > > > > > On Fri, Aug 11, 2023 at 09:11:39AM +0000, Sakari Ailus wrote:
> > > > > > > > On Fri, Aug 11, 2023 at 08:31:16AM +0200, Jacopo Mondi wrote:
> > > > > > > > > > > +V4L2_META_FMT_GENERIC_CSI2_10
> > > > > > > > > > > +-----------------------------
> > > > > > > > > > > +
> > > > > > > > > > > +V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
> > > > > > > > > > > +for each 8 bits of data. Every four bytes of metadata is followed by a single
> > > > > > > > > > > +byte of padding. The way the data is stored follows the CSI-2 specification.
> > > > > > > > > > > +
> > > > > > > > > > > +This format is also used on CSI-2 on 20 bits per sample format that packs two
> > > > > > > > > > > +bytes of metadata into one sample.
> > > > > > > > > > > +
> > > > > > > > > > > +This format is little endian.
> > > > > > > > > > > +
> > > > > > > > > > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
> > > > > > > > > > > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > > > > > > > > >
> > > > > > > > > > I think you should document whether the padding is always 0 or can be any value.
> > > > > > > > > > Perhaps 'X' is a better 'name' for the padding byte in the latter case.
> > > > > > > > > 
> > > > > > > > > Did I get this right that this format is supposed to work as the RAW10
> > > > > > > > > CSI-2 packed image format, where 4 bytes contain the higher 8 bits of
> > > > > > > > > the 10 bits sample and the 5th byte every 4 contains the lower 2 bits of
> > > > > > > > > the previous 4 sample ?
> > > > > > > > > 
> > > > > > > > > If that's the case, is 'padding' the correct term here ?
> > > > > > > > 
> > > > > > > > What else would you call it? It'll be zeros that exist just due to the bit
> > > > > > > > depth used and as such not interesting at all.
> > > > > > > 
> > > > > > > It's actually not 0, CCS requires the padding bytes to be 0x55.
> > > > > > > 
> > > > > > > I wonder if the conformance test suite tests the contents of the padding
> > > > > > > bytes.
> > > > > > 
> > > > > > I don't know. I could add the value is unspecified but as it has not been
> > > > > > specified, there's no change in meaning (just size).
> > > > > 
> > > > > I started writing that I don't see how it could help applications to
> > > > > know that the padding byte is 0x55, but the SMIA++ embedded data parser
> > > > > in libcamera actually checks for it, and considers the embedded data to
> > > > > be erroneous if it has a different value.
> > > > 
> > > > I think it's fine to check for it if you know it's CCS/SMIA++/SMIA embedded
> > > > data. But documenting it here isn't a great idea as then other uses of this
> > > > format definition would be excluded.
> > > 
> > > I'm OK with that, but note that you've mentioned in a different patch in
> > > the same series that you wouldn't use the CCS media bus code for sensors
> > > that are compliant with the CCS packing and encoding but not the CCS
> > > register set. That's not very consistent :-)
> > 
> > Which specific patch are you referring to?
> 
> "[PATCH v3 09/10] media: Add media bus codes for MIPI CCS embedded data"
> 
> You wrote
> 
> "There are sensors that aren't fully compatible with CCS (including
> those compatible with SMIA and SMIA++) but I wouldn't expect the format
> to be used by devices that are entirely incompatible with CCS."

Ah, right.

I meant that if a sensor implementation isn't related to any of these
standards, it's highly unlikely to use the same embedded data format. Of
course, if it does, then I think we should use the mbus code, too. But I'm
not holding my breath.

-- 
Regards,

Sakari Ailus

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

[Sakari Ailus]

On Thu, Sep 07, 2023 at 08:06:10AM +0000, Sakari Ailus wrote:
> Hi Laurent,
> 
> On Wed, Sep 06, 2023 at 04:47:37PM +0300, Laurent Pinchart wrote:
> > On Wed, Sep 06, 2023 at 01:39:59PM +0000, Sakari Ailus wrote:
> > > Hi Laurent,
> > > 
> > > On Wed, Sep 06, 2023 at 04:30:57PM +0300, Laurent Pinchart wrote:
> > > > On Wed, Sep 06, 2023 at 01:25:53PM +0000, Sakari Ailus wrote:
> > > > > On Wed, Sep 06, 2023 at 03:36:58PM +0300, Laurent Pinchart wrote:
> > > > > > On Wed, Sep 06, 2023 at 11:36:45AM +0000, Sakari Ailus wrote:
> > > > > > > On Tue, Sep 05, 2023 at 07:47:20PM +0300, Laurent Pinchart wrote:
> > > > > > > > On Fri, Aug 11, 2023 at 09:11:39AM +0000, Sakari Ailus wrote:
> > > > > > > > > On Fri, Aug 11, 2023 at 08:31:16AM +0200, Jacopo Mondi wrote:
> > > > > > > > > > > > +V4L2_META_FMT_GENERIC_CSI2_10
> > > > > > > > > > > > +-----------------------------
> > > > > > > > > > > > +
> > > > > > > > > > > > +V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
> > > > > > > > > > > > +for each 8 bits of data. Every four bytes of metadata is followed by a single
> > > > > > > > > > > > +byte of padding. The way the data is stored follows the CSI-2 specification.
> > > > > > > > > > > > +
> > > > > > > > > > > > +This format is also used on CSI-2 on 20 bits per sample format that packs two
> > > > > > > > > > > > +bytes of metadata into one sample.
> > > > > > > > > > > > +
> > > > > > > > > > > > +This format is little endian.
> > > > > > > > > > > > +
> > > > > > > > > > > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
> > > > > > > > > > > > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > > > > > > > > > >
> > > > > > > > > > > I think you should document whether the padding is always 0 or can be any value.
> > > > > > > > > > > Perhaps 'X' is a better 'name' for the padding byte in the latter case.
> > > > > > > > > > 
> > > > > > > > > > Did I get this right that this format is supposed to work as the RAW10
> > > > > > > > > > CSI-2 packed image format, where 4 bytes contain the higher 8 bits of
> > > > > > > > > > the 10 bits sample and the 5th byte every 4 contains the lower 2 bits of
> > > > > > > > > > the previous 4 sample ?
> > > > > > > > > > 
> > > > > > > > > > If that's the case, is 'padding' the correct term here ?
> > > > > > > > > 
> > > > > > > > > What else would you call it? It'll be zeros that exist just due to the bit
> > > > > > > > > depth used and as such not interesting at all.
> > > > > > > > 
> > > > > > > > It's actually not 0, CCS requires the padding bytes to be 0x55.
> > > > > > > > 
> > > > > > > > I wonder if the conformance test suite tests the contents of the padding
> > > > > > > > bytes.
> > > > > > > 
> > > > > > > I don't know. I could add the value is unspecified but as it has not been
> > > > > > > specified, there's no change in meaning (just size).
> > > > > > 
> > > > > > I started writing that I don't see how it could help applications to
> > > > > > know that the padding byte is 0x55, but the SMIA++ embedded data parser
> > > > > > in libcamera actually checks for it, and considers the embedded data to
> > > > > > be erroneous if it has a different value.
> > > > > 
> > > > > I think it's fine to check for it if you know it's CCS/SMIA++/SMIA embedded
> > > > > data. But documenting it here isn't a great idea as then other uses of this
> > > > > format definition would be excluded.
> > > > 
> > > > I'm OK with that, but note that you've mentioned in a different patch in
> > > > the same series that you wouldn't use the CCS media bus code for sensors
> > > > that are compliant with the CCS packing and encoding but not the CCS
> > > > register set. That's not very consistent :-)
> > > 
> > > Which specific patch are you referring to?
> > 
> > "[PATCH v3 09/10] media: Add media bus codes for MIPI CCS embedded data"
> > 
> > You wrote
> > 
> > "There are sensors that aren't fully compatible with CCS (including
> > those compatible with SMIA and SMIA++) but I wouldn't expect the format
> > to be used by devices that are entirely incompatible with CCS."
> 
> Ah, right.
> 
> I meant that if a sensor implementation isn't related to any of these
> standards, it's highly unlikely to use the same embedded data format. Of
> course, if it does, then I think we should use the mbus code, too. But I'm
> not holding my breath.

To clarify: to use that format, I'd expect the sensor to use standard CCS
registers. If it doesn't, a new mbus code should be added and documented.
The CCS embedded data header isn't enough to justify the use of the mbus
code --- the documentation here also says "which is used to store the
register configuration used for capturing a given frame".

-- 
Regards,

Sakari Ailus

Re: [PATCH v3 06/10] media: uapi: Add a macro to tell whether an mbus code is metadata

[Sakari Ailus]

On Wed, Sep 06, 2023 at 01:06:36PM +0000, Sakari Ailus wrote:
> On Wed, Sep 06, 2023 at 03:23:08PM +0300, Laurent Pinchart wrote:
> > On Wed, Sep 06, 2023 at 11:33:30AM +0000, Sakari Ailus wrote:
> > > On Tue, Sep 05, 2023 at 08:06:04PM +0300, Laurent Pinchart wrote:
> > > > On Tue, Sep 05, 2023 at 10:37:45AM +0000, Sakari Ailus wrote:
> > > > > On Tue, Sep 05, 2023 at 12:47:21PM +0300, Tomi Valkeinen wrote:
> > > > > > On 08/08/2023 10:55, Sakari Ailus wrote:
> > > > > > > Add a macro to tell whether a given mbus code is metadata.
> > > > > > > 
> > > > > > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > > > > > ---
> > > > > > >   include/uapi/linux/media-bus-format.h | 3 +++
> > > > > > >   1 file changed, 3 insertions(+)
> > > > > > > 
> > > > > > > diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
> > > > > > > index 9ee031397372..2486b4178c5f 100644
> > > > > > > --- a/include/uapi/linux/media-bus-format.h
> > > > > > > +++ b/include/uapi/linux/media-bus-format.h
> > > > > > > @@ -182,4 +182,7 @@
> > > > > > >   #define MEDIA_BUS_FMT_META_20			0x8006
> > > > > > >   #define MEDIA_BUS_FMT_META_24			0x8007
> > > > > > > +#define MEDIA_BUS_FMT_IS_META(code)		\
> > > > > > > +	((code) & 0xf000 == 0x7000 || (code) & 0xf000 == 0x8000)
> > > > > > > +
> > > > > > >   #endif /* __LINUX_MEDIA_BUS_FORMAT_H */
> > > > > > 
> > > > > > mbus code seems to be u32, so the above won't work. Maybe:
> > > > > > 
> > > > > > (((code) & ~0xfffULL) == 0x7000 || ((code) & ~0xfffULL) == 0x8000)
> > > > > > 
> > > > > > Also, embedded formats with 0x9nnn codes are added later in the series.
> > > > > 
> > > > > Thanks, I'll address these for v4.
> > > > 
> > > > It would be nice to make this an inline function, to avoid evaluating
> > > > the code twice. I think you can move it to an internal kernel header, it
> > > > doesn't need to be exposed to userspace.
> > > 
> > > This was meant for user space consumption for setting the buffer type to
> > > the video node accordingly. It's certainly possible to do that without this
> > > macro, too.
> > 
> > I think userspace would be fine without it :-)
> 
> I'll see if this would seem useful in yavta support before dropping it: the
> buffer type needs to be specified correctly and knowing the format is a
> metadata format maybe helpful. Let's see.

As yavta doesn't deal with mbus codes, this isn't useful for it. I'll drop
it for now.

-- 
Sakari Ailus

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

[Sakari Ailus]

On Wed, Sep 06, 2023 at 11:56:47AM +0000, Sakari Ailus wrote:
> Hi Laurent,
> 
> On Tue, Sep 05, 2023 at 07:55:58PM +0300, Laurent Pinchart wrote:
> > On Tue, Aug 08, 2023 at 10:55:35AM +0300, Sakari Ailus wrote:
> > > Generic 8-bit metadata formats define the in-memory data layout but not
> > > the format of the data itself. The reasoning for having such formats is to
> > > allow CSI-2 receiver drivers to receive and DMA drivers to write the data
> > > to memory without knowing a large number of device specific formats.
> > > 
> > > These formats may be used only in conjunction of a Media controller
> > > pipeline where the internal pad of the source sub-device defines the
> > > specific format of the data (using an mbus code).
> > > 
> > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > ---
> > >  .../userspace-api/media/v4l/meta-formats.rst  |   1 +
> > >  .../media/v4l/metafmt-generic.rst             | 331 ++++++++++++++++++
> > >  drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
> > >  include/uapi/linux/videodev2.h                |   9 +
> > >  4 files changed, 349 insertions(+)
> > >  create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > > 
> > > diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
> > > index 0bb61fc5bc00..919f595576b9 100644
> > > --- a/Documentation/userspace-api/media/v4l/meta-formats.rst
> > > +++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
> > > @@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only.
> > >      metafmt-vsp1-hgo
> > >      metafmt-vsp1-hgt
> > >      metafmt-vivid
> > > +    metafmt-generic
> > > diff --git a/Documentation/userspace-api/media/v4l/metafmt-generic.rst b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > > new file mode 100644
> > > index 000000000000..a27bfc721edf
> > > --- /dev/null
> > > +++ b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > > @@ -0,0 +1,331 @@
> > > +.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
> > > +
> > > +**************************************************************************************************************************************************************************************************************************************************************************************************************************
> > > +V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O'), V4L2_META_FMT_GENERIC_CSI2_2_24 ('MC2O')
> > > +**************************************************************************************************************************************************************************************************************************************************************************************************************************
> > > +
> > > +
> > > +Generic line-based metadata formats
> > > +
> > > +
> > > +Description
> > > +===========
> > > +
> > > +These generic line-based metadata formats define the memory layout of the data
> > > +without defining the format or meaning of the metadata itself. These formats may
> > > +only be used with a Media controller pipeline where the more specific format is
> > > +defined in an :ref:`internal source pad <MEDIA-PAD-FL-INTERNAL>` of the source
> > > +sub-device. See also :ref:`source routes <v4l2-subdev-source-routes>`.
> > > +
> > > +.. _v4l2-meta-fmt-generic-8:
> > > +
> > > +V4L2_META_FMT_GENERIC_8
> > > +-----------------------
> > > +
> > > +The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format.
> > > +
> > > +This format is also used on CSI-2 on both 8 bits per sample as well as on
> > 
> > s/also on/by/
> > 
> > I would also mention "MIPI CCS" instead of "CSI-2".
> 
> If CCS were to be mentioned here, then all uses of this format should be
> included as well.
> 
> > 
> > > +16 bits per sample when two bytes of metadata are packed into one sample.
> > 
> > "bits per sample" is very ill-defined for metadata, as there's no
> > sample. I would write "for both the RAW8 packing and the 2 bytes RAW16
> > packing" or something similar.
> > 
> > Similar comments for below.
> 
> From CSI-2 bus point of view there's no difference between pixel and
> embedded data when it comes to encoding that data. "Sample" is the next
> best term beyond "pixel", as the bus can carry samples that may or may not
> be pixel data. But I'm fine with changing the wording if you think it makes
> it more understandable.

I'll use "data unit" in v4.

> 
> > 
> > > +
> > > +**Byte Order Of V4L2_META_FMT_GENERIC_8.**
> > > +Each cell is one byte. "M" denotes a byte of metadata.
> > > +
> > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|
> > > +
> > > +.. flat-table::
> > > +    :header-rows:  0
> > > +    :stub-columns: 0
> > > +    :widths: 12 8 8 8 8
> > > +
> > > +    * - start + 0:
> > > +      - M\ :sub:`00`
> > > +      - M\ :sub:`10`
> > > +      - M\ :sub:`20`
> > > +      - M\ :sub:`30`
> > > +    * - start + 4:
> > > +      - M\ :sub:`01`
> > > +      - M\ :sub:`11`
> > > +      - M\ :sub:`21`
> > > +      - M\ :sub:`31`
> > > +
> > > +.. _v4l2-meta-fmt-generic-csi2-10:
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_10
> > > +-----------------------------
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
> > > +for each 8 bits of data. Every four bytes of metadata is followed by a single
> > > +byte of padding.
> > 
> > It sounds really weird to write that this format writes 10 bits for each
> > 8 bits of data, when essentially it adds a packing byte every four
> > bytes.
> 
> That's how the hardware has been implemented and probably there is a
> hardware implementation related reason for this.
> 
> > 
> > > The way the data is stored follows the CSI-2 specification.
> > 
> > Again, it's CCS, not CSI-2.
> 
> The packing comes from CSI-2 but CCS adds padding. I'll add this
> information.
> 
> > 
> > > +
> > > +This format is also used on CSI-2 on 20 bits per sample format that packs two
> > > +bytes of metadata into one sample.
> > > +
> > > +This format is little endian.
> > > +
> > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
> > > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > > +
> > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
> > > +
> > > +.. flat-table::
> > > +    :header-rows:  0
> > > +    :stub-columns: 0
> > > +    :widths: 12 8 8 8 8 8
> > > +
> > > +    * - start + 0:
> > > +      - M\ :sub:`00`
> > > +      - M\ :sub:`10`
> > > +      - M\ :sub:`20`
> > > +      - M\ :sub:`30`
> > > +      - p
> > > +    * - start + 5:
> > > +      - M\ :sub:`01`
> > > +      - M\ :sub:`11`
> > > +      - M\ :sub:`21`
> > > +      - M\ :sub:`31`
> > > +      - p
> > > +
> > > +.. _v4l2-meta-fmt-generic-csi2-12:
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_12
> > > +-----------------------------
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_12 contains packed 8-bit generic metadata, 12 bits
> > > +for each 8 bits of data. Every four bytes of metadata is followed by two bytes
> > > +of padding. The way the data is stored follows the CSI-2 specification.
> > > +
> > > +This format is little endian.
> > > +
> > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_12.**
> > > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > > +
> > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> > > +
> > > +.. flat-table::
> > > +    :header-rows:  0
> > > +    :stub-columns: 0
> > > +    :widths: 12 8 8 8 8 8 8
> > > +
> > > +    * - start + 0:
> > > +      - M\ :sub:`00`
> > > +      - M\ :sub:`10`
> > > +      - M\ :sub:`20`
> > > +      - M\ :sub:`30`
> > > +      - p
> > > +      - p
> > > +    * - start + 6:
> > > +      - M\ :sub:`01`
> > > +      - M\ :sub:`11`
> > > +      - M\ :sub:`21`
> > > +      - M\ :sub:`31`
> > > +      - p
> > > +      - p
> > > +
> > > +.. _v4l2-meta-fmt-generic-csi2-14:
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_14
> > > +-----------------------------
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_14 contains packed 8-bit generic metadata, 14 bits
> > > +for each 8 bits of data. Every four bytes of metadata is followed by three
> > > +bytes of padding. The way the data is stored follows the CSI-2 specification.
> > > +
> > > +This format is little endian.
> > > +
> > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_14.**
> > > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > > +
> > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{.8cm}|
> > > +
> > > +.. flat-table::
> > > +    :header-rows:  0
> > > +    :stub-columns: 0
> > > +    :widths: 12 8 8 8 8 8 8 8
> > > +
> > > +    * - start + 0:
> > > +      - M\ :sub:`00`
> > > +      - M\ :sub:`10`
> > > +      - M\ :sub:`20`
> > > +      - M\ :sub:`30`
> > > +      - p
> > > +      - p
> > > +      - p
> > > +    * - start + 7:
> > > +      - M\ :sub:`01`
> > > +      - M\ :sub:`11`
> > > +      - M\ :sub:`21`
> > > +      - M\ :sub:`31`
> > > +      - p
> > > +      - p
> > > +      - p
> > > +
> > > +.. _v4l2-meta-fmt-generic-csi2-16:
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_16
> > > +-----------------------------
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_16 contains packed 8-bit generic metadata, 16 bits
> > > +for each 8 bits of data. Every byte of metadata is followed by one byte of
> > > +padding. The way the data is stored follows the CSI-2 specification.
> > > +
> > > +This format is little endian.
> > > +
> > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_16.**
> > > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > > +
> > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|
> > > +
> > > +.. flat-table::
> > > +    :header-rows:  0
> > > +    :stub-columns: 0
> > > +    :widths: 12 8 8 8 8 8 8 8 8
> > > +
> > > +    * - start + 0:
> > > +      - M\ :sub:`00`
> > > +      - p
> > > +      - M\ :sub:`10`
> > > +      - p
> > > +      - M\ :sub:`20`
> > > +      - p
> > > +      - M\ :sub:`30`
> > > +      - p
> > > +    * - start + 8:
> > > +      - M\ :sub:`01`
> > > +      - p
> > > +      - M\ :sub:`11`
> > > +      - p
> > > +      - M\ :sub:`21`
> > > +      - p
> > > +      - M\ :sub:`31`
> > > +      - p
> > > +
> > > +.. _v4l2-meta-fmt-generic-csi2-20:
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_20
> > > +-----------------------------
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_20 contains packed 8-bit generic metadata, 20 bits
> > > +for each 8 bits of data. Every byte of metadata is followed by alternating one
> > > +and two bytes of padding. The way the data is stored follows the CSI-2
> > > +specification.
> > > +
> > > +This format is little endian.
> > > +
> > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_20.**
> > > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > > +
> > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> > > +
> > > +.. flat-table::
> > > +    :header-rows:  0
> > > +    :stub-columns: 0
> > > +    :widths: 12 8 8 8 8 8 8 8 8 8 8
> > > +
> > > +    * - start + 0:
> > > +      - M\ :sub:`00`
> > > +      - p
> > > +      - M\ :sub:`10`
> > > +      - p
> > > +      - p
> > > +      - M\ :sub:`20`
> > > +      - p
> > > +      - M\ :sub:`30`
> > > +      - p
> > > +      - p
> > > +    * - start + 10:
> > > +      - M\ :sub:`01`
> > > +      - p
> > > +      - M\ :sub:`11`
> > > +      - p
> > > +      - p
> > > +      - M\ :sub:`21`
> > > +      - p
> > > +      - M\ :sub:`31`
> > > +      - p
> > > +      - p
> > > +
> > > +.. _v4l2-meta-fmt-generic-csi2-24:
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_24
> > > +-----------------------------
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_24 contains packed 8-bit generic metadata, 24 bits
> > > +for each 8 bits of data. Every byte of metadata is followed by two bytes of
> > > +padding. The way the data is stored follows the CSI-2 specification.
> > > +
> > > +This format is little endian.
> > > +
> > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_24.**
> > > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > > +
> > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> > > +
> > > +.. flat-table::
> > > +    :header-rows:  0
> > > +    :stub-columns: 0
> > > +    :widths: 12 8 8 8 8 8 8 8 8 8 8 8 8
> > > +
> > > +    * - start + 0:
> > > +      - M\ :sub:`00`
> > > +      - p
> > > +      - p
> > > +      - M\ :sub:`10`
> > > +      - p
> > > +      - p
> > > +      - M\ :sub:`20`
> > > +      - p
> > > +      - p
> > > +      - M\ :sub:`30`
> > > +      - p
> > > +      - p
> > > +    * - start + 12:
> > > +      - M\ :sub:`01`
> > > +      - p
> > > +      - p
> > > +      - M\ :sub:`11`
> > > +      - p
> > > +      - p
> > > +      - M\ :sub:`21`
> > > +      - p
> > > +      - p
> > > +      - M\ :sub:`31`
> > > +      - p
> > > +      - p
> > > +
> > > +.. _v4l2-meta-fmt-generic-csi2-2-24:
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_2_24
> > > +-------------------------------
> > > +
> > > +V4L2_META_FMT_GENERIC_CSI2_2_24 contains packed 8-bit generic metadata, 24 bits
> > > +for each two times 8 bits of data. Every two bytes of metadata are followed by
> > > +one byte of padding. The way the data is stored follows the CSI-2
> > > +specification.
> > > +
> > > +This format is little endian.
> > > +
> > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_2_24.**
> > > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > > +
> > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
> > > +
> > > +.. flat-table::
> > > +    :header-rows:  0
> > > +    :stub-columns: 0
> > > +    :widths: 12 8 8 8 8 8 8
> > > +
> > > +    * - start + 0:
> > > +      - M\ :sub:`00`
> > > +      - M\ :sub:`10`
> > > +      - p
> > > +      - M\ :sub:`20`
> > > +      - M\ :sub:`30`
> > > +      - p
> > > +    * - start + 6:
> > > +      - M\ :sub:`01`
> > > +      - M\ :sub:`11`
> > > +      - p
> > > +      - M\ :sub:`21`
> > > +      - M\ :sub:`31`
> > > +      - p
> > > +
> > > diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> > > index f4d9d6279094..fbbddc333a30 100644
> > > --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> > > +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> > > @@ -1452,6 +1452,14 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
> > >  	case V4L2_PIX_FMT_Y210:		descr = "10-bit YUYV Packed"; break;
> > >  	case V4L2_PIX_FMT_Y212:		descr = "12-bit YUYV Packed"; break;
> > >  	case V4L2_PIX_FMT_Y216:		descr = "16-bit YUYV Packed"; break;
> > > +	case V4L2_META_FMT_GENERIC_8:	descr = "8-bit Generic Metadata"; break;
> > > +	case V4L2_META_FMT_GENERIC_CSI2_10:	descr = "8b Generic Meta, 10b CSI-2"; break;
> > > +	case V4L2_META_FMT_GENERIC_CSI2_12:	descr = "8b Generic Meta, 12b CSI-2"; break;
> > > +	case V4L2_META_FMT_GENERIC_CSI2_14:	descr = "8b Generic Meta, 14b CSI-2"; break;
> > > +	case V4L2_META_FMT_GENERIC_CSI2_16:	descr = "8b Generic Meta, 16b CSI-2"; break;
> > > +	case V4L2_META_FMT_GENERIC_CSI2_20:	descr = "8b Generic Meta, 20b CSI-2"; break;
> > > +	case V4L2_META_FMT_GENERIC_CSI2_24:	descr = "8b Generic Meta, 24b CSI-2"; break;
> > > +	case V4L2_META_FMT_GENERIC_CSI2_2_24:	descr = "2x8b Generic Meta, 24b CSI-2"; break;
> > >  
> > >  	default:
> > >  		/* Compressed formats */
> > > diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> > > index 78260e5d9985..b4284a564025 100644
> > > --- a/include/uapi/linux/videodev2.h
> > > +++ b/include/uapi/linux/videodev2.h
> > > @@ -838,6 +838,15 @@ struct v4l2_pix_format {
> > >  #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
> > >  #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
> > >  
> > > +#define V4L2_META_FMT_GENERIC_8		v4l2_fourcc('M', 'E', 'T', '8') /* Generic 8-bit metadata */
> > > +#define V4L2_META_FMT_GENERIC_CSI2_10	v4l2_fourcc('M', 'C', '1', 'A') /* 10-bit CSI-2 packed 8-bit metadata */
> > > +#define V4L2_META_FMT_GENERIC_CSI2_12	v4l2_fourcc('M', 'C', '1', 'C') /* 12-bit CSI-2 packed 8-bit metadata */
> > > +#define V4L2_META_FMT_GENERIC_CSI2_14	v4l2_fourcc('M', 'C', '1', 'E') /* 14-bit CSI-2 packed 8-bit metadata */
> > > +#define V4L2_META_FMT_GENERIC_CSI2_16	v4l2_fourcc('M', 'C', '1', 'G') /* 16-bit CSI-2 packed 8-bit metadata */
> > > +#define V4L2_META_FMT_GENERIC_CSI2_20	v4l2_fourcc('M', 'C', '1', 'K') /* 20-bit CSI-2 packed 8-bit metadata */
> > > +#define V4L2_META_FMT_GENERIC_CSI2_24	v4l2_fourcc('M', 'C', '1', 'O') /* 24-bit CSI-2 packed 8-bit metadata */
> > > +#define V4L2_META_FMT_GENERIC_CSI2_2_24	v4l2_fourcc('M', 'C', '2', 'O') /* 2 bytes of 8-bit metadata, 24-bit CSI-2 packed */
> > > +
> > >  /* priv field value to indicates that subsequent fields are valid. */
> > >  #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
> > >  
> > 
> 
> -- 
> Regards,
> 
> Sakari Ailus

-- 
Sakari Ailus

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

[Laurent Pinchart]

On Thu, Sep 07, 2023 at 08:36:43AM +0000, Sakari Ailus wrote:
> On Wed, Sep 06, 2023 at 11:56:47AM +0000, Sakari Ailus wrote:
> > On Tue, Sep 05, 2023 at 07:55:58PM +0300, Laurent Pinchart wrote:
> > > On Tue, Aug 08, 2023 at 10:55:35AM +0300, Sakari Ailus wrote:
> > > > Generic 8-bit metadata formats define the in-memory data layout but not
> > > > the format of the data itself. The reasoning for having such formats is to
> > > > allow CSI-2 receiver drivers to receive and DMA drivers to write the data
> > > > to memory without knowing a large number of device specific formats.
> > > > 
> > > > These formats may be used only in conjunction of a Media controller
> > > > pipeline where the internal pad of the source sub-device defines the
> > > > specific format of the data (using an mbus code).
> > > > 
> > > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > > ---
> > > >  .../userspace-api/media/v4l/meta-formats.rst  |   1 +
> > > >  .../media/v4l/metafmt-generic.rst             | 331 ++++++++++++++++++
> > > >  drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
> > > >  include/uapi/linux/videodev2.h                |   9 +
> > > >  4 files changed, 349 insertions(+)
> > > >  create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > > > 
> > > > diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
> > > > index 0bb61fc5bc00..919f595576b9 100644
> > > > --- a/Documentation/userspace-api/media/v4l/meta-formats.rst
> > > > +++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
> > > > @@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only.
> > > >      metafmt-vsp1-hgo
> > > >      metafmt-vsp1-hgt
> > > >      metafmt-vivid
> > > > +    metafmt-generic
> > > > diff --git a/Documentation/userspace-api/media/v4l/metafmt-generic.rst b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > > > new file mode 100644
> > > > index 000000000000..a27bfc721edf
> > > > --- /dev/null
> > > > +++ b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > > > @@ -0,0 +1,331 @@
> > > > +.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
> > > > +
> > > > +**************************************************************************************************************************************************************************************************************************************************************************************************************************
> > > > +V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O'), V4L2_META_FMT_GENERIC_CSI2_2_24 ('MC2O')
> > > > +**************************************************************************************************************************************************************************************************************************************************************************************************************************
> > > > +
> > > > +
> > > > +Generic line-based metadata formats
> > > > +
> > > > +
> > > > +Description
> > > > +===========
> > > > +
> > > > +These generic line-based metadata formats define the memory layout of the data
> > > > +without defining the format or meaning of the metadata itself. These formats may
> > > > +only be used with a Media controller pipeline where the more specific format is
> > > > +defined in an :ref:`internal source pad <MEDIA-PAD-FL-INTERNAL>` of the source
> > > > +sub-device. See also :ref:`source routes <v4l2-subdev-source-routes>`.
> > > > +
> > > > +.. _v4l2-meta-fmt-generic-8:
> > > > +
> > > > +V4L2_META_FMT_GENERIC_8
> > > > +-----------------------
> > > > +
> > > > +The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format.
> > > > +
> > > > +This format is also used on CSI-2 on both 8 bits per sample as well as on
> > > 
> > > s/also on/by/
> > > 
> > > I would also mention "MIPI CCS" instead of "CSI-2".
> > 
> > If CCS were to be mentioned here, then all uses of this format should be
> > included as well.
> > 
> > > > +16 bits per sample when two bytes of metadata are packed into one sample.
> > > 
> > > "bits per sample" is very ill-defined for metadata, as there's no
> > > sample. I would write "for both the RAW8 packing and the 2 bytes RAW16
> > > packing" or something similar.
> > > 
> > > Similar comments for below.
> > 
> > From CSI-2 bus point of view there's no difference between pixel and
> > embedded data when it comes to encoding that data. "Sample" is the next
> > best term beyond "pixel", as the bus can carry samples that may or may not
> > be pixel data. But I'm fine with changing the wording if you think it makes
> > it more understandable.
> 
> I'll use "data unit" in v4.

Please define the term :-)

> > > > +
> > > > +**Byte Order Of V4L2_META_FMT_GENERIC_8.**
> > > > +Each cell is one byte. "M" denotes a byte of metadata.
> > > > +
> > > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|
> > > > +
> > > > +.. flat-table::
> > > > +    :header-rows:  0
> > > > +    :stub-columns: 0
> > > > +    :widths: 12 8 8 8 8
> > > > +
> > > > +    * - start + 0:
> > > > +      - M\ :sub:`00`
> > > > +      - M\ :sub:`10`
> > > > +      - M\ :sub:`20`
> > > > +      - M\ :sub:`30`
> > > > +    * - start + 4:
> > > > +      - M\ :sub:`01`
> > > > +      - M\ :sub:`11`
> > > > +      - M\ :sub:`21`
> > > > +      - M\ :sub:`31`
> > > > +
> > > > +.. _v4l2-meta-fmt-generic-csi2-10:
> > > > +
> > > > +V4L2_META_FMT_GENERIC_CSI2_10
> > > > +-----------------------------
> > > > +
> > > > +V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
> > > > +for each 8 bits of data. Every four bytes of metadata is followed by a single
> > > > +byte of padding.
> > > 
> > > It sounds really weird to write that this format writes 10 bits for each
> > > 8 bits of data, when essentially it adds a packing byte every four
> > > bytes.
> > 
> > That's how the hardware has been implemented and probably there is a
> > hardware implementation related reason for this.
> > 
> > > 
> > > > The way the data is stored follows the CSI-2 specification.
> > > 
> > > Again, it's CCS, not CSI-2.
> > 
> > The packing comes from CSI-2 but CCS adds padding. I'll add this
> > information.
> > 
> > > 
> > > > +
> > > > +This format is also used on CSI-2 on 20 bits per sample format that packs two
> > > > +bytes of metadata into one sample.
> > > > +
> > > > +This format is little endian.
> > > > +
> > > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
> > > > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > > > +
> > > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
> > > > +
> > > > +.. flat-table::
> > > > +    :header-rows:  0
> > > > +    :stub-columns: 0
> > > > +    :widths: 12 8 8 8 8 8
> > > > +
> > > > +    * - start + 0:
> > > > +      - M\ :sub:`00`
> > > > +      - M\ :sub:`10`
> > > > +      - M\ :sub:`20`
> > > > +      - M\ :sub:`30`
> > > > +      - p
> > > > +    * - start + 5:
> > > > +      - M\ :sub:`01`
> > > > +      - M\ :sub:`11`
> > > > +      - M\ :sub:`21`
> > > > +      - M\ :sub:`31`
> > > > +      - p
> > > > +
> > > > +.. _v4l2-meta-fmt-generic-csi2-12:
> > > > +
> > > > +V4L2_META_FMT_GENERIC_CSI2_12
> > > > +-----------------------------
> > > > +
> > > > +V4L2_META_FMT_GENERIC_CSI2_12 contains packed 8-bit generic metadata, 12 bits
> > > > +for each 8 bits of data. Every four bytes of metadata is followed by two bytes
> > > > +of padding. The way the data is stored follows the CSI-2 specification.
> > > > +
> > > > +This format is little endian.
> > > > +
> > > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_12.**
> > > > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > > > +
> > > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> > > > +
> > > > +.. flat-table::
> > > > +    :header-rows:  0
> > > > +    :stub-columns: 0
> > > > +    :widths: 12 8 8 8 8 8 8
> > > > +
> > > > +    * - start + 0:
> > > > +      - M\ :sub:`00`
> > > > +      - M\ :sub:`10`
> > > > +      - M\ :sub:`20`
> > > > +      - M\ :sub:`30`
> > > > +      - p
> > > > +      - p
> > > > +    * - start + 6:
> > > > +      - M\ :sub:`01`
> > > > +      - M\ :sub:`11`
> > > > +      - M\ :sub:`21`
> > > > +      - M\ :sub:`31`
> > > > +      - p
> > > > +      - p
> > > > +
> > > > +.. _v4l2-meta-fmt-generic-csi2-14:
> > > > +
> > > > +V4L2_META_FMT_GENERIC_CSI2_14
> > > > +-----------------------------
> > > > +
> > > > +V4L2_META_FMT_GENERIC_CSI2_14 contains packed 8-bit generic metadata, 14 bits
> > > > +for each 8 bits of data. Every four bytes of metadata is followed by three
> > > > +bytes of padding. The way the data is stored follows the CSI-2 specification.
> > > > +
> > > > +This format is little endian.
> > > > +
> > > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_14.**
> > > > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > > > +
> > > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{.8cm}|
> > > > +
> > > > +.. flat-table::
> > > > +    :header-rows:  0
> > > > +    :stub-columns: 0
> > > > +    :widths: 12 8 8 8 8 8 8 8
> > > > +
> > > > +    * - start + 0:
> > > > +      - M\ :sub:`00`
> > > > +      - M\ :sub:`10`
> > > > +      - M\ :sub:`20`
> > > > +      - M\ :sub:`30`
> > > > +      - p
> > > > +      - p
> > > > +      - p
> > > > +    * - start + 7:
> > > > +      - M\ :sub:`01`
> > > > +      - M\ :sub:`11`
> > > > +      - M\ :sub:`21`
> > > > +      - M\ :sub:`31`
> > > > +      - p
> > > > +      - p
> > > > +      - p
> > > > +
> > > > +.. _v4l2-meta-fmt-generic-csi2-16:
> > > > +
> > > > +V4L2_META_FMT_GENERIC_CSI2_16
> > > > +-----------------------------
> > > > +
> > > > +V4L2_META_FMT_GENERIC_CSI2_16 contains packed 8-bit generic metadata, 16 bits
> > > > +for each 8 bits of data. Every byte of metadata is followed by one byte of
> > > > +padding. The way the data is stored follows the CSI-2 specification.
> > > > +
> > > > +This format is little endian.
> > > > +
> > > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_16.**
> > > > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > > > +
> > > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|
> > > > +
> > > > +.. flat-table::
> > > > +    :header-rows:  0
> > > > +    :stub-columns: 0
> > > > +    :widths: 12 8 8 8 8 8 8 8 8
> > > > +
> > > > +    * - start + 0:
> > > > +      - M\ :sub:`00`
> > > > +      - p
> > > > +      - M\ :sub:`10`
> > > > +      - p
> > > > +      - M\ :sub:`20`
> > > > +      - p
> > > > +      - M\ :sub:`30`
> > > > +      - p
> > > > +    * - start + 8:
> > > > +      - M\ :sub:`01`
> > > > +      - p
> > > > +      - M\ :sub:`11`
> > > > +      - p
> > > > +      - M\ :sub:`21`
> > > > +      - p
> > > > +      - M\ :sub:`31`
> > > > +      - p
> > > > +
> > > > +.. _v4l2-meta-fmt-generic-csi2-20:
> > > > +
> > > > +V4L2_META_FMT_GENERIC_CSI2_20
> > > > +-----------------------------
> > > > +
> > > > +V4L2_META_FMT_GENERIC_CSI2_20 contains packed 8-bit generic metadata, 20 bits
> > > > +for each 8 bits of data. Every byte of metadata is followed by alternating one
> > > > +and two bytes of padding. The way the data is stored follows the CSI-2
> > > > +specification.
> > > > +
> > > > +This format is little endian.
> > > > +
> > > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_20.**
> > > > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > > > +
> > > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> > > > +
> > > > +.. flat-table::
> > > > +    :header-rows:  0
> > > > +    :stub-columns: 0
> > > > +    :widths: 12 8 8 8 8 8 8 8 8 8 8
> > > > +
> > > > +    * - start + 0:
> > > > +      - M\ :sub:`00`
> > > > +      - p
> > > > +      - M\ :sub:`10`
> > > > +      - p
> > > > +      - p
> > > > +      - M\ :sub:`20`
> > > > +      - p
> > > > +      - M\ :sub:`30`
> > > > +      - p
> > > > +      - p
> > > > +    * - start + 10:
> > > > +      - M\ :sub:`01`
> > > > +      - p
> > > > +      - M\ :sub:`11`
> > > > +      - p
> > > > +      - p
> > > > +      - M\ :sub:`21`
> > > > +      - p
> > > > +      - M\ :sub:`31`
> > > > +      - p
> > > > +      - p
> > > > +
> > > > +.. _v4l2-meta-fmt-generic-csi2-24:
> > > > +
> > > > +V4L2_META_FMT_GENERIC_CSI2_24
> > > > +-----------------------------
> > > > +
> > > > +V4L2_META_FMT_GENERIC_CSI2_24 contains packed 8-bit generic metadata, 24 bits
> > > > +for each 8 bits of data. Every byte of metadata is followed by two bytes of
> > > > +padding. The way the data is stored follows the CSI-2 specification.
> > > > +
> > > > +This format is little endian.
> > > > +
> > > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_24.**
> > > > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > > > +
> > > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> > > > +
> > > > +.. flat-table::
> > > > +    :header-rows:  0
> > > > +    :stub-columns: 0
> > > > +    :widths: 12 8 8 8 8 8 8 8 8 8 8 8 8
> > > > +
> > > > +    * - start + 0:
> > > > +      - M\ :sub:`00`
> > > > +      - p
> > > > +      - p
> > > > +      - M\ :sub:`10`
> > > > +      - p
> > > > +      - p
> > > > +      - M\ :sub:`20`
> > > > +      - p
> > > > +      - p
> > > > +      - M\ :sub:`30`
> > > > +      - p
> > > > +      - p
> > > > +    * - start + 12:
> > > > +      - M\ :sub:`01`
> > > > +      - p
> > > > +      - p
> > > > +      - M\ :sub:`11`
> > > > +      - p
> > > > +      - p
> > > > +      - M\ :sub:`21`
> > > > +      - p
> > > > +      - p
> > > > +      - M\ :sub:`31`
> > > > +      - p
> > > > +      - p
> > > > +
> > > > +.. _v4l2-meta-fmt-generic-csi2-2-24:
> > > > +
> > > > +V4L2_META_FMT_GENERIC_CSI2_2_24
> > > > +-------------------------------
> > > > +
> > > > +V4L2_META_FMT_GENERIC_CSI2_2_24 contains packed 8-bit generic metadata, 24 bits
> > > > +for each two times 8 bits of data. Every two bytes of metadata are followed by
> > > > +one byte of padding. The way the data is stored follows the CSI-2
> > > > +specification.
> > > > +
> > > > +This format is little endian.
> > > > +
> > > > +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_2_24.**
> > > > +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> > > > +
> > > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
> > > > +
> > > > +.. flat-table::
> > > > +    :header-rows:  0
> > > > +    :stub-columns: 0
> > > > +    :widths: 12 8 8 8 8 8 8
> > > > +
> > > > +    * - start + 0:
> > > > +      - M\ :sub:`00`
> > > > +      - M\ :sub:`10`
> > > > +      - p
> > > > +      - M\ :sub:`20`
> > > > +      - M\ :sub:`30`
> > > > +      - p
> > > > +    * - start + 6:
> > > > +      - M\ :sub:`01`
> > > > +      - M\ :sub:`11`
> > > > +      - p
> > > > +      - M\ :sub:`21`
> > > > +      - M\ :sub:`31`
> > > > +      - p
> > > > +
> > > > diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> > > > index f4d9d6279094..fbbddc333a30 100644
> > > > --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> > > > +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> > > > @@ -1452,6 +1452,14 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
> > > >  	case V4L2_PIX_FMT_Y210:		descr = "10-bit YUYV Packed"; break;
> > > >  	case V4L2_PIX_FMT_Y212:		descr = "12-bit YUYV Packed"; break;
> > > >  	case V4L2_PIX_FMT_Y216:		descr = "16-bit YUYV Packed"; break;
> > > > +	case V4L2_META_FMT_GENERIC_8:	descr = "8-bit Generic Metadata"; break;
> > > > +	case V4L2_META_FMT_GENERIC_CSI2_10:	descr = "8b Generic Meta, 10b CSI-2"; break;
> > > > +	case V4L2_META_FMT_GENERIC_CSI2_12:	descr = "8b Generic Meta, 12b CSI-2"; break;
> > > > +	case V4L2_META_FMT_GENERIC_CSI2_14:	descr = "8b Generic Meta, 14b CSI-2"; break;
> > > > +	case V4L2_META_FMT_GENERIC_CSI2_16:	descr = "8b Generic Meta, 16b CSI-2"; break;
> > > > +	case V4L2_META_FMT_GENERIC_CSI2_20:	descr = "8b Generic Meta, 20b CSI-2"; break;
> > > > +	case V4L2_META_FMT_GENERIC_CSI2_24:	descr = "8b Generic Meta, 24b CSI-2"; break;
> > > > +	case V4L2_META_FMT_GENERIC_CSI2_2_24:	descr = "2x8b Generic Meta, 24b CSI-2"; break;
> > > >  
> > > >  	default:
> > > >  		/* Compressed formats */
> > > > diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> > > > index 78260e5d9985..b4284a564025 100644
> > > > --- a/include/uapi/linux/videodev2.h
> > > > +++ b/include/uapi/linux/videodev2.h
> > > > @@ -838,6 +838,15 @@ struct v4l2_pix_format {
> > > >  #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
> > > >  #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
> > > >  
> > > > +#define V4L2_META_FMT_GENERIC_8		v4l2_fourcc('M', 'E', 'T', '8') /* Generic 8-bit metadata */
> > > > +#define V4L2_META_FMT_GENERIC_CSI2_10	v4l2_fourcc('M', 'C', '1', 'A') /* 10-bit CSI-2 packed 8-bit metadata */
> > > > +#define V4L2_META_FMT_GENERIC_CSI2_12	v4l2_fourcc('M', 'C', '1', 'C') /* 12-bit CSI-2 packed 8-bit metadata */
> > > > +#define V4L2_META_FMT_GENERIC_CSI2_14	v4l2_fourcc('M', 'C', '1', 'E') /* 14-bit CSI-2 packed 8-bit metadata */
> > > > +#define V4L2_META_FMT_GENERIC_CSI2_16	v4l2_fourcc('M', 'C', '1', 'G') /* 16-bit CSI-2 packed 8-bit metadata */
> > > > +#define V4L2_META_FMT_GENERIC_CSI2_20	v4l2_fourcc('M', 'C', '1', 'K') /* 20-bit CSI-2 packed 8-bit metadata */
> > > > +#define V4L2_META_FMT_GENERIC_CSI2_24	v4l2_fourcc('M', 'C', '1', 'O') /* 24-bit CSI-2 packed 8-bit metadata */
> > > > +#define V4L2_META_FMT_GENERIC_CSI2_2_24	v4l2_fourcc('M', 'C', '2', 'O') /* 2 bytes of 8-bit metadata, 24-bit CSI-2 packed */
> > > > +
> > > >  /* priv field value to indicates that subsequent fields are valid. */
> > > >  #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
> > > >  

-- 
Regards,

Laurent Pinchart

Re: [PATCH v3 08/10] media: v4l: Support line-based metadata capture

[Sakari Ailus]

On Wed, Sep 06, 2023 at 12:24:26PM +0000, Sakari Ailus wrote:
> Hi Laurent,
> 
> On Wed, Sep 06, 2023 at 09:21:42AM +0200, Jacopo Mondi wrote:
> > Hi Laurent
> > 
> > On Tue, Sep 05, 2023 at 08:15:33PM +0300, Laurent Pinchart wrote:
> > > Hi Sakari,
> > >
> > > On Mon, Aug 14, 2023 at 11:02:40AM +0000, Sakari Ailus wrote:
> > > > On Thu, Aug 10, 2023 at 05:24:14PM +0200, Jacopo Mondi wrote:
> > > > > On Tue, Aug 08, 2023 at 10:55:36AM +0300, Sakari Ailus wrote:
> > > > > > many camera sensors, among other devices, transmit embedded data and image
> > >
> > > s/many/Many/
> > >
> > > > > > data for each CSI-2 frame. This embedded data typically contains register
> > > > > > configuration of the sensor that has been used to capture the image data
> > > > > > of the same frame.
> > > > > >
> > > > > > The embedded data is received by the CSI-2 receiver and has the same
> > > > > > properties as the image data, including that it is line based: it has
> > > > > > width, height and bytesperline (stride).
> > > > > >
> > > > > > Add these fields to struct v4l2_meta_format and document them.
> > > > > >
> > > > > > Also add V4L2_FMT_FLAG_META_LINE_BASED to tell a given format is
> > > > > > line-based i.e. these fields of struct v4l2_meta_format are valid for it.
> > > > > >
> > > > > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > > > > ---
> > > > > >  .../userspace-api/media/v4l/dev-meta.rst          | 15 +++++++++++++++
> > > > > >  .../userspace-api/media/v4l/vidioc-enum-fmt.rst   |  7 +++++++
> > > > > >  .../media/videodev2.h.rst.exceptions              |  1 +
> > > > > >  drivers/media/v4l2-core/v4l2-ioctl.c              |  5 +++--
> > > > > >  include/uapi/linux/videodev2.h                    | 10 ++++++++++
> > > > > >  5 files changed, 36 insertions(+), 2 deletions(-)
> > > > > >
> > > > > > diff --git a/Documentation/userspace-api/media/v4l/dev-meta.rst b/Documentation/userspace-api/media/v4l/dev-meta.rst
> > > > > > index 0e7e1ee1471a..4b24bae6e171 100644
> > > > > > --- a/Documentation/userspace-api/media/v4l/dev-meta.rst
> > > > > > +++ b/Documentation/userspace-api/media/v4l/dev-meta.rst
> > > > > > @@ -65,3 +65,18 @@ to 0.
> > > > > >        - ``buffersize``
> > > > > >        - Maximum buffer size in bytes required for data. The value is set by the
> > > > > >          driver.
> > > > > > +    * - __u32
> > > > > > +      - ``width``
> > > > > > +      - Width of a line of metadata in samples. Valid when :c:type`v4l2_fmtdesc`
> > > > > > +	flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
> > > > > > +	:c:func:`VIDIOC_ENUM_FMT`.
> > > > > > +    * - __u32
> > > > > > +      - ``height``
> > > > > > +      - Number of rows of metadata. Valid when :c:type`v4l2_fmtdesc` flag
> > > > > > +	``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
> > > > > > +	:c:func:`VIDIOC_ENUM_FMT`.
> > > > > > +    * - __u32
> > > > > > +      - ``bytesperline``
> > > > > > +      - Offset in bytes between the beginning of two consecutive lines. Valid
> > > > > > +	when :c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is
> > > > > > +	set, otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
> > > > > > diff --git a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> > > > > > index 000c154b0f98..6d7664345a4e 100644
> > > > > > --- a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> > > > > > +++ b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> > > > > > @@ -227,6 +227,13 @@ the ``mbus_code`` field is handled differently:
> > > > > >  	The application can ask to configure the quantization of the capture
> > > > > >  	device when calling the :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl with
> > > > > >  	:ref:`V4L2_PIX_FMT_FLAG_SET_CSC <v4l2-pix-fmt-flag-set-csc>` set.
> > > > > > +    * - ``V4L2_FMT_FLAG_META_LINE_BASED``
> > > > > > +      - 0x0200
> > > > > > +      - The metadata format is line-based. In this case the ``width``,
> > > > > > +	``height`` and ``bytesperline`` fields of :c:type:`v4l2_meta_format` are
> > > > > > +	valid. The buffer consists of ``height`` lines, each having ``width``
> > > > > > +	bytes of data and offset between the beginning of each two consecutive
> > > > >
> > > > > Isn't ``width`` in samples ?
> > > >
> > > > Indeed, it's better to refer to samples for clarity. I'll fix for v4.
> > >
> > > How do you define a "sample" in this case ? I wonder if it wouldn't be
> > > simpler for both userspace and kernel drivers if the width was specified
> > > in bytes, including the padding bytes.
> > 
> > Wouldn't this make the image line length (expressed in 'samples')
> > different than the embedded data line length ? Would this confuse
> > userspace or is it fine ?
> 
> If padding is included to width, then the user needs to calculate how many
> bytes of metadata there is, apart from the padding (which is redundant).
> That value is provided to the user for this purpose --- just like for image
> data.

Think of this: the embedded data parser is split into two, the access
function that accesses metadata bytes depending on the generic V4L2
metadata format and the rest, which cares about the specific mbus code. The
latter will need the information how many bytes of metadata there is,
instead it calculates the position in the buffer based on bytesperline,
width and the format itself.

Of course, the amount of padding can be calculated from the total amount,
by adding a second function that does it, but still it's somehow backwards
way to do that. (Makes me think of negabinaries.)

-- 
Regards,

Sakari Ailus

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

[Sakari Ailus]

Hi Laurent,

On Thu, Sep 07, 2023 at 11:47:12AM +0300, Laurent Pinchart wrote:
> > > > > +V4L2_META_FMT_GENERIC_8
> > > > > +-----------------------
> > > > > +
> > > > > +The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format.
> > > > > +
> > > > > +This format is also used on CSI-2 on both 8 bits per sample as well as on
> > > > 
> > > > s/also on/by/
> > > > 
> > > > I would also mention "MIPI CCS" instead of "CSI-2".
> > > 
> > > If CCS were to be mentioned here, then all uses of this format should be
> > > included as well.
> > > 
> > > > > +16 bits per sample when two bytes of metadata are packed into one sample.
> > > > 
> > > > "bits per sample" is very ill-defined for metadata, as there's no
> > > > sample. I would write "for both the RAW8 packing and the 2 bytes RAW16
> > > > packing" or something similar.
> > > > 
> > > > Similar comments for below.
> > > 
> > > From CSI-2 bus point of view there's no difference between pixel and
> > > embedded data when it comes to encoding that data. "Sample" is the next
> > > best term beyond "pixel", as the bus can carry samples that may or may not
> > > be pixel data. But I'm fine with changing the wording if you think it makes
> > > it more understandable.
> > 
> > I'll use "data unit" in v4.
> 
> Please define the term :-)

To the glossary, how about this:

    Data unit

	Unit of data transported by a bus. On parallel buses, this is
	called a sample while on serial buses the data unit is logical. If
	the data unit is image data, it may also be called a pixel.

-- 
Sakari Ailus

Re: [PATCH v3 01/10] media: Documentation: Align numbered list

[Sakari Ailus]

On Wed, Sep 06, 2023 at 03:50:57PM +0300, Laurent Pinchart wrote:
> On Wed, Sep 06, 2023 at 12:43:44PM +0000, Sakari Ailus wrote:
> > > > -2) Streams. Streams are declared and their routing is configured by
> > > > -setting the routing table for the sub-device using
> > > > -:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that
> > > > -setting the routing table will reset formats and selections in the
> > > > -sub-device to default values.
> > > > +2) Streams. Streams are declared and their routing is configured by setting the
> > > > +   routing table for the sub-device using :ref:`VIDIOC_SUBDEV_S_ROUTING
> > > > +   <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that setting the routing table will
> > > > +   reset formats and selections in the sub-device to default values.
> > > >  
> > > > -3) Configure formats and selections. Formats and selections of each stream
> > > > -are configured separately as documented for plain sub-devices in
> > > > -:ref:`format-propagation`. The stream ID is set to the same stream ID
> > > > -associated with either sink or source pads of routes configured using the
> > > > -:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl.
> > > > +3) Configure formats and selections. Formats and selections of each stream are
> > > > +   configured separately as documented for plain sub-devices in
> > > > +   :ref:`format-propagation`. The stream ID is set to the same stream ID
> > > > +   associated with either sink or source pads of routes configured using the
> > > > +   :ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl.
> > > >  
> > > >  Multiplexed streams setup example
> > > >  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > > @@ -619,8 +618,8 @@ modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
> > > >  To configure this pipeline, the userspace must take the following steps:
> > > >  
> > > >  1) Set up media links between entities: connect the sensors to the bridge,
> > > > -bridge to the receiver, and the receiver to the DMA engines. This step does
> > > > -not differ from normal non-multiplexed media controller setup.
> > > > +   bridge to the receiver, and the receiver to the DMA engines. This step does
> > > > +   not differ from normal non-multiplexed media controller setup.
> > > >  
> > > >  2) Configure routing
> > > >  
> > > 
> > > 3) below contains two paragraphs that are not indented.
> > 
> > From the documentation output this does not matter. Or was your comment
> > about aligning the formatting in the source? (I prefer it as-is but no
> > strong opinion about it either. Just slightly easier to edit that way. :-))
> 
> I like consistency, both in the source and the output :-)

This doesn't seem to be possible in all cases --- Sphinx thinks the
paragraph is part of the preceding table if it's indented.

-- 
Sakari Ailus

Re: [PATCH v3 04/10] media: uapi: Add generic serial metadata mbus formats

[Sakari Ailus]

Hi Laurent,

On Wed, Sep 06, 2023 at 03:50:09PM +0300, Laurent Pinchart wrote:
> Hi Sakari,
> 
> On Wed, Sep 06, 2023 at 12:34:10PM +0000, Sakari Ailus wrote:
> > On Wed, Sep 06, 2023 at 02:31:17PM +0300, Laurent Pinchart wrote:
> > > On Wed, Sep 06, 2023 at 11:28:37AM +0300, Tomi Valkeinen wrote:
> > > > On 05/09/2023 19:38, Laurent Pinchart wrote:
> > > > > On Thu, Aug 24, 2023 at 11:26:56AM +0300, Tomi Valkeinen wrote:
> > > > >> On 24/08/2023 10:24, Sakari Ailus wrote:
> > > > >>> On Wed, Aug 23, 2023 at 04:16:13PM +0300, Tomi Valkeinen wrote:
> > > > >>>> On 08/08/2023 10:55, Sakari Ailus wrote:
> > > > >>>>> Add generic serial metadata mbus formats. These formats describe data
> > > > >>>>> width and packing but not the content itself. The reason for specifying
> > > > >>>>> such formats is that the formats as such are fairly device specific but
> > > > >>>>> they are still handled by CSI-2 receiver drivers that should not be aware
> > > > > 
> > > > > Do we want these formats to be CSI-2-specific ?
> > > > >
> > > > >>>>> of device specific formats. What makes generic metadata formats possible
> > > > >>>>> is that these formats are parsed by software only, after capturing the
> > > > >>>>> data to system memory.
> > > > >>>>
> > > > >>>> If I'm not mistaken, the CSI-2 spec doesn't say much about embedded data,
> > > > >>>> except that it may exist. Afaics, in CSI-2, the embedded data is split into
> > > > >>>> "lines", although the amount of data can be different than in the video
> > > > >>>> lines.
> > > > >>>>
> > > > >>>> The CCS specs talks more about embedded data. Some of it is quite odd, like:
> > > > >>>> "The length of the embedded data line shall not exceed the length of the
> > > > >>>> image data line. The embedded data line should have the same length as the
> > > > >>>> image data line.". I think it means the embedded line can be shorter than
> > > > >>>> image line, but not longer.
> > > > >>>
> > > > >>> That's what it means, yes. The CCS also has means to obtain the actual
> > > > >>> length --- frame format descriptors.
> > > > >>>
> > > > >>>> CCS also says that an embedded line should use 0x07 as padding at the end of
> > > > >>>> the line, if there's less data than the image line.
> > > > >>>>
> > > > >>>> CCS also talks about how the embedded data would be packed, and in some
> > > > >>>> cases that packing would be the same as for pixel data.
> > > > >>>
> > > > >>> In fact the packing is the same as for pixel data: the CSI-2 does not
> > > > >>> really differentiate there.
> > > > >>
> > > > >> If I understand the CCS spec right, the packing is not the same as for
> > > > >> the pixel data. You can have RAW12 pixel data but 8-byte embedded data.
> > > > >> But if you meant that the different packing style options are the same
> > > > >> for pixel and embedded data, yes.
> > > > > 
> > > > > The idea is to allow CSI-2 receivers to unpack embedded data the same
> > > > > way as pixel data at the hardware level, and guarantee that no embedded
> > > > 
> > > > Yes, but my comment was to Sakari's "the packing is the same as for 
> > > > pixel data" comment. Afaics, there's nothing in CSI-2 or CCS that says 
> > > > the packing is the same for pixel data and embedded data.
> > > > 
> > > > In fact, the CSS says "By default, top-embedded data shall use the same 
> > > > data packing as the Visible pixel data with one embedded data byte per 
> > > > pixel", implying that it would not always be the case. It continues that 
> > > > for RAW16/RAW20/RAW24 pixels the embedded data could be sent with 
> > > > RAW8/RAW10/RAW12.
> > > 
> > > The spec if a bit ambiguous :-S "by default" implies that something else
> > > could be done, but it's not clear what the other options are.
> > 
> > The options for packing are detailed in the spec. For higher bit depths
> > there are options (as detailed by Tomi above).
> 
> The "by default" wording could be construed as allowing any packing. I
> think the intent was to only allow an alternative packing for RAW16 and
> higher, but it's not very clear.

The optimised packing convention is only available for RAW16 and higher ---
otherwise you won't have enough bits needed to carry two bytes of data. The
packing used is configured using emb_data_ctrl register.

Maybe annex A could be merged to section 7.15 in order to make things
clearer.

> 
> > > > That was just for top-embedded data, and I couldn't right away see 
> > > > mention of other embedded data.
> > > 
> > > The specification also mentions PDAF data for instance.
> > 
> > The bottom PDAF data is vendor defined (as it's processed) so the vendor is
> > free to do whatever there. Note that the better way to support PDAF is via
> > embedded PDAF pixels, not via separate bottom embedded (PDAF) data. The
> > bottom embedded data could also be different kinds of statistics.
> 
> Lots of ISPs don't support hardware processing of embedded PDAF pixels,
> so sensor-side processing is often nice to have. I wouldn't consider the
> embedded PDAF lines to be always worse than embedded PDAF pixels.

Not necessarily, indeed, but these formats tend not to be documented. :-(

-- 
Regards,

Sakari Ailus

Re: [PATCH v3 09/10] media: Add media bus codes for MIPI CCS embedded data

[Sakari Ailus]

Hi Laurent,

On Wed, Sep 06, 2023 at 04:15:10PM +0300, Laurent Pinchart wrote:
> On Wed, Sep 06, 2023 at 01:03:18PM +0000, Sakari Ailus wrote:
> > On Tue, Sep 05, 2023 at 08:25:35PM +0300, Laurent Pinchart wrote:
> > > On Tue, Aug 08, 2023 at 10:55:37AM +0300, Sakari Ailus wrote:
> > > > Add new MIPI CCS embedded data media bus formats.
> > > > 
> > > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > > ---
> > > >  .../media/v4l/subdev-formats.rst              | 32 +++++++++++++++++++
> > > >  include/uapi/linux/media-bus-format.h         | 10 +++++-
> > > >  2 files changed, 41 insertions(+), 1 deletion(-)
> > > > 
> > > > diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > > > index c615da08502d..5d5407738af9 100644
> > > > --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > > > +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > > > @@ -8491,3 +8491,35 @@ and finally the bit number in subscript. "p" indicates a padding bit.
> > > >        - p
> > > >        - p
> > > >        - p
> > > > +
> > > > +MIPI CCS Embedded Data Formats
> > > > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > > +
> > > > +`MIPI CCS <https://www.mipi.org/specifications/camera-command-set>`_ defines an
> > > 
> > > s/an$/a/
> > 
> > Yes. I think I had "embedded" there in the past...
> > 
> > > > +metadata format for sensor embedded data, which is used to store the register
> > > > +configuration used for capturing a given frame. The format is defined in the CCS
> > > > +specification.
> > > 
> > > Strictly speaking, the MIPI CCS embedded data format specifies not just
> > > the data packing (insertion of padding bytes) and the data encoding (the
> > > data format byte and the tag codes), but also the register addresses and
> > > values that are reported in the embedded data. Do you envision the media
> > > bus formats defined here as being applicable to sensors that use the
> > > same packing and encoding as CCS, but different registers, or only to
> > > fully compliant CCS sensors ?
> > 
> > There are sensors that aren't fully compatible with CCS (including those
> > compatible with SMIA and SMIA++) but I wouldn't expect the format to be
> > used by devices that are entirely incompatible with CCS.
> 
> So if a sensor uses the same packing and encoding as CCS, but a
> different register set, you would require other media bus codes ? If so,
> how would you require those media bus codes to be documented ? The
> packing and encoding could be documented as being identical to CCS, but
> what about the data itself ? Would you require the datasheet to be
> public ? Or the entire register set being documented in the V4L2 spec ?

I'd hope the registers the values of which are written in the embedded data
should be documented, either in V4L2 spec or other documentation.

But then, what do you do if you don't have documentation for all the
registers while you'd like to use what in general is needed there, exposure
time and gain? Do we say no, you can't do that because everything isn't
documented, or do we say yes, you can document this format partially.
That's what we should discuss and decide.

Note that commonly the case is that whoever would like to use the
information in the embedded data may well not have the full documentation
either. Also, even without documentation, it is in the vast majority of
the cases trivial to figure out where these values can be found.

-- 
Regards,

Sakari Ailus

Re: [PATCH v3 08/10] media: v4l: Support line-based metadata capture

[Sakari Ailus]

Hi Laurent,

On Wed, Sep 06, 2023 at 04:20:37PM +0300, Laurent Pinchart wrote:
> On Wed, Sep 06, 2023 at 12:24:26PM +0000, Sakari Ailus wrote:
> > On Wed, Sep 06, 2023 at 09:21:42AM +0200, Jacopo Mondi wrote:
> > > On Tue, Sep 05, 2023 at 08:15:33PM +0300, Laurent Pinchart wrote:
> > > > On Mon, Aug 14, 2023 at 11:02:40AM +0000, Sakari Ailus wrote:
> > > > > On Thu, Aug 10, 2023 at 05:24:14PM +0200, Jacopo Mondi wrote:
> > > > > > On Tue, Aug 08, 2023 at 10:55:36AM +0300, Sakari Ailus wrote:
> > > > > > > many camera sensors, among other devices, transmit embedded data and image
> > > >
> > > > s/many/Many/
> > > >
> > > > > > > data for each CSI-2 frame. This embedded data typically contains register
> > > > > > > configuration of the sensor that has been used to capture the image data
> > > > > > > of the same frame.
> > > > > > >
> > > > > > > The embedded data is received by the CSI-2 receiver and has the same
> > > > > > > properties as the image data, including that it is line based: it has
> > > > > > > width, height and bytesperline (stride).
> > > > > > >
> > > > > > > Add these fields to struct v4l2_meta_format and document them.
> > > > > > >
> > > > > > > Also add V4L2_FMT_FLAG_META_LINE_BASED to tell a given format is
> > > > > > > line-based i.e. these fields of struct v4l2_meta_format are valid for it.
> > > > > > >
> > > > > > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > > > > > ---
> > > > > > >  .../userspace-api/media/v4l/dev-meta.rst          | 15 +++++++++++++++
> > > > > > >  .../userspace-api/media/v4l/vidioc-enum-fmt.rst   |  7 +++++++
> > > > > > >  .../media/videodev2.h.rst.exceptions              |  1 +
> > > > > > >  drivers/media/v4l2-core/v4l2-ioctl.c              |  5 +++--
> > > > > > >  include/uapi/linux/videodev2.h                    | 10 ++++++++++
> > > > > > >  5 files changed, 36 insertions(+), 2 deletions(-)
> > > > > > >
> > > > > > > diff --git a/Documentation/userspace-api/media/v4l/dev-meta.rst b/Documentation/userspace-api/media/v4l/dev-meta.rst
> > > > > > > index 0e7e1ee1471a..4b24bae6e171 100644
> > > > > > > --- a/Documentation/userspace-api/media/v4l/dev-meta.rst
> > > > > > > +++ b/Documentation/userspace-api/media/v4l/dev-meta.rst
> > > > > > > @@ -65,3 +65,18 @@ to 0.
> > > > > > >        - ``buffersize``
> > > > > > >        - Maximum buffer size in bytes required for data. The value is set by the
> > > > > > >          driver.
> > > > > > > +    * - __u32
> > > > > > > +      - ``width``
> > > > > > > +      - Width of a line of metadata in samples. Valid when :c:type`v4l2_fmtdesc`
> > > > > > > +	flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
> > > > > > > +	:c:func:`VIDIOC_ENUM_FMT`.
> > > > > > > +    * - __u32
> > > > > > > +      - ``height``
> > > > > > > +      - Number of rows of metadata. Valid when :c:type`v4l2_fmtdesc` flag
> > > > > > > +	``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
> > > > > > > +	:c:func:`VIDIOC_ENUM_FMT`.
> > > > > > > +    * - __u32
> > > > > > > +      - ``bytesperline``
> > > > > > > +      - Offset in bytes between the beginning of two consecutive lines. Valid
> > > > > > > +	when :c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is
> > > > > > > +	set, otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
> > > > > > > diff --git a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> > > > > > > index 000c154b0f98..6d7664345a4e 100644
> > > > > > > --- a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> > > > > > > +++ b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> > > > > > > @@ -227,6 +227,13 @@ the ``mbus_code`` field is handled differently:
> > > > > > >  	The application can ask to configure the quantization of the capture
> > > > > > >  	device when calling the :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl with
> > > > > > >  	:ref:`V4L2_PIX_FMT_FLAG_SET_CSC <v4l2-pix-fmt-flag-set-csc>` set.
> > > > > > > +    * - ``V4L2_FMT_FLAG_META_LINE_BASED``
> > > > > > > +      - 0x0200
> > > > > > > +      - The metadata format is line-based. In this case the ``width``,
> > > > > > > +	``height`` and ``bytesperline`` fields of :c:type:`v4l2_meta_format` are
> > > > > > > +	valid. The buffer consists of ``height`` lines, each having ``width``
> > > > > > > +	bytes of data and offset between the beginning of each two consecutive
> > > > > >
> > > > > > Isn't ``width`` in samples ?
> > > > >
> > > > > Indeed, it's better to refer to samples for clarity. I'll fix for v4.
> > > >
> > > > How do you define a "sample" in this case ? I wonder if it wouldn't be
> > > > simpler for both userspace and kernel drivers if the width was specified
> > > > in bytes, including the padding bytes.
> > > 
> > > Wouldn't this make the image line length (expressed in 'samples')
> > > different than the embedded data line length ? Would this confuse
> > > userspace or is it fine ?
> > 
> > If padding is included to width, then the user needs to calculate how many
> > bytes of metadata there is, apart from the padding (which is redundant).
> > That value is provided to the user for this purpose --- just like for image
> > data.
> 
> The bytesperline value includes padding at the end of the line to
> achieve a particular stride, so that doesn't tell how many bytes to
> parse. If the width is specified in "samples", then the parser will need
> to compute the number of bytes it spans, and then parse those bytes.

The parser is interested in metadata bytes only, not the padding. The
padding should be skipped by the data access function in order to avoid
complicating the parser with different padding options.

That's why width should be in data units (bytes of metadata without
padding).

-- 
Regards,

Sakari Ailus

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

[Sakari Ailus]

Hi Laurent,

On Wed, Sep 06, 2023 at 04:07:29PM +0300, Laurent Pinchart wrote:
> On Wed, Sep 06, 2023 at 11:56:47AM +0000, Sakari Ailus wrote:
> > Hi Laurent,
> > 
> > On Tue, Sep 05, 2023 at 07:55:58PM +0300, Laurent Pinchart wrote:
> > > On Tue, Aug 08, 2023 at 10:55:35AM +0300, Sakari Ailus wrote:
> > > > Generic 8-bit metadata formats define the in-memory data layout but not
> > > > the format of the data itself. The reasoning for having such formats is to
> > > > allow CSI-2 receiver drivers to receive and DMA drivers to write the data
> > > > to memory without knowing a large number of device specific formats.
> > > > 
> > > > These formats may be used only in conjunction of a Media controller
> > > > pipeline where the internal pad of the source sub-device defines the
> > > > specific format of the data (using an mbus code).
> > > > 
> > > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > > ---
> > > >  .../userspace-api/media/v4l/meta-formats.rst  |   1 +
> > > >  .../media/v4l/metafmt-generic.rst             | 331 ++++++++++++++++++
> > > >  drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
> > > >  include/uapi/linux/videodev2.h                |   9 +
> > > >  4 files changed, 349 insertions(+)
> > > >  create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > > > 
> > > > diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
> > > > index 0bb61fc5bc00..919f595576b9 100644
> > > > --- a/Documentation/userspace-api/media/v4l/meta-formats.rst
> > > > +++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
> > > > @@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only.
> > > >      metafmt-vsp1-hgo
> > > >      metafmt-vsp1-hgt
> > > >      metafmt-vivid
> > > > +    metafmt-generic
> > > > diff --git a/Documentation/userspace-api/media/v4l/metafmt-generic.rst b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > > > new file mode 100644
> > > > index 000000000000..a27bfc721edf
> > > > --- /dev/null
> > > > +++ b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > > > @@ -0,0 +1,331 @@
> > > > +.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
> > > > +
> > > > +**************************************************************************************************************************************************************************************************************************************************************************************************************************
> > > > +V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O'), V4L2_META_FMT_GENERIC_CSI2_2_24 ('MC2O')
> > > > +**************************************************************************************************************************************************************************************************************************************************************************************************************************
> > > > +
> > > > +
> > > > +Generic line-based metadata formats
> > > > +
> > > > +
> > > > +Description
> > > > +===========
> > > > +
> > > > +These generic line-based metadata formats define the memory layout of the data
> > > > +without defining the format or meaning of the metadata itself. These formats may
> > > > +only be used with a Media controller pipeline where the more specific format is
> > > > +defined in an :ref:`internal source pad <MEDIA-PAD-FL-INTERNAL>` of the source
> > > > +sub-device. See also :ref:`source routes <v4l2-subdev-source-routes>`.
> > > > +
> > > > +.. _v4l2-meta-fmt-generic-8:
> > > > +
> > > > +V4L2_META_FMT_GENERIC_8
> > > > +-----------------------
> > > > +
> > > > +The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format.
> > > > +
> > > > +This format is also used on CSI-2 on both 8 bits per sample as well as on
> > > 
> > > s/also on/by/
> > > 
> > > I would also mention "MIPI CCS" instead of "CSI-2".
> > 
> > If CCS were to be mentioned here, then all uses of this format should be
> > included as well.
> > 
> > > > +16 bits per sample when two bytes of metadata are packed into one sample.
> > > 
> > > "bits per sample" is very ill-defined for metadata, as there's no
> > > sample. I would write "for both the RAW8 packing and the 2 bytes RAW16
> > > packing" or something similar.
> > > 
> > > Similar comments for below.
> > 
> > From CSI-2 bus point of view there's no difference between pixel and
> > embedded data when it comes to encoding that data. "Sample" is the next
> > best term beyond "pixel", as the bus can carry samples that may or may not
> > be pixel data. But I'm fine with changing the wording if you think it makes
> > it more understandable.
> 
> The CSI-2 specification doesn't seem to define any "sample" concept.
> 
> > > > +
> > > > +**Byte Order Of V4L2_META_FMT_GENERIC_8.**
> > > > +Each cell is one byte. "M" denotes a byte of metadata.
> > > > +
> > > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|
> > > > +
> > > > +.. flat-table::
> > > > +    :header-rows:  0
> > > > +    :stub-columns: 0
> > > > +    :widths: 12 8 8 8 8
> > > > +
> > > > +    * - start + 0:
> > > > +      - M\ :sub:`00`
> > > > +      - M\ :sub:`10`
> > > > +      - M\ :sub:`20`
> > > > +      - M\ :sub:`30`
> > > > +    * - start + 4:
> > > > +      - M\ :sub:`01`
> > > > +      - M\ :sub:`11`
> > > > +      - M\ :sub:`21`
> > > > +      - M\ :sub:`31`
> > > > +
> > > > +.. _v4l2-meta-fmt-generic-csi2-10:
> > > > +
> > > > +V4L2_META_FMT_GENERIC_CSI2_10
> > > > +-----------------------------
> > > > +
> > > > +V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
> > > > +for each 8 bits of data. Every four bytes of metadata is followed by a single
> > > > +byte of padding.
> > > 
> > > It sounds really weird to write that this format writes 10 bits for each
> > > 8 bits of data, when essentially it adds a packing byte every four
> > > bytes.
> > 
> > That's how the hardware has been implemented and probably there is a
> > hardware implementation related reason for this.
> 
> I can imagine the specification has been designed to make it possible to
> push embedded data and pixel data through the same serialization
> hardware, but I can't tell if that's how hardware has been implemented.
> Still, from an application point of view, when documenting pixel
> formats, it sounds confusing. Can't we instead say that there's a 0x55
> padding after every four bytes of data ?

What if we get an implementation that does not use 0x55 value for padding?

It'd be safer not to define the value of the padding byte, it doesn't
matter from parsing point in any case.

-- 
Sakari Ailus

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

[Laurent Pinchart]

On Fri, Sep 22, 2023 at 08:50:11AM +0000, Sakari Ailus wrote:
> On Wed, Sep 06, 2023 at 04:07:29PM +0300, Laurent Pinchart wrote:
> > On Wed, Sep 06, 2023 at 11:56:47AM +0000, Sakari Ailus wrote:
> > > On Tue, Sep 05, 2023 at 07:55:58PM +0300, Laurent Pinchart wrote:
> > > > On Tue, Aug 08, 2023 at 10:55:35AM +0300, Sakari Ailus wrote:
> > > > > Generic 8-bit metadata formats define the in-memory data layout but not
> > > > > the format of the data itself. The reasoning for having such formats is to
> > > > > allow CSI-2 receiver drivers to receive and DMA drivers to write the data
> > > > > to memory without knowing a large number of device specific formats.
> > > > > 
> > > > > These formats may be used only in conjunction of a Media controller
> > > > > pipeline where the internal pad of the source sub-device defines the
> > > > > specific format of the data (using an mbus code).
> > > > > 
> > > > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > > > ---
> > > > >  .../userspace-api/media/v4l/meta-formats.rst  |   1 +
> > > > >  .../media/v4l/metafmt-generic.rst             | 331 ++++++++++++++++++
> > > > >  drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
> > > > >  include/uapi/linux/videodev2.h                |   9 +
> > > > >  4 files changed, 349 insertions(+)
> > > > >  create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > > > > 
> > > > > diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
> > > > > index 0bb61fc5bc00..919f595576b9 100644
> > > > > --- a/Documentation/userspace-api/media/v4l/meta-formats.rst
> > > > > +++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
> > > > > @@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only.
> > > > >      metafmt-vsp1-hgo
> > > > >      metafmt-vsp1-hgt
> > > > >      metafmt-vivid
> > > > > +    metafmt-generic
> > > > > diff --git a/Documentation/userspace-api/media/v4l/metafmt-generic.rst b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > > > > new file mode 100644
> > > > > index 000000000000..a27bfc721edf
> > > > > --- /dev/null
> > > > > +++ b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > > > > @@ -0,0 +1,331 @@
> > > > > +.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
> > > > > +
> > > > > +**************************************************************************************************************************************************************************************************************************************************************************************************************************
> > > > > +V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O'), V4L2_META_FMT_GENERIC_CSI2_2_24 ('MC2O')
> > > > > +**************************************************************************************************************************************************************************************************************************************************************************************************************************
> > > > > +
> > > > > +
> > > > > +Generic line-based metadata formats
> > > > > +
> > > > > +
> > > > > +Description
> > > > > +===========
> > > > > +
> > > > > +These generic line-based metadata formats define the memory layout of the data
> > > > > +without defining the format or meaning of the metadata itself. These formats may
> > > > > +only be used with a Media controller pipeline where the more specific format is
> > > > > +defined in an :ref:`internal source pad <MEDIA-PAD-FL-INTERNAL>` of the source
> > > > > +sub-device. See also :ref:`source routes <v4l2-subdev-source-routes>`.
> > > > > +
> > > > > +.. _v4l2-meta-fmt-generic-8:
> > > > > +
> > > > > +V4L2_META_FMT_GENERIC_8
> > > > > +-----------------------
> > > > > +
> > > > > +The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format.
> > > > > +
> > > > > +This format is also used on CSI-2 on both 8 bits per sample as well as on
> > > > 
> > > > s/also on/by/
> > > > 
> > > > I would also mention "MIPI CCS" instead of "CSI-2".
> > > 
> > > If CCS were to be mentioned here, then all uses of this format should be
> > > included as well.
> > > 
> > > > > +16 bits per sample when two bytes of metadata are packed into one sample.
> > > > 
> > > > "bits per sample" is very ill-defined for metadata, as there's no
> > > > sample. I would write "for both the RAW8 packing and the 2 bytes RAW16
> > > > packing" or something similar.
> > > > 
> > > > Similar comments for below.
> > > 
> > > From CSI-2 bus point of view there's no difference between pixel and
> > > embedded data when it comes to encoding that data. "Sample" is the next
> > > best term beyond "pixel", as the bus can carry samples that may or may not
> > > be pixel data. But I'm fine with changing the wording if you think it makes
> > > it more understandable.
> > 
> > The CSI-2 specification doesn't seem to define any "sample" concept.
> > 
> > > > > +
> > > > > +**Byte Order Of V4L2_META_FMT_GENERIC_8.**
> > > > > +Each cell is one byte. "M" denotes a byte of metadata.
> > > > > +
> > > > > +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|
> > > > > +
> > > > > +.. flat-table::
> > > > > +    :header-rows:  0
> > > > > +    :stub-columns: 0
> > > > > +    :widths: 12 8 8 8 8
> > > > > +
> > > > > +    * - start + 0:
> > > > > +      - M\ :sub:`00`
> > > > > +      - M\ :sub:`10`
> > > > > +      - M\ :sub:`20`
> > > > > +      - M\ :sub:`30`
> > > > > +    * - start + 4:
> > > > > +      - M\ :sub:`01`
> > > > > +      - M\ :sub:`11`
> > > > > +      - M\ :sub:`21`
> > > > > +      - M\ :sub:`31`
> > > > > +
> > > > > +.. _v4l2-meta-fmt-generic-csi2-10:
> > > > > +
> > > > > +V4L2_META_FMT_GENERIC_CSI2_10
> > > > > +-----------------------------
> > > > > +
> > > > > +V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
> > > > > +for each 8 bits of data. Every four bytes of metadata is followed by a single
> > > > > +byte of padding.
> > > > 
> > > > It sounds really weird to write that this format writes 10 bits for each
> > > > 8 bits of data, when essentially it adds a packing byte every four
> > > > bytes.
> > > 
> > > That's how the hardware has been implemented and probably there is a
> > > hardware implementation related reason for this.
> > 
> > I can imagine the specification has been designed to make it possible to
> > push embedded data and pixel data through the same serialization
> > hardware, but I can't tell if that's how hardware has been implemented.
> > Still, from an application point of view, when documenting pixel
> > formats, it sounds confusing. Can't we instead say that there's a 0x55
> > padding after every four bytes of data ?
> 
> What if we get an implementation that does not use 0x55 value for padding?
> 
> It'd be safer not to define the value of the padding byte, it doesn't
> matter from parsing point in any case.

I suppose that would be OK. The parser will likely be sensor-specific
anyway, so userspace will know what value to expect for the padding
byte, even if not conveyed by the format definition.

-- 
Regards,

Laurent Pinchart