[RFC 0/7] Generic line based metadata support, internal pads

[RFC 0/7] Generic line based metadata support, internal pads

[Sakari Ailus]

Hi folks,

Here are a few patches to add support generic, line based metadata as well
as internal source pads. While the amount of code is not very large, to
the contrary it is quite small actually IMO, I presume what this is about
and why it is being proposed requires some explaining.

Metadata mbus codes and formats have existed for some time in V4L2. They
however have been only used by drivers that produce the data itself and
effectively this metadata has always been statistics of some sort (at
least when it comes to ISPs). What is different here is that we intend to
add support for metadata originating from camera sensors.

Camera sensors produce different kinds of metadata, embedded data (usually
register address--value pairs used to capture the frame, in a more or less
sensor specific format), histograms (in a very sensor specific format),
dark pixels etc. The number of these formats is probably going to be about
as large as image data formats if not larger, as the image data formats
are much better standardised but a smaller subset of them will be
supported by V4L2, at least initially but possibly much more in the long
run.

Having this many device specific formats would be a major problem for all
the other drivers along that pipeline (not to mention the users of those
drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2
receiver drivers that have DMA: the poor driver developer would not only
need to know camera sensor specific formats but to choose the specific
packing of that format suitable for the DMA used by the hardware. It is
unlikely many of these would ever get tested while being present on the
driver API. Also adding new sensors with new embedded data formats would
involve updating all bridge and CSI-2 receiver drivers. I don't expect
this to be a workable approach.

Instead what I'm proposing is to use specific metadata formats on the
sensor devices only, on internal pads (more about those soon) of the
sensors, only visible in the UAPI, and then generic mbus formats along the
pipeline and finally generic V4L2 metadata formats on the DMAs (specific
to bit depth and packing). This would unsnarl the two, defining what data
there is (specific mbus code) and how that is transported and packed
(generic mbus codes and V4L2 formats).

The user space would be required to "know" the path of that data from the
sensor's internal pad to the V4L2 video node. I do not see this as these
devices require at least some knowledge of the pipeline, i.e. hardware at
hand. Separating what the data means and how it is packed may even be
beneficial: it allows separating code that interprets the data (sensor
internal mbus code) from the code that accesses it (packing).

These formats are in practice line based, meaning that there may be
padding at the end of the line, depending on the bus as well as the DMA.
If non-line based formats are needed, it is always possible to set the
"height" field to 1.

The internal (source) pads are an alternative to source routes [1]. The
source routes were not universally liked and I do have to say I like
re-using existing interface concepts (pads and everything you can do with
pads, including access format, selections etc.) wherever it makes sense,
instead of duplicating functionality.

Effectively internal source pads behave mostly just like sink pads, but
they describe a flow of data that originates from a sub-device instead of
arriving to a sub-device. The SUBDEV_S_ROUTING IOCTLs are used to enable
and disable routes from internal source pads to sub-device's source pads.
The subdev format IOCTLs are usable, too, so one can find which subdev
format is available on given internal source pad.

This set depends on these patches:

<URL:https://lore.kernel.org/linux-media/20230505205416.55002-1-sakari.ailus@linux.intel.com/T/#t>

I've also pushed these here and I'll keep updating the branch:

<URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=metadata>

Questions and comments are most welcome.


[1] <URL:https://lore.kernel.org/linux-media/20220831141357.1396081-20-tomi.valkeinen@ideasonboard.com/>

Sakari Ailus (7):
  media: mc: Add INTERNAL_SOURCE pad type flag
  media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs
  media: uapi: v4l: Document source routes
  media: mc: Check pad flag validity
  media: uapi: Add generic serial metadata mbus formats
  media: uapi: Add generic 8-bit metadata format definitions
  media: v4l: Support line-based metadata capture

 .../media/mediactl/media-types.rst            |   7 +
 .../userspace-api/media/v4l/dev-meta.rst      |  15 +
 .../userspace-api/media/v4l/dev-subdev.rst    |  18 +
 .../userspace-api/media/v4l/meta-formats.rst  |   1 +
 .../media/v4l/metafmt-generic.rst             | 317 ++++++++++++++++++
 .../media/v4l/subdev-formats.rst              | 257 ++++++++++++++
 .../media/v4l/vidioc-enum-fmt.rst             |   7 +
 .../media/v4l/vidioc-subdev-g-routing.rst     |   5 +
 drivers/media/mc/mc-entity.c                  |  20 +-
 drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
 drivers/media/v4l2-core/v4l2-subdev.c         |   6 +-
 include/uapi/linux/media-bus-format.h         |   9 +
 include/uapi/linux/media.h                    |   1 +
 include/uapi/linux/v4l2-subdev.h              |   6 +-
 include/uapi/linux/videodev2.h                |  19 ++
 15 files changed, 691 insertions(+), 5 deletions(-)
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst

-- 
2.30.2

[RFC 1/7] media: mc: Add INTERNAL_SOURCE pad type flag

[Sakari Ailus]

Internal source pads will be used as routing endpoints in V4L2
[GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.

Also prevent creating links to pads that have been flagged as internal.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../userspace-api/media/mediactl/media-types.rst          | 7 +++++++
 drivers/media/mc/mc-entity.c                              | 8 +++++++-
 include/uapi/linux/media.h                                | 1 +
 3 files changed, 15 insertions(+), 1 deletion(-)

diff --git a/Documentation/userspace-api/media/mediactl/media-types.rst b/Documentation/userspace-api/media/mediactl/media-types.rst
index 0ffeece1e0c8..c724139ad46c 100644
--- a/Documentation/userspace-api/media/mediactl/media-types.rst
+++ b/Documentation/userspace-api/media/mediactl/media-types.rst
@@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
 .. _MEDIA-PAD-FL-SINK:
 .. _MEDIA-PAD-FL-SOURCE:
 .. _MEDIA-PAD-FL-MUST-CONNECT:
+.. _MEDIA-PAD-FL-INTERNAL-SOURCE:
 
 .. flat-table:: Media pad flags
     :header-rows:  0
@@ -382,6 +383,12 @@ Types and flags used to represent the media graph elements
 	  when this flag isn't set; the absence of the flag doesn't imply
 	  there is none.
 
+    *  -  ``MEDIA_PAD_FL_INTERNAL_SOURCE``
+       -  This flag indicates an internal pad that has no external
+	  connections. Such a pad may not be connected with a link. The internal
+	  flag indicates that the stream either starts or ends in the
+	  entity. For a given pad, the INTERNAL_SOURCE flag may not be set if
+	  either SINK or SOURCE flags is set.
 
 One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
 must be set for every pad.
diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index ef34ddd715c9..ed99160a2487 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -1062,7 +1062,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
 
 	for (i = 0; i < entity->num_pads; i++) {
 		if ((entity->pads[i].flags &
-		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
+		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
+		      MEDIA_PAD_FL_INTERNAL_SOURCE)) != pad_type)
 			continue;
 
 		if (entity->pads[i].sig_type == sig_type)
@@ -1087,6 +1088,11 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
 		return -EINVAL;
 	if (WARN_ON(!(sink->pads[sink_pad].flags & MEDIA_PAD_FL_SINK)))
 		return -EINVAL;
+	if (WARN_ON(sink->pads[sink_pad].flags &
+		    MEDIA_PAD_FL_INTERNAL_SOURCE) ||
+	    WARN_ON(source->pads[source_pad].flags &
+		    MEDIA_PAD_FL_INTERNAL_SOURCE))
+		return -EINVAL;
 
 	link = media_add_link(&source->links);
 	if (link == NULL)
diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
index edb8dfef9eba..0e2577e8b425 100644
--- a/include/uapi/linux/media.h
+++ b/include/uapi/linux/media.h
@@ -208,6 +208,7 @@ struct media_entity_desc {
 #define MEDIA_PAD_FL_SINK			(1U << 0)
 #define MEDIA_PAD_FL_SOURCE			(1U << 1)
 #define MEDIA_PAD_FL_MUST_CONNECT		(1U << 2)
+#define MEDIA_PAD_FL_INTERNAL_SOURCE		(1U << 3)
 
 struct media_pad_desc {
 	__u32 entity;		/* entity ID */
-- 
2.30.2

[RFC 2/7] media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs

[Sakari Ailus]

Take the new INTERNAL_SOURCE pad flag into account in validating routing
IOCTL argument. Effectively this is a SINK pad in this respect. Due to the
union there's no need to check for the particular name.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 2ec179cd1264..6312fc2bf1f0 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -906,7 +906,8 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 				return -EINVAL;
 
 			if (!(pads[route->sink_pad].flags &
-			      MEDIA_PAD_FL_SINK))
+			      (MEDIA_PAD_FL_SINK |
+			       MEDIA_PAD_FL_INTERNAL_SOURCE)))
 				return -EINVAL;
 
 			if (route->source_pad >= sd->entity.num_pads)
@@ -1787,7 +1788,8 @@ int v4l2_subdev_routing_validate(struct v4l2_subdev *sd,
 
 		/* Validate the sink and source pad numbers. */
 		if (route->sink_pad >= sd->entity.num_pads ||
-		    !(sd->entity.pads[route->sink_pad].flags & MEDIA_PAD_FL_SINK)) {
+		    !(sd->entity.pads[route->sink_pad].flags &
+		      (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL_SOURCE))) {
 			dev_dbg(sd->dev, "route %u sink (%u) is not a sink pad\n",
 				i, route->sink_pad);
 			goto out;
-- 
2.30.2

[RFC 3/7] media: uapi: v4l: Document source routes

[Sakari Ailus]

Document how internal pads are used on source routes.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../userspace-api/media/v4l/dev-subdev.rst     | 18 ++++++++++++++++++
 .../media/v4l/vidioc-subdev-g-routing.rst      |  5 +++++
 include/uapi/linux/v4l2-subdev.h               |  6 +++++-
 3 files changed, 28 insertions(+), 1 deletion(-)

diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index a4f1df7093e8..395e06d2f0f2 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -551,6 +551,24 @@ A stream at a specific point in the media pipeline is identified by the
 sub-device and a (pad, stream) pair. For sub-devices that do not support
 multiplexed streams the 'stream' field is always 0.
 
+.. _v4l2-subdev-source-routes:
+
+Source routes
+^^^^^^^^^^^^^
+
+Cases where a single sub-device pad is a source of multiple streams are special
+as there is no sink pad for such a route. In those cases, an internal pad is
+used as the sink pad. Such pads have the :ref:`MEDIA_PAD_FL_INTERNAL_SOURCE
+<MEDIA-PAD-FL-INTERNAL-SOURCE>` flag set.
+
+Internal pads have all the properties of a sink pad in such case, including
+formats and selections. The format in this case is the source format of the
+stream. An internal pad always has a single stream only (0).
+
+Generally source routes are not modifiable but they can be activated and
+inactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE
+<v4l2-subdev-routing-flags>` flag, depending on driver capabilities.
+
 Interaction between routes, streams, formats and selections
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
index 68ca343c3b44..e00622071b64 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
@@ -94,6 +94,11 @@ for all the route entries and call ``VIDIOC_SUBDEV_G_ROUTING`` again.
     * - __u32
       - ``sink_pad``
       - Sink pad number.
+    * - __u32
+      - ``internal_source_pad``
+      - Internal source pad number. For pads with :ref:`internal source pad flag
+	<MEDIA-PAD-FL-INTERNAL-SOURCE>`, for use with :ref:`source routes
+	<v4l2-subdev-source-routes>`.
     * - __u32
       - ``sink_stream``
       - Sink pad stream number.
diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
index 4a195b68f28f..703e3a1f199b 100644
--- a/include/uapi/linux/v4l2-subdev.h
+++ b/include/uapi/linux/v4l2-subdev.h
@@ -203,6 +203,7 @@ struct v4l2_subdev_capability {
  * struct v4l2_subdev_route - A route inside a subdev
  *
  * @sink_pad: the sink pad index
+ * @internal_source_pad: the internal source pad
  * @sink_stream: the sink stream identifier
  * @source_pad: the source pad index
  * @source_stream: the source stream identifier
@@ -210,7 +211,10 @@ struct v4l2_subdev_capability {
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_route {
-	__u32 sink_pad;
+	union {
+		__u32 sink_pad;
+		__u32 internal_source_pad;
+	};
 	__u32 sink_stream;
 	__u32 source_pad;
 	__u32 source_stream;
-- 
2.30.2

[RFC 4/7] media: mc: Check pad flag validity

[Sakari Ailus]

Check the validity of pad flags on entity init. Exactly one of the flags
must be set.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 drivers/media/mc/mc-entity.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index ed99160a2487..c1bc48c4d239 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -197,6 +197,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 	struct media_device *mdev = entity->graph_obj.mdev;
 	struct media_pad *iter;
 	unsigned int i = 0;
+	int ret = 0;
 
 	if (num_pads >= MEDIA_ENTITY_MAX_PADS)
 		return -E2BIG;
@@ -210,6 +211,15 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 	media_entity_for_each_pad(entity, iter) {
 		iter->entity = entity;
 		iter->index = i++;
+
+		if (hweight32(iter->flags & (MEDIA_PAD_FL_SINK |
+					     MEDIA_PAD_FL_SOURCE |
+					     MEDIA_PAD_FL_INTERNAL_SOURCE))
+		    != 1) {
+			ret = -EINVAL;
+			break;
+		}
+
 		if (mdev)
 			media_gobj_create(mdev, MEDIA_GRAPH_PAD,
 					  &iter->graph_obj);
@@ -218,7 +228,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 	if (mdev)
 		mutex_unlock(&mdev->graph_mutex);
 
-	return 0;
+	return ret;
 }
 EXPORT_SYMBOL_GPL(media_entity_pads_init);
 
-- 
2.30.2

[RFC 5/7] media: uapi: Add generic serial metadata mbus formats

[Sakari Ailus]

Add generic serial metadata mbus formats. These formats describe data
width and packing but not the content itself. The reason for specifying
such formats is that the formats as such are fairly device specific but
they are still handled by CSI-2 receiver drivers that should not be aware
of device specific formats. What makes generic metadata formats possible
is that these formats are parsed by software only, after capturing the
data to system memory.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../media/v4l/subdev-formats.rst              | 257 ++++++++++++++++++
 include/uapi/linux/media-bus-format.h         |   9 +
 2 files changed, 266 insertions(+)

diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
index a3a35eeed708..1492fff58426 100644
--- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
+++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
@@ -8234,3 +8234,260 @@ The following table lists the existing metadata formats.
 	both sides of the link and the bus format is a fixed
 	metadata format that is not configurable from userspace.
 	Width and height will be set to 0 for this format.
+
+Generic Serial Metadata Formats
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Generic serial metadata formats are used on serial busses where the actual data
+content is more or less device specific but the data is transmitted and received
+by multiple devices that do not process the data in any way, simply writing
+it to system memory for processing in software at the end of the pipeline.
+
+The more specific variant describing the actual data is used on the internal
+source pad of the originating sub-device.
+
+"b" in an array cell signifies a byte of data, followed by the number of byte
+and finally the bit number in subscript. "p" indicates a padding bit.
+
+.. _media-bus-format-generic-meta:
+
+.. cssclass: longtable
+
+.. flat-table:: Generic Serial Metadata Formats
+    :header-rows:  2
+    :stub-columns: 0
+
+    * - Identifier
+      - Code
+      -
+      - :cspan:`23` Data organization
+    * -
+      -
+      - Bit
+      - 23
+      - 22
+      - 21
+      - 20
+      - 19
+      - 18
+      - 17
+      - 16
+      - 15
+      - 14
+      - 13
+      - 12
+      - 11
+      - 10
+      - 9
+      - 8
+      - 7
+      - 6
+      - 5
+      - 4
+      - 3
+      - 2
+      - 1
+      - 0
+    * .. _MEDIA-BUS-FMT-META-1X8-8:
+
+      - MEDIA_BUS_FMT_META_1X8_8
+      - 0x8001
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+    * .. _MEDIA-BUS-FMT-META-1X8-10:
+
+      - MEDIA_BUS_FMT_META_1X8_10
+      - 0x8002
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-1X8-12:
+
+      - MEDIA_BUS_FMT_META_1X8_12
+      - 0x8003
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-1X8-14:
+
+      - MEDIA_BUS_FMT_META_1X8_14
+      - 0x8004
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-1X8-16:
+
+      - MEDIA_BUS_FMT_META_1X8_16
+      - 0x8005
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-1X8-20:
+
+      - MEDIA_BUS_FMT_META_1X8_20
+      - 0x8007
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-1X8-24:
+
+      - MEDIA_BUS_FMT_META_1X8_24
+      - 0x8009
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
index a03c543cb072..722463523bbd 100644
--- a/include/uapi/linux/media-bus-format.h
+++ b/include/uapi/linux/media-bus-format.h
@@ -173,4 +173,13 @@
  */
 #define MEDIA_BUS_FMT_METADATA_FIXED		0x7001
 
+/* Generic line based metadata formats for serial buses. Next is 0x800b. */
+#define MEDIA_BUS_FMT_META_1X8_8		0x8001
+#define MEDIA_BUS_FMT_META_1X8_10		0x8002
+#define MEDIA_BUS_FMT_META_1X8_12		0x8003
+#define MEDIA_BUS_FMT_META_1X8_14		0x8004
+#define MEDIA_BUS_FMT_META_1X8_16		0x8005
+#define MEDIA_BUS_FMT_META_1X8_20		0x8007
+#define MEDIA_BUS_FMT_META_1X8_24		0x8009
+
 #endif /* __LINUX_MEDIA_BUS_FORMAT_H */
-- 
2.30.2

[RFC 6/7] media: uapi: Add generic 8-bit metadata format definitions

[Sakari Ailus]

Generic 8-bit metadata formats define the in-memory data layout but not
the format of the data itself. The reasoning for having such formats is to
allow CSI-2 receiver drivers to receive and DMA drivers to write the data
to memory without knowing a large number of device specific formats.

These formats may be used only in conjunction of a Media controller
pipeline where the internal pad of the source sub-device defines the
specific format of the data (using an mbus code).

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../userspace-api/media/v4l/meta-formats.rst  |   1 +
 .../media/v4l/metafmt-generic.rst             | 317 ++++++++++++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
 include/uapi/linux/videodev2.h                |   9 +
 4 files changed, 335 insertions(+)
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst

diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
index 0bb61fc5bc00..919f595576b9 100644
--- a/Documentation/userspace-api/media/v4l/meta-formats.rst
+++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
@@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only.
     metafmt-vsp1-hgo
     metafmt-vsp1-hgt
     metafmt-vivid
+    metafmt-generic
diff --git a/Documentation/userspace-api/media/v4l/metafmt-generic.rst b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
new file mode 100644
index 000000000000..09d3fabb1e94
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
@@ -0,0 +1,317 @@
+.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
+
+**************************************************************************************************************************************************************************************************************************************************************************************************************************
+V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O'), V4L2_META_FMT_GENERIC_CSI2_2_24 ('MC2O')
+**************************************************************************************************************************************************************************************************************************************************************************************************************************
+
+
+Generic line-based metadata formats
+
+
+Description
+===========
+
+These generic line-based metadata formats define the memory layout of the data
+without defining the format or meaning of the metadata itself. These formats may
+only be used with a Media controller pipeline where the more specific format is
+defined in an :ref:`internal source pad <MEDIA-PAD-FL-INTERNAL-SOURCE>` of the
+source sub-device. See also :ref:`source routes <v4l2-subdev-source-routes>`.
+
+.. _v4l2-meta-fmt-generic-8:
+
+V4L2_META_FMT_GENERIC_8
+-----------------------
+
+The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format whereas the
+rest of the formats have CSI-2 packing that are typically used on camera sensors
+in conjunction with pixel data of the same bit depth.
+
+This format is also used on CSI-2 on both 8 bits per sample as well as on
+16 bits per sample when two bytes of metadata are packed into one sample.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_8.**
+Each cell is one byte. "M" denotes a byte of metadata.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+    * - start + 4:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+
+.. _v4l2-meta-fmt-generic-csi2-10:
+
+V4L2_META_FMT_GENERIC_CSI2_10
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_10 contains 8-bit generic metadata with CSI-2
+packing, 10 bits for each 8 bits of data. Every four bytes of metadata is
+followed by a single byte of padding.
+
+This format is also used on CSI2 on 20 bits per sample format that packs two
+bytes of metadata into one sample.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+    * - start + 5:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-12:
+
+V4L2_META_FMT_GENERIC_CSI2_12
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_12 contains 8-bit generic metadata with CSI-2
+packing, 12 bits for each 8 bits of data. Every four bytes of metadata is
+followed by two bytes of padding.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_12.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+      - p
+    * - start + 6:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-14:
+
+V4L2_META_FMT_GENERIC_CSI2_14
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_14 contains 8-bit generic metadata with CSI-2
+packing, 14 bits for each 8 bits of data. Every four bytes of metadata is
+followed by three bytes of padding.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_14.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+      - p
+      - p
+    * - start + 7:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-16:
+
+V4L2_META_FMT_GENERIC_CSI2_16
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_16 contains 8-bit generic metadata with CSI-2
+packing, 16 bits for each 8 bits of data. Every byte of metadata is
+followed by one byte of padding.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_16.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - p
+      - M\ :sub:`10`
+      - p
+      - M\ :sub:`20`
+      - p
+      - M\ :sub:`30`
+      - p
+    * - start + 8:
+      - M\ :sub:`01`
+      - p
+      - M\ :sub:`11`
+      - p
+      - M\ :sub:`21`
+      - p
+      - M\ :sub:`31`
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-20:
+
+V4L2_META_FMT_GENERIC_CSI2_20
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_20 contains 8-bit generic metadata with CSI-2
+packing, 20 bits for each 8 bits of data. Every byte of metadata is followed by
+alternating one and two bytes of padding.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_20.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - p
+      - M\ :sub:`10`
+      - p
+      - p
+      - M\ :sub:`20`
+      - p
+      - M\ :sub:`30`
+      - p
+      - p
+    * - start + 10:
+      - M\ :sub:`01`
+      - p
+      - M\ :sub:`11`
+      - p
+      - p
+      - M\ :sub:`21`
+      - p
+      - M\ :sub:`31`
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-24:
+
+V4L2_META_FMT_GENERIC_CSI2_24
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_24 contains 8-bit generic metadata with CSI-2
+packing, 24 bits for each 8 bits of data. Every byte of metadata is followed by
+alternating one and two bytes of padding.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_24.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - p
+      - p
+      - M\ :sub:`10`
+      - p
+      - p
+      - M\ :sub:`20`
+      - p
+      - p
+      - M\ :sub:`30`
+      - p
+      - p
+    * - start + 12:
+      - M\ :sub:`01`
+      - p
+      - p
+      - M\ :sub:`11`
+      - p
+      - p
+      - M\ :sub:`21`
+      - p
+      - p
+      - M\ :sub:`31`
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-2-24:
+
+V4L2_META_FMT_GENERIC_CSI2_2_24
+-------------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_2_24 contains 8-bit generic metadata with CSI-2
+packing, 24 bits for each two times 8 bits of data. Every two bytes of metadata
+are followed by one byte of padding.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_2_24.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - p
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+    * - start + 6:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - p
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index a858acea6547..7781b0bd3e7d 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -1451,6 +1451,14 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
 	case V4L2_PIX_FMT_Y210:		descr = "10-bit YUYV Packed"; break;
 	case V4L2_PIX_FMT_Y212:		descr = "12-bit YUYV Packed"; break;
 	case V4L2_PIX_FMT_Y216:		descr = "16-bit YUYV Packed"; break;
+	case V4L2_META_FMT_GENERIC_8:	descr = "8-bit Generic Metadata"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_10:	descr = "8b Generic Meta, 10b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_12:	descr = "8b Generic Meta, 12b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_14:	descr = "8b Generic Meta, 14b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_16:	descr = "8b Generic Meta, 16b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_20:	descr = "8b Generic Meta, 20b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_24:	descr = "8b Generic Meta, 24b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_2_24:	descr = "2x8b Generic Meta, 24b CSI-2"; break;
 
 	default:
 		/* Compressed formats */
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index aee75eb9e686..adcbdc15dcdb 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -834,6 +834,15 @@ struct v4l2_pix_format {
 #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
 #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
 
+#define V4L2_META_FMT_GENERIC_8		v4l2_fourcc('M', 'E', 'T', '8') /* Generic 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_10	v4l2_fourcc('M', 'C', '1', 'A') /* 10-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_12	v4l2_fourcc('M', 'C', '1', 'C') /* 12-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_14	v4l2_fourcc('M', 'C', '1', 'E') /* 14-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_16	v4l2_fourcc('M', 'C', '1', 'G') /* 16-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_20	v4l2_fourcc('M', 'C', '1', 'K') /* 20-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_24	v4l2_fourcc('M', 'C', '1', 'O') /* 24-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_2_24	v4l2_fourcc('M', 'C', '2', 'O') /* 2 bytes of 8-bit metadata, 24-bit CSI-2 packed */
+
 /* priv field value to indicates that subsequent fields are valid. */
 #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
 
-- 
2.30.2

[RFC 7/7] media: v4l: Support line-based metadata capture

[Sakari Ailus]

many camera sensors, among other devices, transmit embedded data and image
data for each CSI-2 frame. This embedded data typically contains register
configuration of the sensor that has been used to capture the image data
of the same frame.

The embedded data is received by the CSI-2 receiver and has the same
properties as the image data, including that it is line based: it has
width, height and bytesperline (stride).

Add these fields to struct v4l2_meta_format and document them.

Also add V4L2_FMT_FLAG_META_LINE_BASED to tell a given format is
line-based i.e. these fields of struct v4l2_meta_format are valid for it.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
 .../userspace-api/media/v4l/dev-meta.rst          | 15 +++++++++++++++
 .../userspace-api/media/v4l/vidioc-enum-fmt.rst   |  7 +++++++
 include/uapi/linux/videodev2.h                    | 10 ++++++++++
 3 files changed, 32 insertions(+)

diff --git a/Documentation/userspace-api/media/v4l/dev-meta.rst b/Documentation/userspace-api/media/v4l/dev-meta.rst
index 0e7e1ee1471a..7d3a64514db0 100644
--- a/Documentation/userspace-api/media/v4l/dev-meta.rst
+++ b/Documentation/userspace-api/media/v4l/dev-meta.rst
@@ -65,3 +65,18 @@ to 0.
       - ``buffersize``
       - Maximum buffer size in bytes required for data. The value is set by the
         driver.
+    * - __u32
+      - ``width``
+      - Width of a line of metadata in bytes. Valid when :c:type`v4l2_fmtdesc`
+	flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
+	:c:func:`VIDIOC_ENUM_FMT`.
+    * - __u32
+      - ``height``
+      - Height of a line of metadata in bytes. Valid when :c:type`v4l2_fmtdesc`
+	flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
+	:c:func:`VIDIOC_ENUM_FMT`.
+    * - __u32
+      - ``bytesperlines``
+      - Offset in bytes between the beginning of two consecutive lines. Valid
+	when :c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is
+	set, otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
diff --git a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
index 000c154b0f98..6d7664345a4e 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
@@ -227,6 +227,13 @@ the ``mbus_code`` field is handled differently:
 	The application can ask to configure the quantization of the capture
 	device when calling the :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl with
 	:ref:`V4L2_PIX_FMT_FLAG_SET_CSC <v4l2-pix-fmt-flag-set-csc>` set.
+    * - ``V4L2_FMT_FLAG_META_LINE_BASED``
+      - 0x0200
+      - The metadata format is line-based. In this case the ``width``,
+	``height`` and ``bytesperline`` fields of :c:type:`v4l2_meta_format` are
+	valid. The buffer consists of ``height`` lines, each having ``width``
+	bytes of data and offset between the beginning of each two consecutive
+	lines is ``bytesperline``.
 
 Return Value
 ============
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index adcbdc15dcdb..3681b2c15901 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -873,6 +873,7 @@ struct v4l2_fmtdesc {
 #define V4L2_FMT_FLAG_CSC_YCBCR_ENC		0x0080
 #define V4L2_FMT_FLAG_CSC_HSV_ENC		V4L2_FMT_FLAG_CSC_YCBCR_ENC
 #define V4L2_FMT_FLAG_CSC_QUANTIZATION		0x0100
+#define V4L2_FMT_FLAG_META_LINE_BASED		0x0200
 
 	/* Frame Size and frame rate enumeration */
 /*
@@ -2407,10 +2408,19 @@ struct v4l2_sdr_format {
  * struct v4l2_meta_format - metadata format definition
  * @dataformat:		little endian four character code (fourcc)
  * @buffersize:		maximum size in bytes required for data
+ * @width:		number of bytes of data per line (valid for line based
+ *			formats only, see format documentation)
+ * @height:		number of lines of data per buffer (valid for line based
+ *			formats only)
+ * @bytesperline:	offset between the beginnings of two adjacent lines in
+ *			bytes (valid for line based formats only)
  */
 struct v4l2_meta_format {
 	__u32				dataformat;
 	__u32				buffersize;
+	__u32				width;
+	__u32				height;
+	__u32				bytesperline;
 } __attribute__ ((packed));
 
 /**
-- 
2.30.2

Re: [RFC 1/7] media: mc: Add INTERNAL_SOURCE pad type flag

[Tomi Valkeinen]

On 06/05/2023 00:52, Sakari Ailus wrote:
> Internal source pads will be used as routing endpoints in V4L2
> [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
> 
> Also prevent creating links to pads that have been flagged as internal.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>   .../userspace-api/media/mediactl/media-types.rst          | 7 +++++++
>   drivers/media/mc/mc-entity.c                              | 8 +++++++-
>   include/uapi/linux/media.h                                | 1 +
>   3 files changed, 15 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/userspace-api/media/mediactl/media-types.rst b/Documentation/userspace-api/media/mediactl/media-types.rst
> index 0ffeece1e0c8..c724139ad46c 100644
> --- a/Documentation/userspace-api/media/mediactl/media-types.rst
> +++ b/Documentation/userspace-api/media/mediactl/media-types.rst
> @@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
>   .. _MEDIA-PAD-FL-SINK:
>   .. _MEDIA-PAD-FL-SOURCE:
>   .. _MEDIA-PAD-FL-MUST-CONNECT:
> +.. _MEDIA-PAD-FL-INTERNAL-SOURCE:
>   
>   .. flat-table:: Media pad flags
>       :header-rows:  0
> @@ -382,6 +383,12 @@ Types and flags used to represent the media graph elements
>   	  when this flag isn't set; the absence of the flag doesn't imply
>   	  there is none.
>   
> +    *  -  ``MEDIA_PAD_FL_INTERNAL_SOURCE``
> +       -  This flag indicates an internal pad that has no external
> +	  connections. Such a pad may not be connected with a link. The internal

"must not"? Or "shall not"?

  Tomi

Re: [RFC 2/7] media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs

[Tomi Valkeinen]

On 06/05/2023 00:52, Sakari Ailus wrote:
> Take the new INTERNAL_SOURCE pad flag into account in validating routing
> IOCTL argument. Effectively this is a SINK pad in this respect. Due to the
> union there's no need to check for the particular name.

What union? The one you add in the next patch?

As a concept the internal source pads sound good, and they work in 
practice in my tests. But this union is what grates me a bit. We have a 
flag, MEDIA_PAD_FL_INTERNAL_SOURCE, which tells which field of the union 
to use, and then we go and do not use the new union field. Well, and 
also the naming, as we normally have source and sink pads, but now we 
also have internal source pads, which are actually identical to sink pads...

I understand the idea and reasoning, but the two points above do confuse 
me and I'm sure would confuse others also.

I wonder if it would be less or more confusing to simplify this by just 
adding a MEDIA_PAD_FL_INTERNAL, which could be applied to either a 
source or a sink pad, and would prevent linking to it. The flag would 
indicate that the stream from/to that pad is generated/consumed 
internally. (I don't know when you would need an internal pad to consume 
data, but... who knows, the need might pop up =).

That would mean that an "internal sink pad" would generate a data 
stream, i.e. it's a "source", but I think a normal sink pad is almost 
the same anyway: when considering entity's internal routing, the normal 
sink pad is a "source", and the same logic would apply with the internal 
pads too.

An internal sink pad that generates a stream is a bit more confusing 
than an internal source pad, but I think that confusion is less than the 
rest of the confusion in the code-side that comes with the internal 
source pads.

  Tomi

> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>   drivers/media/v4l2-core/v4l2-subdev.c | 6 ++++--
>   1 file changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 2ec179cd1264..6312fc2bf1f0 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -906,7 +906,8 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>   				return -EINVAL;
>   
>   			if (!(pads[route->sink_pad].flags &
> -			      MEDIA_PAD_FL_SINK))
> +			      (MEDIA_PAD_FL_SINK |
> +			       MEDIA_PAD_FL_INTERNAL_SOURCE)))
>   				return -EINVAL;
>   
>   			if (route->source_pad >= sd->entity.num_pads)
> @@ -1787,7 +1788,8 @@ int v4l2_subdev_routing_validate(struct v4l2_subdev *sd,
>   
>   		/* Validate the sink and source pad numbers. */
>   		if (route->sink_pad >= sd->entity.num_pads ||
> -		    !(sd->entity.pads[route->sink_pad].flags & MEDIA_PAD_FL_SINK)) {
> +		    !(sd->entity.pads[route->sink_pad].flags &
> +		      (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL_SOURCE))) {
>   			dev_dbg(sd->dev, "route %u sink (%u) is not a sink pad\n",
>   				i, route->sink_pad);
>   			goto out;

Re: [RFC 3/7] media: uapi: v4l: Document source routes

[Tomi Valkeinen]

On 06/05/2023 00:52, Sakari Ailus wrote:
> Document how internal pads are used on source routes.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>   .../userspace-api/media/v4l/dev-subdev.rst     | 18 ++++++++++++++++++
>   .../media/v4l/vidioc-subdev-g-routing.rst      |  5 +++++
>   include/uapi/linux/v4l2-subdev.h               |  6 +++++-
>   3 files changed, 28 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> index a4f1df7093e8..395e06d2f0f2 100644
> --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> @@ -551,6 +551,24 @@ A stream at a specific point in the media pipeline is identified by the
>   sub-device and a (pad, stream) pair. For sub-devices that do not support
>   multiplexed streams the 'stream' field is always 0.
>   
> +.. _v4l2-subdev-source-routes:
> +
> +Source routes
> +^^^^^^^^^^^^^
> +
> +Cases where a single sub-device pad is a source of multiple streams are special
> +as there is no sink pad for such a route. In those cases, an internal pad is
> +used as the sink pad. Such pads have the :ref:`MEDIA_PAD_FL_INTERNAL_SOURCE
> +<MEDIA-PAD-FL-INTERNAL-SOURCE>` flag set.
> +
> +Internal pads have all the properties of a sink pad in such case, including
> +formats and selections. The format in this case is the source format of the
> +stream. An internal pad always has a single stream only (0).
> +
> +Generally source routes are not modifiable but they can be activated and
> +inactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +<v4l2-subdev-routing-flags>` flag, depending on driver capabilities.
> +

I find the above chapter a bit difficult to read, but I think we need to 
conclude on the internal-pad vs internal-source-pad discussion first. 
However, one point/question:

You write that there's only one stream in an internal pad. I wonder if 
that's a good or a necessary limitation... Do you see that allowing 
multiple streams brings extra complexity? It's still up to each driver 
to decide how many streams they support (most would just support a 
single one), so each driver can easily enforce that limit.

I'm fine with starting with only single-stream support, but if we later 
need to loosen that restriction, I wonder if it'll be difficult if we 
have specified that there can be only one stream. I.e. will the drivers 
and the userspace depend on that limit.

  Tomi

Re: [RFC 1/7] media: mc: Add INTERNAL_SOURCE pad type flag

[Sakari Ailus]

Hi Tomi,

Thank you for the review.

On Mon, May 08, 2023 at 12:52:24PM +0300, Tomi Valkeinen wrote:
> On 06/05/2023 00:52, Sakari Ailus wrote:
> > Internal source pads will be used as routing endpoints in V4L2
> > [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
> > 
> > Also prevent creating links to pads that have been flagged as internal.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >   .../userspace-api/media/mediactl/media-types.rst          | 7 +++++++
> >   drivers/media/mc/mc-entity.c                              | 8 +++++++-
> >   include/uapi/linux/media.h                                | 1 +
> >   3 files changed, 15 insertions(+), 1 deletion(-)
> > 
> > diff --git a/Documentation/userspace-api/media/mediactl/media-types.rst b/Documentation/userspace-api/media/mediactl/media-types.rst
> > index 0ffeece1e0c8..c724139ad46c 100644
> > --- a/Documentation/userspace-api/media/mediactl/media-types.rst
> > +++ b/Documentation/userspace-api/media/mediactl/media-types.rst
> > @@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
> >   .. _MEDIA-PAD-FL-SINK:
> >   .. _MEDIA-PAD-FL-SOURCE:
> >   .. _MEDIA-PAD-FL-MUST-CONNECT:
> > +.. _MEDIA-PAD-FL-INTERNAL-SOURCE:
> >   .. flat-table:: Media pad flags
> >       :header-rows:  0
> > @@ -382,6 +383,12 @@ Types and flags used to represent the media graph elements
> >   	  when this flag isn't set; the absence of the flag doesn't imply
> >   	  there is none.
> > +    *  -  ``MEDIA_PAD_FL_INTERNAL_SOURCE``
> > +       -  This flag indicates an internal pad that has no external
> > +	  connections. Such a pad may not be connected with a link. The internal
> 
> "must not"? Or "shall not"?

I think "may not" is appropriate. I'd be fine with shall, too, albeit it
wouldn't change the meaning in practice.

-- 
Sakari Ailus

Re: [RFC 1/7] media: mc: Add INTERNAL_SOURCE pad type flag

[Tomi Valkeinen]

On 08/05/2023 15:04, Sakari Ailus wrote:
> Hi Tomi,
> 
> Thank you for the review.
> 
> On Mon, May 08, 2023 at 12:52:24PM +0300, Tomi Valkeinen wrote:
>> On 06/05/2023 00:52, Sakari Ailus wrote:
>>> Internal source pads will be used as routing endpoints in V4L2
>>> [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
>>>
>>> Also prevent creating links to pads that have been flagged as internal.
>>>
>>> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
>>> ---
>>>    .../userspace-api/media/mediactl/media-types.rst          | 7 +++++++
>>>    drivers/media/mc/mc-entity.c                              | 8 +++++++-
>>>    include/uapi/linux/media.h                                | 1 +
>>>    3 files changed, 15 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/Documentation/userspace-api/media/mediactl/media-types.rst b/Documentation/userspace-api/media/mediactl/media-types.rst
>>> index 0ffeece1e0c8..c724139ad46c 100644
>>> --- a/Documentation/userspace-api/media/mediactl/media-types.rst
>>> +++ b/Documentation/userspace-api/media/mediactl/media-types.rst
>>> @@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
>>>    .. _MEDIA-PAD-FL-SINK:
>>>    .. _MEDIA-PAD-FL-SOURCE:
>>>    .. _MEDIA-PAD-FL-MUST-CONNECT:
>>> +.. _MEDIA-PAD-FL-INTERNAL-SOURCE:
>>>    .. flat-table:: Media pad flags
>>>        :header-rows:  0
>>> @@ -382,6 +383,12 @@ Types and flags used to represent the media graph elements
>>>    	  when this flag isn't set; the absence of the flag doesn't imply
>>>    	  there is none.
>>> +    *  -  ``MEDIA_PAD_FL_INTERNAL_SOURCE``
>>> +       -  This flag indicates an internal pad that has no external
>>> +	  connections. Such a pad may not be connected with a link. The internal
>>
>> "must not"? Or "shall not"?
> 
> I think "may not" is appropriate. I'd be fine with shall, too, albeit it
> wouldn't change the meaning in practice.

Ok. I don't speak standardize, and I'm not a native English speaker. But 
to me "may not be connected" sounds like it is possibly not connected, 
but also that it is possible for it to be connected.

  Tomi

Re: [RFC 2/7] media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs

[Sakari Ailus]

Moi,

On Mon, May 08, 2023 at 01:14:07PM +0300, Tomi Valkeinen wrote:
> On 06/05/2023 00:52, Sakari Ailus wrote:
> > Take the new INTERNAL_SOURCE pad flag into account in validating routing
> > IOCTL argument. Effectively this is a SINK pad in this respect. Due to the
> > union there's no need to check for the particular name.
> 
> What union? The one you add in the next patch?

Oops. I think we can reorder the patches.

> 
> As a concept the internal source pads sound good, and they work in practice
> in my tests. But this union is what grates me a bit. We have a flag,
> MEDIA_PAD_FL_INTERNAL_SOURCE, which tells which field of the union to use,
> and then we go and do not use the new union field. Well, and also the
> naming, as we normally have source and sink pads, but now we also have
> internal source pads, which are actually identical to sink pads...

The union still should be used by the user space. We're testing flags here
and those flags are the same independently of the INTERNAL_SOURCE flag.

I'm fine by not adding that union though, but for the user space I think
it's better we have it: explaining that the sink_pad has a different
meaning if that flag is set is harder than making it a union member.

> 
> I understand the idea and reasoning, but the two points above do confuse me
> and I'm sure would confuse others also.
> 
> I wonder if it would be less or more confusing to simplify this by just
> adding a MEDIA_PAD_FL_INTERNAL, which could be applied to either a source or
> a sink pad, and would prevent linking to it. The flag would indicate that
> the stream from/to that pad is generated/consumed internally. (I don't know
> when you would need an internal pad to consume data, but... who knows, the
> need might pop up =).

This is another option. But I envision there will be more compatibility
issues. Although... generally using such hardware requires knowledge of the
device, and we haven't obviously had any support for devices needing this
functionality in the tree. So in the end it might not matter much.

> 
> That would mean that an "internal sink pad" would generate a data stream,
> i.e. it's a "source", but I think a normal sink pad is almost the same
> anyway: when considering entity's internal routing, the normal sink pad is a
> "source", and the same logic would apply with the internal pads too.
> 
> An internal sink pad that generates a stream is a bit more confusing than an
> internal source pad, but I think that confusion is less than the rest of the
> confusion in the code-side that comes with the internal source pads.

I prefer having the possible sources of the confusion in the framework than
in the drivers. Therefore I think INTERNAL_SOURCE flag is a (slightly)
better option.

I wonder what Llaurent thinks.

-- 
Kind regards,

Sakari Ailus

Re: [RFC 1/7] media: mc: Add INTERNAL_SOURCE pad type flag

[Sakari Ailus]

On Mon, May 08, 2023 at 03:07:31PM +0300, Tomi Valkeinen wrote:
> On 08/05/2023 15:04, Sakari Ailus wrote:
> > Hi Tomi,
> > 
> > Thank you for the review.
> > 
> > On Mon, May 08, 2023 at 12:52:24PM +0300, Tomi Valkeinen wrote:
> > > On 06/05/2023 00:52, Sakari Ailus wrote:
> > > > Internal source pads will be used as routing endpoints in V4L2
> > > > [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
> > > > 
> > > > Also prevent creating links to pads that have been flagged as internal.
> > > > 
> > > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > > ---
> > > >    .../userspace-api/media/mediactl/media-types.rst          | 7 +++++++
> > > >    drivers/media/mc/mc-entity.c                              | 8 +++++++-
> > > >    include/uapi/linux/media.h                                | 1 +
> > > >    3 files changed, 15 insertions(+), 1 deletion(-)
> > > > 
> > > > diff --git a/Documentation/userspace-api/media/mediactl/media-types.rst b/Documentation/userspace-api/media/mediactl/media-types.rst
> > > > index 0ffeece1e0c8..c724139ad46c 100644
> > > > --- a/Documentation/userspace-api/media/mediactl/media-types.rst
> > > > +++ b/Documentation/userspace-api/media/mediactl/media-types.rst
> > > > @@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
> > > >    .. _MEDIA-PAD-FL-SINK:
> > > >    .. _MEDIA-PAD-FL-SOURCE:
> > > >    .. _MEDIA-PAD-FL-MUST-CONNECT:
> > > > +.. _MEDIA-PAD-FL-INTERNAL-SOURCE:
> > > >    .. flat-table:: Media pad flags
> > > >        :header-rows:  0
> > > > @@ -382,6 +383,12 @@ Types and flags used to represent the media graph elements
> > > >    	  when this flag isn't set; the absence of the flag doesn't imply
> > > >    	  there is none.
> > > > +    *  -  ``MEDIA_PAD_FL_INTERNAL_SOURCE``
> > > > +       -  This flag indicates an internal pad that has no external
> > > > +	  connections. Such a pad may not be connected with a link. The internal
> > > 
> > > "must not"? Or "shall not"?
> > 
> > I think "may not" is appropriate. I'd be fine with shall, too, albeit it
> > wouldn't change the meaning in practice.
> 
> Ok. I don't speak standardize, and I'm not a native English speaker. But to
> me "may not be connected" sounds like it is possibly not connected, but also
> that it is possible for it to be connected.

Ah, yes, I think you're actually right. But that is what I thought when
writing the text. ;-) Shall we use shall, then?

-- 
Sakari Ailus

Re: [RFC 3/7] media: uapi: v4l: Document source routes

[Sakari Ailus]

Moi,

On Mon, May 08, 2023 at 01:33:24PM +0300, Tomi Valkeinen wrote:
> On 06/05/2023 00:52, Sakari Ailus wrote:
> > Document how internal pads are used on source routes.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >   .../userspace-api/media/v4l/dev-subdev.rst     | 18 ++++++++++++++++++
> >   .../media/v4l/vidioc-subdev-g-routing.rst      |  5 +++++
> >   include/uapi/linux/v4l2-subdev.h               |  6 +++++-
> >   3 files changed, 28 insertions(+), 1 deletion(-)
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > index a4f1df7093e8..395e06d2f0f2 100644
> > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > @@ -551,6 +551,24 @@ A stream at a specific point in the media pipeline is identified by the
> >   sub-device and a (pad, stream) pair. For sub-devices that do not support
> >   multiplexed streams the 'stream' field is always 0.
> > +.. _v4l2-subdev-source-routes:
> > +
> > +Source routes
> > +^^^^^^^^^^^^^
> > +
> > +Cases where a single sub-device pad is a source of multiple streams are special
> > +as there is no sink pad for such a route. In those cases, an internal pad is
> > +used as the sink pad. Such pads have the :ref:`MEDIA_PAD_FL_INTERNAL_SOURCE
> > +<MEDIA-PAD-FL-INTERNAL-SOURCE>` flag set.
> > +
> > +Internal pads have all the properties of a sink pad in such case, including
> > +formats and selections. The format in this case is the source format of the
> > +stream. An internal pad always has a single stream only (0).
> > +
> > +Generally source routes are not modifiable but they can be activated and
> > +inactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > +<v4l2-subdev-routing-flags>` flag, depending on driver capabilities.
> > +
> 
> I find the above chapter a bit difficult to read, but I think we need to
> conclude on the internal-pad vs internal-source-pad discussion first.
> However, one point/question:
> 
> You write that there's only one stream in an internal pad. I wonder if
> that's a good or a necessary limitation... Do you see that allowing multiple
> streams brings extra complexity? It's still up to each driver to decide how
> many streams they support (most would just support a single one), so each
> driver can easily enforce that limit.

Good question. As we don't seem to have a tangible reason to allow it, I'd
deny it until there's a use case.

Or did you have a use case in mind?

I thought indicating which streams will be bound together (i.e. either are
all disabled or enabled) could be one, but I'm not sure we need that at the
moment at least.

> 
> I'm fine with starting with only single-stream support, but if we later need
> to loosen that restriction, I wonder if it'll be difficult if we have
> specified that there can be only one stream. I.e. will the drivers and the
> userspace depend on that limit.

We can always allow what wasn't allowed previously so that's a non-issue,
really. But it needs to bring new functionality, otherwise there's no
reason to do that.

-- 
Kind regards,

Sakari Ailus

Re: [RFC 3/7] media: uapi: v4l: Document source routes

[Tomi Valkeinen]

On 08/05/2023 19:26, Sakari Ailus wrote:
> Moi,
> 
> On Mon, May 08, 2023 at 01:33:24PM +0300, Tomi Valkeinen wrote:
>> On 06/05/2023 00:52, Sakari Ailus wrote:
>>> Document how internal pads are used on source routes.
>>>
>>> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
>>> ---
>>>    .../userspace-api/media/v4l/dev-subdev.rst     | 18 ++++++++++++++++++
>>>    .../media/v4l/vidioc-subdev-g-routing.rst      |  5 +++++
>>>    include/uapi/linux/v4l2-subdev.h               |  6 +++++-
>>>    3 files changed, 28 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
>>> index a4f1df7093e8..395e06d2f0f2 100644
>>> --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
>>> +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
>>> @@ -551,6 +551,24 @@ A stream at a specific point in the media pipeline is identified by the
>>>    sub-device and a (pad, stream) pair. For sub-devices that do not support
>>>    multiplexed streams the 'stream' field is always 0.
>>> +.. _v4l2-subdev-source-routes:
>>> +
>>> +Source routes
>>> +^^^^^^^^^^^^^
>>> +
>>> +Cases where a single sub-device pad is a source of multiple streams are special
>>> +as there is no sink pad for such a route. In those cases, an internal pad is
>>> +used as the sink pad. Such pads have the :ref:`MEDIA_PAD_FL_INTERNAL_SOURCE
>>> +<MEDIA-PAD-FL-INTERNAL-SOURCE>` flag set.
>>> +
>>> +Internal pads have all the properties of a sink pad in such case, including
>>> +formats and selections. The format in this case is the source format of the
>>> +stream. An internal pad always has a single stream only (0).
>>> +
>>> +Generally source routes are not modifiable but they can be activated and
>>> +inactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE
>>> +<v4l2-subdev-routing-flags>` flag, depending on driver capabilities.
>>> +
>>
>> I find the above chapter a bit difficult to read, but I think we need to
>> conclude on the internal-pad vs internal-source-pad discussion first.
>> However, one point/question:
>>
>> You write that there's only one stream in an internal pad. I wonder if
>> that's a good or a necessary limitation... Do you see that allowing multiple
>> streams brings extra complexity? It's still up to each driver to decide how
>> many streams they support (most would just support a single one), so each
>> driver can easily enforce that limit.
> 
> Good question. As we don't seem to have a tangible reason to allow it, I'd
> deny it until there's a use case.
> 
> Or did you have a use case in mind?
> 
> I thought indicating which streams will be bound together (i.e. either are
> all disabled or enabled) could be one, but I'm not sure we need that at the
> moment at least.

I don't have nothing concrete in mind. Maybe a TPG which also generates 
some kind of metadata. But that could be implemented as two internal pads.

>> I'm fine with starting with only single-stream support, but if we later need
>> to loosen that restriction, I wonder if it'll be difficult if we have
>> specified that there can be only one stream. I.e. will the drivers and the
>> userspace depend on that limit.
> 
> We can always allow what wasn't allowed previously so that's a non-issue,
> really. But it needs to bring new functionality, otherwise there's no
> reason to do that.

It's not always that easy. If the drivers and the userspace expect that 
there's a single route with ID 0, and then with a new kernel there are 
more streams or the single stream is ID 1, things could go wrong.

  Tomi

Re: [RFC 3/7] media: uapi: v4l: Document source routes

[Sakari Ailus]

Moi,

On Mon, May 08, 2023 at 07:35:04PM +0300, Tomi Valkeinen wrote:
> On 08/05/2023 19:26, Sakari Ailus wrote:
> > Moi,
> > 
> > On Mon, May 08, 2023 at 01:33:24PM +0300, Tomi Valkeinen wrote:
> > > On 06/05/2023 00:52, Sakari Ailus wrote:
> > > > Document how internal pads are used on source routes.
> > > > 
> > > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > > ---
> > > >    .../userspace-api/media/v4l/dev-subdev.rst     | 18 ++++++++++++++++++
> > > >    .../media/v4l/vidioc-subdev-g-routing.rst      |  5 +++++
> > > >    include/uapi/linux/v4l2-subdev.h               |  6 +++++-
> > > >    3 files changed, 28 insertions(+), 1 deletion(-)
> > > > 
> > > > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > > index a4f1df7093e8..395e06d2f0f2 100644
> > > > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > > @@ -551,6 +551,24 @@ A stream at a specific point in the media pipeline is identified by the
> > > >    sub-device and a (pad, stream) pair. For sub-devices that do not support
> > > >    multiplexed streams the 'stream' field is always 0.
> > > > +.. _v4l2-subdev-source-routes:
> > > > +
> > > > +Source routes
> > > > +^^^^^^^^^^^^^
> > > > +
> > > > +Cases where a single sub-device pad is a source of multiple streams are special
> > > > +as there is no sink pad for such a route. In those cases, an internal pad is
> > > > +used as the sink pad. Such pads have the :ref:`MEDIA_PAD_FL_INTERNAL_SOURCE
> > > > +<MEDIA-PAD-FL-INTERNAL-SOURCE>` flag set.
> > > > +
> > > > +Internal pads have all the properties of a sink pad in such case, including
> > > > +formats and selections. The format in this case is the source format of the
> > > > +stream. An internal pad always has a single stream only (0).
> > > > +
> > > > +Generally source routes are not modifiable but they can be activated and
> > > > +inactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > > > +<v4l2-subdev-routing-flags>` flag, depending on driver capabilities.
> > > > +
> > > 
> > > I find the above chapter a bit difficult to read, but I think we need to
> > > conclude on the internal-pad vs internal-source-pad discussion first.
> > > However, one point/question:
> > > 
> > > You write that there's only one stream in an internal pad. I wonder if
> > > that's a good or a necessary limitation... Do you see that allowing multiple
> > > streams brings extra complexity? It's still up to each driver to decide how
> > > many streams they support (most would just support a single one), so each
> > > driver can easily enforce that limit.
> > 
> > Good question. As we don't seem to have a tangible reason to allow it, I'd
> > deny it until there's a use case.
> > 
> > Or did you have a use case in mind?
> > 
> > I thought indicating which streams will be bound together (i.e. either are
> > all disabled or enabled) could be one, but I'm not sure we need that at the
> > moment at least.
> 
> I don't have nothing concrete in mind. Maybe a TPG which also generates some
> kind of metadata. But that could be implemented as two internal pads.
> 
> > > I'm fine with starting with only single-stream support, but if we later need
> > > to loosen that restriction, I wonder if it'll be difficult if we have
> > > specified that there can be only one stream. I.e. will the drivers and the
> > > userspace depend on that limit.
> > 
> > We can always allow what wasn't allowed previously so that's a non-issue,
> > really. But it needs to bring new functionality, otherwise there's no
> > reason to do that.
> 
> It's not always that easy. If the drivers and the userspace expect that
> there's a single route with ID 0, and then with a new kernel there are more
> streams or the single stream is ID 1, things could go wrong.

Other drivers are a non-issue IMO as this isn't visible to other drivers.

I think for user space this could be an issue if it were entirely generic.
But that's probably not going to be the case, is it? I don't mean test
programs.

-- 
Terveisin,

Sakari Ailus

Re: [RFC 0/7] Generic line based metadata support, internal pads

[Naushir Patuck]

Hi Sakari,

Thank you for working on this.  Sensor metadata is something that Raspberry Pi
do make extensive use of, and our downstream changes to support it, although a
bit hacky, are not too dissimilar to your proposal here.

On Fri, 5 May 2023 at 22:53, Sakari Ailus <sakari.ailus@linux.intel.com> wrote:
>
> Hi folks,
>
> Here are a few patches to add support generic, line based metadata as well
> as internal source pads. While the amount of code is not very large, to
> the contrary it is quite small actually IMO, I presume what this is about
> and why it is being proposed requires some explaining.
>
> Metadata mbus codes and formats have existed for some time in V4L2. They
> however have been only used by drivers that produce the data itself and
> effectively this metadata has always been statistics of some sort (at
> least when it comes to ISPs). What is different here is that we intend to
> add support for metadata originating from camera sensors.
>
> Camera sensors produce different kinds of metadata, embedded data (usually
> register address--value pairs used to capture the frame, in a more or less
> sensor specific format), histograms (in a very sensor specific format),
> dark pixels etc. The number of these formats is probably going to be about
> as large as image data formats if not larger, as the image data formats
> are much better standardised but a smaller subset of them will be
> supported by V4L2, at least initially but possibly much more in the long
> run.
>
> Having this many device specific formats would be a major problem for all
> the other drivers along that pipeline (not to mention the users of those
> drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2
> receiver drivers that have DMA: the poor driver developer would not only
> need to know camera sensor specific formats but to choose the specific
> packing of that format suitable for the DMA used by the hardware. It is
> unlikely many of these would ever get tested while being present on the
> driver API. Also adding new sensors with new embedded data formats would
> involve updating all bridge and CSI-2 receiver drivers. I don't expect
> this to be a workable approach.
>
> Instead what I'm proposing is to use specific metadata formats on the
> sensor devices only, on internal pads (more about those soon) of the
> sensors, only visible in the UAPI, and then generic mbus formats along the
> pipeline and finally generic V4L2 metadata formats on the DMAs (specific
> to bit depth and packing). This would unsnarl the two, defining what data
> there is (specific mbus code) and how that is transported and packed
> (generic mbus codes and V4L2 formats).
>
> The user space would be required to "know" the path of that data from the
> sensor's internal pad to the V4L2 video node. I do not see this as these
> devices require at least some knowledge of the pipeline, i.e. hardware at
> hand. Separating what the data means and how it is packed may even be
> beneficial: it allows separating code that interprets the data (sensor
> internal mbus code) from the code that accesses it (packing).
>
> These formats are in practice line based, meaning that there may be
> padding at the end of the line, depending on the bus as well as the DMA.
> If non-line based formats are needed, it is always possible to set the
> "height" field to 1.

One thing that may be worth considering or clarifying - for the case of the
BCM2835 Unicam CSI-2 device, we only have 2x DMA output channels.  So one will
match image data packets, and the other will match "everything else".  Typically
"everything else" would only be CSI-2 embedded data, but in the case of the
Raspberry Pi Camera v3 (IMX708), it includes embedded data, PDAF data, and
HDR histogram data.  Each of these outputs can be programmed to use a different
packet ID in the sensor, but since Unicam only has a single DMA for "everything
else", it all gets dumped into one metadata buffer.  But given we know the exact
structure of the data streams, it's trivial for useland to find the right bits
in this buffer.  Of course, other CSI-2 receivers with more DMA channels might
allow these streams to end up in their own buffers.

Nothing in your series seems to stop us operating Unicam in this way,
particularly because there is no fixed definition of the data format for
V4L2_META_FMT_GENERIC_8. So I don't think it's a problem, but perhaps it's worth
documenting that the metadata might include multiple streams from the sensor?

Regards,
Naush

>
> The internal (source) pads are an alternative to source routes [1]. The
> source routes were not universally liked and I do have to say I like
> re-using existing interface concepts (pads and everything you can do with
> pads, including access format, selections etc.) wherever it makes sense,
> instead of duplicating functionality.
>
> Effectively internal source pads behave mostly just like sink pads, but
> they describe a flow of data that originates from a sub-device instead of
> arriving to a sub-device. The SUBDEV_S_ROUTING IOCTLs are used to enable
> and disable routes from internal source pads to sub-device's source pads.
> The subdev format IOCTLs are usable, too, so one can find which subdev
> format is available on given internal source pad.
>
> This set depends on these patches:
>
> <URL:https://lore.kernel.org/linux-media/20230505205416.55002-1-sakari.ailus@linux.intel.com/T/#t>
>
> I've also pushed these here and I'll keep updating the branch:
>
> <URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=metadata>
>
> Questions and comments are most welcome.
>
>
> [1] <URL:https://lore.kernel.org/linux-media/20220831141357.1396081-20-tomi.valkeinen@ideasonboard.com/>
>
> Sakari Ailus (7):
>   media: mc: Add INTERNAL_SOURCE pad type flag
>   media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs
>   media: uapi: v4l: Document source routes
>   media: mc: Check pad flag validity
>   media: uapi: Add generic serial metadata mbus formats
>   media: uapi: Add generic 8-bit metadata format definitions
>   media: v4l: Support line-based metadata capture
>
>  .../media/mediactl/media-types.rst            |   7 +
>  .../userspace-api/media/v4l/dev-meta.rst      |  15 +
>  .../userspace-api/media/v4l/dev-subdev.rst    |  18 +
>  .../userspace-api/media/v4l/meta-formats.rst  |   1 +
>  .../media/v4l/metafmt-generic.rst             | 317 ++++++++++++++++++
>  .../media/v4l/subdev-formats.rst              | 257 ++++++++++++++
>  .../media/v4l/vidioc-enum-fmt.rst             |   7 +
>  .../media/v4l/vidioc-subdev-g-routing.rst     |   5 +
>  drivers/media/mc/mc-entity.c                  |  20 +-
>  drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
>  drivers/media/v4l2-core/v4l2-subdev.c         |   6 +-
>  include/uapi/linux/media-bus-format.h         |   9 +
>  include/uapi/linux/media.h                    |   1 +
>  include/uapi/linux/v4l2-subdev.h              |   6 +-
>  include/uapi/linux/videodev2.h                |  19 ++
>  15 files changed, 691 insertions(+), 5 deletions(-)
>  create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
>
> --
> 2.30.2
>

Re: [RFC 0/7] Generic line based metadata support, internal pads

[Sakari Ailus]

Hi Naushir,

On Fri, Jun 02, 2023 at 08:54:35AM +0100, Naushir Patuck wrote:
> Hi Sakari,
> 
> Thank you for working on this. Sensor metadata is something that
> Raspberry Pi do make extensive use of, and our downstream changes to
> support it, although a bit hacky, are not too dissimilar to your proposal
> here.
> 
> On Fri, 5 May 2023 at 22:53, Sakari Ailus <sakari.ailus@linux.intel.com> wrote:
> >
> > Hi folks,
> >
> > Here are a few patches to add support generic, line based metadata as well
> > as internal source pads. While the amount of code is not very large, to
> > the contrary it is quite small actually IMO, I presume what this is about
> > and why it is being proposed requires some explaining.
> >
> > Metadata mbus codes and formats have existed for some time in V4L2. They
> > however have been only used by drivers that produce the data itself and
> > effectively this metadata has always been statistics of some sort (at
> > least when it comes to ISPs). What is different here is that we intend to
> > add support for metadata originating from camera sensors.
> >
> > Camera sensors produce different kinds of metadata, embedded data (usually
> > register address--value pairs used to capture the frame, in a more or less
> > sensor specific format), histograms (in a very sensor specific format),
> > dark pixels etc. The number of these formats is probably going to be about
> > as large as image data formats if not larger, as the image data formats
> > are much better standardised but a smaller subset of them will be
> > supported by V4L2, at least initially but possibly much more in the long
> > run.
> >
> > Having this many device specific formats would be a major problem for all
> > the other drivers along that pipeline (not to mention the users of those
> > drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2
> > receiver drivers that have DMA: the poor driver developer would not only
> > need to know camera sensor specific formats but to choose the specific
> > packing of that format suitable for the DMA used by the hardware. It is
> > unlikely many of these would ever get tested while being present on the
> > driver API. Also adding new sensors with new embedded data formats would
> > involve updating all bridge and CSI-2 receiver drivers. I don't expect
> > this to be a workable approach.
> >
> > Instead what I'm proposing is to use specific metadata formats on the
> > sensor devices only, on internal pads (more about those soon) of the
> > sensors, only visible in the UAPI, and then generic mbus formats along the
> > pipeline and finally generic V4L2 metadata formats on the DMAs (specific
> > to bit depth and packing). This would unsnarl the two, defining what data
> > there is (specific mbus code) and how that is transported and packed
> > (generic mbus codes and V4L2 formats).
> >
> > The user space would be required to "know" the path of that data from the
> > sensor's internal pad to the V4L2 video node. I do not see this as these
> > devices require at least some knowledge of the pipeline, i.e. hardware at
> > hand. Separating what the data means and how it is packed may even be
> > beneficial: it allows separating code that interprets the data (sensor
> > internal mbus code) from the code that accesses it (packing).
> >
> > These formats are in practice line based, meaning that there may be
> > padding at the end of the line, depending on the bus as well as the DMA.
> > If non-line based formats are needed, it is always possible to set the
> > "height" field to 1.
> 
> One thing that may be worth considering or clarifying - for the case of
> the BCM2835 Unicam CSI-2 device, we only have 2x DMA output channels. So
> one will match image data packets, and the other will match "everything
> else". Typically "everything else" would only be CSI-2 embedded data, but
> in the case of the Raspberry Pi Camera v3 (IMX708), it includes embedded
> data, PDAF data, and HDR histogram data. Each of these outputs can be
> programmed to use a different packet ID in the sensor, but since Unicam
> only has a single DMA for "everything else", it all gets dumped into one
> metadata buffer. But given we know the exact structure of the data
> streams, it's trivial for useland to find the right bits in this buffer.
> Of course, other CSI-2 receivers with more DMA channels might allow these
> streams to end up in their own buffers.
> 
> Nothing in your series seems to stop us operating Unicam in this way,
> particularly because there is no fixed definition of the data format for
> V4L2_META_FMT_GENERIC_8. So I don't think it's a problem, but perhaps
> it's worth documenting that the metadata might include multiple streams
> from the sensor?

I believe this happens on other hardware, too, indeed. Currently the
documentation says that

	Any number of routes from streams on sink pads towards streams on
	source pads is allowed, to the extent supported by drivers. For
	every stream on a source pad, however, only a single route is
	allowed.

	(Documentation/userspace-api/media/v4l/dev-subdev.rst)

This probably needs to be changed to allow what you'd need?

-- 
Kind regards,

Sakari Ailus

Re: [RFC 0/7] Generic line based metadata support, internal pads

[Laurent Pinchart]

Hello,

On Fri, Jun 02, 2023 at 08:54:35AM +0100, Naushir Patuck wrote:
> Hi Sakari,
> 
> Thank you for working on this.  Sensor metadata is something that Raspberry Pi
> do make extensive use of, and our downstream changes to support it, although a
> bit hacky, are not too dissimilar to your proposal here.
> 
> On Fri, 5 May 2023 at 22:53, Sakari Ailus wrote:
> >
> > Hi folks,
> >
> > Here are a few patches to add support generic, line based metadata as well
> > as internal source pads. While the amount of code is not very large, to
> > the contrary it is quite small actually IMO, I presume what this is about
> > and why it is being proposed requires some explaining.
> >
> > Metadata mbus codes and formats have existed for some time in V4L2. They
> > however have been only used by drivers that produce the data itself and
> > effectively this metadata has always been statistics of some sort (at
> > least when it comes to ISPs). What is different here is that we intend to
> > add support for metadata originating from camera sensors.
> >
> > Camera sensors produce different kinds of metadata, embedded data (usually
> > register address--value pairs used to capture the frame, in a more or less
> > sensor specific format), histograms (in a very sensor specific format),
> > dark pixels etc.

Optical dark pixels are image data, I wouldn't include them in the
"metadata" category. They can of course be transmitted over a different
stream, so they're relevant to the API being designed.

> > The number of these formats is probably going to be about
> > as large as image data formats if not larger, as the image data formats
> > are much better standardised but a smaller subset of them will be
> > supported by V4L2, at least initially but possibly much more in the long
> > run.

Strictly speaking, the number of metadata formats depends on how we
define "format". Today, we can use the GREY pixel format to capture
greyscale images in the visible spectrum, but also IR images, thermal
images, or even depth images. They're all one pixel format. On the other
hand, we have Y16 for greyscale visible and IR images, and Z16 for depth
maps. It's already a mess, even without metadata :-)

> > Having this many device specific formats would be a major problem for all
> > the other drivers along that pipeline (not to mention the users of those
> > drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2
> > receiver drivers that have DMA: the poor driver developer would not only
> > need to know camera sensor specific formats but to choose the specific
> > packing of that format suitable for the DMA used by the hardware. It is
> > unlikely many of these would ever get tested while being present on the
> > driver API. Also adding new sensors with new embedded data formats would
> > involve updating all bridge and CSI-2 receiver drivers. I don't expect
> > this to be a workable approach.

I'm glad we agree on this.

> > Instead what I'm proposing is to use specific metadata formats on the
> > sensor devices only, on internal pads (more about those soon) of the
> > sensors, only visible in the UAPI, and then generic mbus formats along the
> > pipeline and finally generic V4L2 metadata formats on the DMAs (specific
> > to bit depth and packing). This would unsnarl the two, defining what data
> > there is (specific mbus code) and how that is transported and packed
> > (generic mbus codes and V4L2 formats).

Decoupling the information needed by the kernel (e.g. are we
transporting RAW8 or RAW10 from the sensor through the pipeline) from
the information useful for userspace only (e.g. the sensor embedded data
is encoding using register + value pairs, based on the IMX708 registers
set) is a good idea. I expect the main question to be where to draw the
line between those two categories. Some pieces of information may be
useless to any processing block in the pipeline except for an odd block
in the middle.

This is, I believe, a problem similar to the CFA pattern. That
information is useless for most devices, but required by the demosaicing
block and some other blocks along the pipeline (colour gains for
instance, or some Bayer statistics engines). We currently convey the CFA
pattern in the media bus codes and pixel formats (e.g. SGRBG8 vs.
SRGGB8) through the whole pipeline, while it could be conveyed out of
band (e.g. exposed by the sensor using a control, and set on the devices
that need it using a control as well).

If we come up with a good solution for metadata (and I hope we will),
maybe we'll be able to use a similar mechanism for CFA patterns,
simplifying new drivers and userspace. Or maybe this will remain a pipe
dream given the backward compatibility implications.

> > The user space would be required to "know" the path of that data from the
> > sensor's internal pad to the V4L2 video node. I do not see this as these
> > devices require at least some knowledge of the pipeline, i.e. hardware at
> > hand. Separating what the data means and how it is packed may even be
> > beneficial: it allows separating code that interprets the data (sensor
> > internal mbus code) from the code that accesses it (packing).
> >
> > These formats are in practice line based, meaning that there may be
> > padding at the end of the line, depending on the bus as well as the DMA.
> > If non-line based formats are needed, it is always possible to set the
> > "height" field to 1.
> 
> One thing that may be worth considering or clarifying - for the case of the
> BCM2835 Unicam CSI-2 device, we only have 2x DMA output channels.  So one will
> match image data packets, and the other will match "everything else".  Typically
> "everything else" would only be CSI-2 embedded data, but in the case of the
> Raspberry Pi Camera v3 (IMX708), it includes embedded data, PDAF data, and
> HDR histogram data.  Each of these outputs can be programmed to use a different
> packet ID in the sensor, but since Unicam only has a single DMA for "everything
> else", it all gets dumped into one metadata buffer.  But given we know the exact
> structure of the data streams, it's trivial for useland to find the right bits
> in this buffer.  Of course, other CSI-2 receivers with more DMA channels might
> allow these streams to end up in their own buffers.
> 
> Nothing in your series seems to stop us operating Unicam in this way,
> particularly because there is no fixed definition of the data format for
> V4L2_META_FMT_GENERIC_8. So I don't think it's a problem, but perhaps it's worth
> documenting that the metadata might include multiple streams from the sensor?

Thanks for your feedback Naush. Would you consider reviewing the
individual patches in the series ? :-)

> > The internal (source) pads are an alternative to source routes [1]. The
> > source routes were not universally liked and I do have to say I like
> > re-using existing interface concepts (pads and everything you can do with
> > pads, including access format, selections etc.) wherever it makes sense,
> > instead of duplicating functionality.
> >
> > Effectively internal source pads behave mostly just like sink pads, but
> > they describe a flow of data that originates from a sub-device instead of
> > arriving to a sub-device. The SUBDEV_S_ROUTING IOCTLs are used to enable
> > and disable routes from internal source pads to sub-device's source pads.
> > The subdev format IOCTLs are usable, too, so one can find which subdev
> > format is available on given internal source pad.
> >
> > This set depends on these patches:
> >
> > <URL:https://lore.kernel.org/linux-media/20230505205416.55002-1-sakari.ailus@linux.intel.com/T/#t>
> >
> > I've also pushed these here and I'll keep updating the branch:
> >
> > <URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=metadata>
> >
> > Questions and comments are most welcome.
> >
> >
> > [1] <URL:https://lore.kernel.org/linux-media/20220831141357.1396081-20-tomi.valkeinen@ideasonboard.com/>
> >
> > Sakari Ailus (7):
> >   media: mc: Add INTERNAL_SOURCE pad type flag
> >   media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs
> >   media: uapi: v4l: Document source routes
> >   media: mc: Check pad flag validity
> >   media: uapi: Add generic serial metadata mbus formats
> >   media: uapi: Add generic 8-bit metadata format definitions
> >   media: v4l: Support line-based metadata capture
> >
> >  .../media/mediactl/media-types.rst            |   7 +
> >  .../userspace-api/media/v4l/dev-meta.rst      |  15 +
> >  .../userspace-api/media/v4l/dev-subdev.rst    |  18 +
> >  .../userspace-api/media/v4l/meta-formats.rst  |   1 +
> >  .../media/v4l/metafmt-generic.rst             | 317 ++++++++++++++++++
> >  .../media/v4l/subdev-formats.rst              | 257 ++++++++++++++
> >  .../media/v4l/vidioc-enum-fmt.rst             |   7 +
> >  .../media/v4l/vidioc-subdev-g-routing.rst     |   5 +
> >  drivers/media/mc/mc-entity.c                  |  20 +-
> >  drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
> >  drivers/media/v4l2-core/v4l2-subdev.c         |   6 +-
> >  include/uapi/linux/media-bus-format.h         |   9 +
> >  include/uapi/linux/media.h                    |   1 +
> >  include/uapi/linux/v4l2-subdev.h              |   6 +-
> >  include/uapi/linux/videodev2.h                |  19 ++
> >  15 files changed, 691 insertions(+), 5 deletions(-)
> >  create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst

-- 
Regards,

Laurent Pinchart

Re: [RFC 1/7] media: mc: Add INTERNAL_SOURCE pad type flag

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Sat, May 06, 2023 at 12:52:51AM +0300, Sakari Ailus wrote:
> Internal source pads will be used as routing endpoints in V4L2
> [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
> 
> Also prevent creating links to pads that have been flagged as internal.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../userspace-api/media/mediactl/media-types.rst          | 7 +++++++
>  drivers/media/mc/mc-entity.c                              | 8 +++++++-
>  include/uapi/linux/media.h                                | 1 +
>  3 files changed, 15 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/userspace-api/media/mediactl/media-types.rst b/Documentation/userspace-api/media/mediactl/media-types.rst
> index 0ffeece1e0c8..c724139ad46c 100644
> --- a/Documentation/userspace-api/media/mediactl/media-types.rst
> +++ b/Documentation/userspace-api/media/mediactl/media-types.rst
> @@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
>  .. _MEDIA-PAD-FL-SINK:
>  .. _MEDIA-PAD-FL-SOURCE:
>  .. _MEDIA-PAD-FL-MUST-CONNECT:
> +.. _MEDIA-PAD-FL-INTERNAL-SOURCE:
>  
>  .. flat-table:: Media pad flags
>      :header-rows:  0
> @@ -382,6 +383,12 @@ Types and flags used to represent the media graph elements
>  	  when this flag isn't set; the absence of the flag doesn't imply
>  	  there is none.
>  
> +    *  -  ``MEDIA_PAD_FL_INTERNAL_SOURCE``
> +       -  This flag indicates an internal pad that has no external
> +	  connections. Such a pad may not be connected with a link. The internal
> +	  flag indicates that the stream either starts or ends in the

There's no mention of "stream" (with this meaning) anywhere in the MC
API documentation, so you will need to explain streams first.

Apart from that, and addressing Tomi's comment, this patch seems OK.

> +	  entity. For a given pad, the INTERNAL_SOURCE flag may not be set if
> +	  either SINK or SOURCE flags is set.
>  
>  One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
>  must be set for every pad.
> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> index ef34ddd715c9..ed99160a2487 100644
> --- a/drivers/media/mc/mc-entity.c
> +++ b/drivers/media/mc/mc-entity.c
> @@ -1062,7 +1062,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
>  
>  	for (i = 0; i < entity->num_pads; i++) {
>  		if ((entity->pads[i].flags &
> -		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
> +		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
> +		      MEDIA_PAD_FL_INTERNAL_SOURCE)) != pad_type)
>  			continue;
>  
>  		if (entity->pads[i].sig_type == sig_type)
> @@ -1087,6 +1088,11 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
>  		return -EINVAL;
>  	if (WARN_ON(!(sink->pads[sink_pad].flags & MEDIA_PAD_FL_SINK)))
>  		return -EINVAL;
> +	if (WARN_ON(sink->pads[sink_pad].flags &
> +		    MEDIA_PAD_FL_INTERNAL_SOURCE) ||
> +	    WARN_ON(source->pads[source_pad].flags &
> +		    MEDIA_PAD_FL_INTERNAL_SOURCE))
> +		return -EINVAL;
>  
>  	link = media_add_link(&source->links);
>  	if (link == NULL)
> diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
> index edb8dfef9eba..0e2577e8b425 100644
> --- a/include/uapi/linux/media.h
> +++ b/include/uapi/linux/media.h
> @@ -208,6 +208,7 @@ struct media_entity_desc {
>  #define MEDIA_PAD_FL_SINK			(1U << 0)
>  #define MEDIA_PAD_FL_SOURCE			(1U << 1)
>  #define MEDIA_PAD_FL_MUST_CONNECT		(1U << 2)
> +#define MEDIA_PAD_FL_INTERNAL_SOURCE		(1U << 3)
>  
>  struct media_pad_desc {
>  	__u32 entity;		/* entity ID */

-- 
Regards,

Laurent Pinchart

Re: [RFC 0/7] Generic line based metadata support, internal pads

[Naushir Patuck]

Hi Sakari,

On Fri, 2 Jun 2023 at 09:46, Sakari Ailus <sakari.ailus@linux.intel.com> wrote:
>
> Hi Naushir,
>
> On Fri, Jun 02, 2023 at 08:54:35AM +0100, Naushir Patuck wrote:
> > Hi Sakari,
> >
> > Thank you for working on this. Sensor metadata is something that
> > Raspberry Pi do make extensive use of, and our downstream changes to
> > support it, although a bit hacky, are not too dissimilar to your proposal
> > here.
> >
> > On Fri, 5 May 2023 at 22:53, Sakari Ailus <sakari.ailus@linux.intel.com> wrote:
> > >
> > > Hi folks,
> > >
> > > Here are a few patches to add support generic, line based metadata as well
> > > as internal source pads. While the amount of code is not very large, to
> > > the contrary it is quite small actually IMO, I presume what this is about
> > > and why it is being proposed requires some explaining.
> > >
> > > Metadata mbus codes and formats have existed for some time in V4L2. They
> > > however have been only used by drivers that produce the data itself and
> > > effectively this metadata has always been statistics of some sort (at
> > > least when it comes to ISPs). What is different here is that we intend to
> > > add support for metadata originating from camera sensors.
> > >
> > > Camera sensors produce different kinds of metadata, embedded data (usually
> > > register address--value pairs used to capture the frame, in a more or less
> > > sensor specific format), histograms (in a very sensor specific format),
> > > dark pixels etc. The number of these formats is probably going to be about
> > > as large as image data formats if not larger, as the image data formats
> > > are much better standardised but a smaller subset of them will be
> > > supported by V4L2, at least initially but possibly much more in the long
> > > run.
> > >
> > > Having this many device specific formats would be a major problem for all
> > > the other drivers along that pipeline (not to mention the users of those
> > > drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2
> > > receiver drivers that have DMA: the poor driver developer would not only
> > > need to know camera sensor specific formats but to choose the specific
> > > packing of that format suitable for the DMA used by the hardware. It is
> > > unlikely many of these would ever get tested while being present on the
> > > driver API. Also adding new sensors with new embedded data formats would
> > > involve updating all bridge and CSI-2 receiver drivers. I don't expect
> > > this to be a workable approach.
> > >
> > > Instead what I'm proposing is to use specific metadata formats on the
> > > sensor devices only, on internal pads (more about those soon) of the
> > > sensors, only visible in the UAPI, and then generic mbus formats along the
> > > pipeline and finally generic V4L2 metadata formats on the DMAs (specific
> > > to bit depth and packing). This would unsnarl the two, defining what data
> > > there is (specific mbus code) and how that is transported and packed
> > > (generic mbus codes and V4L2 formats).
> > >
> > > The user space would be required to "know" the path of that data from the
> > > sensor's internal pad to the V4L2 video node. I do not see this as these
> > > devices require at least some knowledge of the pipeline, i.e. hardware at
> > > hand. Separating what the data means and how it is packed may even be
> > > beneficial: it allows separating code that interprets the data (sensor
> > > internal mbus code) from the code that accesses it (packing).
> > >
> > > These formats are in practice line based, meaning that there may be
> > > padding at the end of the line, depending on the bus as well as the DMA.
> > > If non-line based formats are needed, it is always possible to set the
> > > "height" field to 1.
> >
> > One thing that may be worth considering or clarifying - for the case of
> > the BCM2835 Unicam CSI-2 device, we only have 2x DMA output channels. So
> > one will match image data packets, and the other will match "everything
> > else". Typically "everything else" would only be CSI-2 embedded data, but
> > in the case of the Raspberry Pi Camera v3 (IMX708), it includes embedded
> > data, PDAF data, and HDR histogram data. Each of these outputs can be
> > programmed to use a different packet ID in the sensor, but since Unicam
> > only has a single DMA for "everything else", it all gets dumped into one
> > metadata buffer. But given we know the exact structure of the data
> > streams, it's trivial for useland to find the right bits in this buffer.
> > Of course, other CSI-2 receivers with more DMA channels might allow these
> > streams to end up in their own buffers.
> >
> > Nothing in your series seems to stop us operating Unicam in this way,
> > particularly because there is no fixed definition of the data format for
> > V4L2_META_FMT_GENERIC_8. So I don't think it's a problem, but perhaps
> > it's worth documenting that the metadata might include multiple streams
> > from the sensor?
>
> I believe this happens on other hardware, too, indeed. Currently the
> documentation says that
>
>         Any number of routes from streams on sink pads towards streams on
>         source pads is allowed, to the extent supported by drivers. For
>         every stream on a source pad, however, only a single route is
>         allowed.
>
>         (Documentation/userspace-api/media/v4l/dev-subdev.rst)
>
> This probably needs to be changed to allow what you'd need?

Yes, that last sentence sounds like it would (artificially wrt your
series) limit
metadata buffers to only handle a single output stream.  However, I may have got
the context of the paragraph wrong as well :)

Regards,
Naush

>
> --
> Kind regards,
>
> Sakari Ailus

Re: [RFC 0/7] Generic line based metadata support, internal pads

[Naushir Patuck]

On Fri, 2 Jun 2023 at 10:12, Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> Hello,
>
> On Fri, Jun 02, 2023 at 08:54:35AM +0100, Naushir Patuck wrote:
> > Hi Sakari,
> >
> > Thank you for working on this.  Sensor metadata is something that Raspberry Pi
> > do make extensive use of, and our downstream changes to support it, although a
> > bit hacky, are not too dissimilar to your proposal here.
> >
> > On Fri, 5 May 2023 at 22:53, Sakari Ailus wrote:
> > >
> > > Hi folks,
> > >
> > > Here are a few patches to add support generic, line based metadata as well
> > > as internal source pads. While the amount of code is not very large, to
> > > the contrary it is quite small actually IMO, I presume what this is about
> > > and why it is being proposed requires some explaining.
> > >
> > > Metadata mbus codes and formats have existed for some time in V4L2. They
> > > however have been only used by drivers that produce the data itself and
> > > effectively this metadata has always been statistics of some sort (at
> > > least when it comes to ISPs). What is different here is that we intend to
> > > add support for metadata originating from camera sensors.
> > >
> > > Camera sensors produce different kinds of metadata, embedded data (usually
> > > register address--value pairs used to capture the frame, in a more or less
> > > sensor specific format), histograms (in a very sensor specific format),
> > > dark pixels etc.
>
> Optical dark pixels are image data, I wouldn't include them in the
> "metadata" category. They can of course be transmitted over a different
> stream, so they're relevant to the API being designed.
>
> > > The number of these formats is probably going to be about
> > > as large as image data formats if not larger, as the image data formats
> > > are much better standardised but a smaller subset of them will be
> > > supported by V4L2, at least initially but possibly much more in the long
> > > run.
>
> Strictly speaking, the number of metadata formats depends on how we
> define "format". Today, we can use the GREY pixel format to capture
> greyscale images in the visible spectrum, but also IR images, thermal
> images, or even depth images. They're all one pixel format. On the other
> hand, we have Y16 for greyscale visible and IR images, and Z16 for depth
> maps. It's already a mess, even without metadata :-)
>
> > > Having this many device specific formats would be a major problem for all
> > > the other drivers along that pipeline (not to mention the users of those
> > > drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2
> > > receiver drivers that have DMA: the poor driver developer would not only
> > > need to know camera sensor specific formats but to choose the specific
> > > packing of that format suitable for the DMA used by the hardware. It is
> > > unlikely many of these would ever get tested while being present on the
> > > driver API. Also adding new sensors with new embedded data formats would
> > > involve updating all bridge and CSI-2 receiver drivers. I don't expect
> > > this to be a workable approach.
>
> I'm glad we agree on this.
>
> > > Instead what I'm proposing is to use specific metadata formats on the
> > > sensor devices only, on internal pads (more about those soon) of the
> > > sensors, only visible in the UAPI, and then generic mbus formats along the
> > > pipeline and finally generic V4L2 metadata formats on the DMAs (specific
> > > to bit depth and packing). This would unsnarl the two, defining what data
> > > there is (specific mbus code) and how that is transported and packed
> > > (generic mbus codes and V4L2 formats).
>
> Decoupling the information needed by the kernel (e.g. are we
> transporting RAW8 or RAW10 from the sensor through the pipeline) from
> the information useful for userspace only (e.g. the sensor embedded data
> is encoding using register + value pairs, based on the IMX708 registers
> set) is a good idea. I expect the main question to be where to draw the
> line between those two categories. Some pieces of information may be
> useless to any processing block in the pipeline except for an odd block
> in the middle.
>
> This is, I believe, a problem similar to the CFA pattern. That
> information is useless for most devices, but required by the demosaicing
> block and some other blocks along the pipeline (colour gains for
> instance, or some Bayer statistics engines). We currently convey the CFA
> pattern in the media bus codes and pixel formats (e.g. SGRBG8 vs.
> SRGGB8) through the whole pipeline, while it could be conveyed out of
> band (e.g. exposed by the sensor using a control, and set on the devices
> that need it using a control as well).
>
> If we come up with a good solution for metadata (and I hope we will),
> maybe we'll be able to use a similar mechanism for CFA patterns,
> simplifying new drivers and userspace. Or maybe this will remain a pipe
> dream given the backward compatibility implications.
>
> > > The user space would be required to "know" the path of that data from the
> > > sensor's internal pad to the V4L2 video node. I do not see this as these
> > > devices require at least some knowledge of the pipeline, i.e. hardware at
> > > hand. Separating what the data means and how it is packed may even be
> > > beneficial: it allows separating code that interprets the data (sensor
> > > internal mbus code) from the code that accesses it (packing).
> > >
> > > These formats are in practice line based, meaning that there may be
> > > padding at the end of the line, depending on the bus as well as the DMA.
> > > If non-line based formats are needed, it is always possible to set the
> > > "height" field to 1.
> >
> > One thing that may be worth considering or clarifying - for the case of the
> > BCM2835 Unicam CSI-2 device, we only have 2x DMA output channels.  So one will
> > match image data packets, and the other will match "everything else".  Typically
> > "everything else" would only be CSI-2 embedded data, but in the case of the
> > Raspberry Pi Camera v3 (IMX708), it includes embedded data, PDAF data, and
> > HDR histogram data.  Each of these outputs can be programmed to use a different
> > packet ID in the sensor, but since Unicam only has a single DMA for "everything
> > else", it all gets dumped into one metadata buffer.  But given we know the exact
> > structure of the data streams, it's trivial for useland to find the right bits
> > in this buffer.  Of course, other CSI-2 receivers with more DMA channels might
> > allow these streams to end up in their own buffers.
> >
> > Nothing in your series seems to stop us operating Unicam in this way,
> > particularly because there is no fixed definition of the data format for
> > V4L2_META_FMT_GENERIC_8. So I don't think it's a problem, but perhaps it's worth
> > documenting that the metadata might include multiple streams from the sensor?
>
> Thanks for your feedback Naush. Would you consider reviewing the
> individual patches in the series ? :-)

Sure, I'll go through the patches and send some feedback.  This is after all a
change that I'm very much looking forward to :-)

Regards,
Naush

>
> > > The internal (source) pads are an alternative to source routes [1]. The
> > > source routes were not universally liked and I do have to say I like
> > > re-using existing interface concepts (pads and everything you can do with
> > > pads, including access format, selections etc.) wherever it makes sense,
> > > instead of duplicating functionality.
> > >
> > > Effectively internal source pads behave mostly just like sink pads, but
> > > they describe a flow of data that originates from a sub-device instead of
> > > arriving to a sub-device. The SUBDEV_S_ROUTING IOCTLs are used to enable
> > > and disable routes from internal source pads to sub-device's source pads.
> > > The subdev format IOCTLs are usable, too, so one can find which subdev
> > > format is available on given internal source pad.
> > >
> > > This set depends on these patches:
> > >
> > > <URL:https://lore.kernel.org/linux-media/20230505205416.55002-1-sakari.ailus@linux.intel.com/T/#t>
> > >
> > > I've also pushed these here and I'll keep updating the branch:
> > >
> > > <URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=metadata>
> > >
> > > Questions and comments are most welcome.
> > >
> > >
> > > [1] <URL:https://lore.kernel.org/linux-media/20220831141357.1396081-20-tomi.valkeinen@ideasonboard.com/>
> > >
> > > Sakari Ailus (7):
> > >   media: mc: Add INTERNAL_SOURCE pad type flag
> > >   media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs
> > >   media: uapi: v4l: Document source routes
> > >   media: mc: Check pad flag validity
> > >   media: uapi: Add generic serial metadata mbus formats
> > >   media: uapi: Add generic 8-bit metadata format definitions
> > >   media: v4l: Support line-based metadata capture
> > >
> > >  .../media/mediactl/media-types.rst            |   7 +
> > >  .../userspace-api/media/v4l/dev-meta.rst      |  15 +
> > >  .../userspace-api/media/v4l/dev-subdev.rst    |  18 +
> > >  .../userspace-api/media/v4l/meta-formats.rst  |   1 +
> > >  .../media/v4l/metafmt-generic.rst             | 317 ++++++++++++++++++
> > >  .../media/v4l/subdev-formats.rst              | 257 ++++++++++++++
> > >  .../media/v4l/vidioc-enum-fmt.rst             |   7 +
> > >  .../media/v4l/vidioc-subdev-g-routing.rst     |   5 +
> > >  drivers/media/mc/mc-entity.c                  |  20 +-
> > >  drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
> > >  drivers/media/v4l2-core/v4l2-subdev.c         |   6 +-
> > >  include/uapi/linux/media-bus-format.h         |   9 +
> > >  include/uapi/linux/media.h                    |   1 +
> > >  include/uapi/linux/v4l2-subdev.h              |   6 +-
> > >  include/uapi/linux/videodev2.h                |  19 ++
> > >  15 files changed, 691 insertions(+), 5 deletions(-)
> > >  create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
>
> --
> Regards,
>
> Laurent Pinchart

Re: [RFC 2/7] media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs

[Laurent Pinchart]

On Mon, May 08, 2023 at 03:24:10PM +0300, Sakari Ailus wrote:
> Moi,

こんにちは

> On Mon, May 08, 2023 at 01:14:07PM +0300, Tomi Valkeinen wrote:
> > On 06/05/2023 00:52, Sakari Ailus wrote:
> > > Take the new INTERNAL_SOURCE pad flag into account in validating routing
> > > IOCTL argument. Effectively this is a SINK pad in this respect. Due to the
> > > union there's no need to check for the particular name.
> > 
> > What union? The one you add in the next patch?
> 
> Oops. I think we can reorder the patches.
> 
> > As a concept the internal source pads sound good, and they work in practice
> > in my tests. But this union is what grates me a bit. We have a flag,
> > MEDIA_PAD_FL_INTERNAL_SOURCE, which tells which field of the union to use,
> > and then we go and do not use the new union field. Well, and also the
> > naming, as we normally have source and sink pads, but now we also have
> > internal source pads, which are actually identical to sink pads...
> 
> The union still should be used by the user space. We're testing flags here
> and those flags are the same independently of the INTERNAL_SOURCE flag.
> 
> I'm fine by not adding that union though, but for the user space I think
> it's better we have it: explaining that the sink_pad has a different
> meaning if that flag is set is harder than making it a union member.
> 
> > I understand the idea and reasoning, but the two points above do confuse me
> > and I'm sure would confuse others also.
> > 
> > I wonder if it would be less or more confusing to simplify this by just
> > adding a MEDIA_PAD_FL_INTERNAL, which could be applied to either a source or
> > a sink pad, and would prevent linking to it. The flag would indicate that
> > the stream from/to that pad is generated/consumed internally. (I don't know
> > when you would need an internal pad to consume data, but... who knows, the
> > need might pop up =).
> 
> This is another option. But I envision there will be more compatibility
> issues. Although... generally using such hardware requires knowledge of the
> device, and we haven't obviously had any support for devices needing this
> functionality in the tree. So in the end it might not matter much.
>
> > That would mean that an "internal sink pad" would generate a data stream,
> > i.e. it's a "source", but I think a normal sink pad is almost the same
> > anyway: when considering entity's internal routing, the normal sink pad is a
> > "source", and the same logic would apply with the internal pads too.
> > 
> > An internal sink pad that generates a stream is a bit more confusing than an
> > internal source pad, but I think that confusion is less than the rest of the
> > confusion in the code-side that comes with the internal source pads.
> 
> I prefer having the possible sources of the confusion in the framework than
> in the drivers. Therefore I think INTERNAL_SOURCE flag is a (slightly)
> better option.
> 
> I wonder what Llaurent thinks.

I like the idea of a MEDIA_PAD_FL_INTERNAL flag. That's actually how I
read patch 1/7, I didn't notice it used MEDIA_PAD_FL_INTERNAL*_SOURCE*
:-)

We could define MEDIA_PAD_FL_INTERNAL_SOURCE

#define MEDIA_PAD_FL_INTERNAL_SOURCE 	(MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL)

but I'm not sure it would be less confusing.

Regarding isolating the sources of confusion in the framework rather
than spreading them through drivers, I can't disagree, but I think that,
for raw camera sensors at least, the best option would be to create a
new camera sensor object with a much more tailored API than v4l2_subdev
(and of course wrapping that new object in a v4l2_subdev in the
framework). This won't address the other types of drivers, but I'm not
sure we'll get any in the foreseable future.

-- 
Regards,

Laurent Pinchart

Re: [RFC 2/7] media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs

[Laurent Pinchart]

On Fri, Jun 02, 2023 at 12:44:12PM +0300, Laurent Pinchart wrote:
> On Mon, May 08, 2023 at 03:24:10PM +0300, Sakari Ailus wrote:
> > Moi,
> 
> こんにちは
> 
> > On Mon, May 08, 2023 at 01:14:07PM +0300, Tomi Valkeinen wrote:
> > > On 06/05/2023 00:52, Sakari Ailus wrote:
> > > > Take the new INTERNAL_SOURCE pad flag into account in validating routing
> > > > IOCTL argument. Effectively this is a SINK pad in this respect. Due to the
> > > > union there's no need to check for the particular name.
> > > 
> > > What union? The one you add in the next patch?
> > 
> > Oops. I think we can reorder the patches.
> > 
> > > As a concept the internal source pads sound good, and they work in practice
> > > in my tests. But this union is what grates me a bit. We have a flag,
> > > MEDIA_PAD_FL_INTERNAL_SOURCE, which tells which field of the union to use,
> > > and then we go and do not use the new union field. Well, and also the
> > > naming, as we normally have source and sink pads, but now we also have
> > > internal source pads, which are actually identical to sink pads...
> > 
> > The union still should be used by the user space. We're testing flags here
> > and those flags are the same independently of the INTERNAL_SOURCE flag.
> > 
> > I'm fine by not adding that union though, but for the user space I think
> > it's better we have it: explaining that the sink_pad has a different
> > meaning if that flag is set is harder than making it a union member.
> > 
> > > I understand the idea and reasoning, but the two points above do confuse me
> > > and I'm sure would confuse others also.
> > > 
> > > I wonder if it would be less or more confusing to simplify this by just
> > > adding a MEDIA_PAD_FL_INTERNAL, which could be applied to either a source or
> > > a sink pad, and would prevent linking to it. The flag would indicate that
> > > the stream from/to that pad is generated/consumed internally. (I don't know
> > > when you would need an internal pad to consume data, but... who knows, the
> > > need might pop up =).
> > 
> > This is another option. But I envision there will be more compatibility
> > issues. Although... generally using such hardware requires knowledge of the
> > device, and we haven't obviously had any support for devices needing this
> > functionality in the tree. So in the end it might not matter much.
> >
> > > That would mean that an "internal sink pad" would generate a data stream,
> > > i.e. it's a "source", but I think a normal sink pad is almost the same
> > > anyway: when considering entity's internal routing, the normal sink pad is a
> > > "source", and the same logic would apply with the internal pads too.
> > > 
> > > An internal sink pad that generates a stream is a bit more confusing than an
> > > internal source pad, but I think that confusion is less than the rest of the
> > > confusion in the code-side that comes with the internal source pads.
> > 
> > I prefer having the possible sources of the confusion in the framework than
> > in the drivers. Therefore I think INTERNAL_SOURCE flag is a (slightly)
> > better option.
> > 
> > I wonder what Llaurent thinks.
> 
> I like the idea of a MEDIA_PAD_FL_INTERNAL flag. That's actually how I
> read patch 1/7, I didn't notice it used MEDIA_PAD_FL_INTERNAL*_SOURCE*
> :-)
> 
> We could define MEDIA_PAD_FL_INTERNAL_SOURCE
> 
> #define MEDIA_PAD_FL_INTERNAL_SOURCE 	(MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL)

One option would be to find terms different from sink and source in this
case. It would generate less confusion to map (e.g., not a really good
name) MEDIA_PAD_FL_INTERNAL_PRODUCER to MEDIA_PAD_FL_SINK and to the
sink_pad field than using MEDIA_PAD_FL_INTERNAL_SOURCE.

> but I'm not sure it would be less confusing.
> 
> Regarding isolating the sources of confusion in the framework rather
> than spreading them through drivers, I can't disagree, but I think that,
> for raw camera sensors at least, the best option would be to create a
> new camera sensor object with a much more tailored API than v4l2_subdev
> (and of course wrapping that new object in a v4l2_subdev in the
> framework). This won't address the other types of drivers, but I'm not
> sure we'll get any in the foreseable future.

-- 
Regards,

Laurent Pinchart

Re: [RFC 3/7] media: uapi: v4l: Document source routes

[Laurent Pinchart]

Hello,

On Mon, May 08, 2023 at 07:35:04PM +0300, Tomi Valkeinen wrote:
> On 08/05/2023 19:26, Sakari Ailus wrote:
> > On Mon, May 08, 2023 at 01:33:24PM +0300, Tomi Valkeinen wrote:
> >> On 06/05/2023 00:52, Sakari Ailus wrote:
> >>> Document how internal pads are used on source routes.
> >>>
> >>> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> >>> ---
> >>>    .../userspace-api/media/v4l/dev-subdev.rst     | 18 ++++++++++++++++++
> >>>    .../media/v4l/vidioc-subdev-g-routing.rst      |  5 +++++
> >>>    include/uapi/linux/v4l2-subdev.h               |  6 +++++-
> >>>    3 files changed, 28 insertions(+), 1 deletion(-)
> >>>
> >>> diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> >>> index a4f1df7093e8..395e06d2f0f2 100644
> >>> --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> >>> +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> >>> @@ -551,6 +551,24 @@ A stream at a specific point in the media pipeline is identified by the
> >>>    sub-device and a (pad, stream) pair. For sub-devices that do not support
> >>>    multiplexed streams the 'stream' field is always 0.
> >>> +.. _v4l2-subdev-source-routes:
> >>> +
> >>> +Source routes
> >>> +^^^^^^^^^^^^^
> >>> +
> >>> +Cases where a single sub-device pad is a source of multiple streams are special
> >>> +as there is no sink pad for such a route. In those cases, an internal pad is
> >>> +used as the sink pad. Such pads have the :ref:`MEDIA_PAD_FL_INTERNAL_SOURCE
> >>> +<MEDIA-PAD-FL-INTERNAL-SOURCE>` flag set.
> >>> +
> >>> +Internal pads have all the properties of a sink pad in such case, including
> >>> +formats and selections. The format in this case is the source format of the
> >>> +stream. An internal pad always has a single stream only (0).
> >>> +
> >>> +Generally source routes are not modifiable but they can be activated and
> >>> +inactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE
> >>> +<v4l2-subdev-routing-flags>` flag, depending on driver capabilities.
> >>> +
> >>
> >> I find the above chapter a bit difficult to read, but I think we need to
> >> conclude on the internal-pad vs internal-source-pad discussion first.
> >> However, one point/question:

Agreed, it's far from being clear :-( Even the first sentence, "Cases
where a single sub-device pad is a source of multiple streams are
special" is confusing, having a subdev source pad carrying multiple
streams isn't special, it is found in, for instance, FPD-Link or GMSL
receivers that transmit multiple streams from different cameras over a
single CSI-2 output. This may not be what Sakari meant here, but it can
be understood that way.

We need more than 3 paragraphs here, and we need a very clear example
with a diagram. I'd recommend using a camera sensor that produces image
data and embedded data for the example, as that's a common real-life use
case. The text should present the example, explain what the problem is,
and then explain how internal pads fix it. Then it can expand on the
features and usage of internal pads in a more generic way.

> >> You write that there's only one stream in an internal pad. I wonder if
> >> that's a good or a necessary limitation... Do you see that allowing multiple
> >> streams brings extra complexity? It's still up to each driver to decide how
> >> many streams they support (most would just support a single one), so each
> >> driver can easily enforce that limit.
> > 
> > Good question. As we don't seem to have a tangible reason to allow it, I'd
> > deny it until there's a use case.
> > 
> > Or did you have a use case in mind?
> > 
> > I thought indicating which streams will be bound together (i.e. either are
> > all disabled or enabled) could be one, but I'm not sure we need that at the
> > moment at least.
> 
> I don't have nothing concrete in mind. Maybe a TPG which also generates 
> some kind of metadata. But that could be implemented as two internal pads.
> 
> >> I'm fine with starting with only single-stream support, but if we later need
> >> to loosen that restriction, I wonder if it'll be difficult if we have
> >> specified that there can be only one stream. I.e. will the drivers and the
> >> userspace depend on that limit.
> > 
> > We can always allow what wasn't allowed previously so that's a non-issue,
> > really. But it needs to bring new functionality, otherwise there's no
> > reason to do that.
> 
> It's not always that easy. If the drivers and the userspace expect that 
> there's a single route with ID 0, and then with a new kernel there are 
> more streams or the single stream is ID 1, things could go wrong.

I agree with Tomi here. On the kernel side it should be fine (with an
unknown amount of pain), but I'd consider this as a userspace API
breakage. If the specifications indicates that only a single stream can
be used, applications may rely on that, and things could go wrong if new
streams are added.

We can restrict the kernel implementation to a single stream, but the
userspace API has to indicate that multiple streams can co-exist if we
want to allow that in the future.

-- 
Regards,

Laurent Pinchart

Re: [RFC 3/7] media: uapi: v4l: Document source routes

[Laurent Pinchart]

On Mon, May 08, 2023 at 08:41:42PM +0300, Sakari Ailus wrote:
> On Mon, May 08, 2023 at 07:35:04PM +0300, Tomi Valkeinen wrote:
> > On 08/05/2023 19:26, Sakari Ailus wrote:
> > > On Mon, May 08, 2023 at 01:33:24PM +0300, Tomi Valkeinen wrote:
> > > > On 06/05/2023 00:52, Sakari Ailus wrote:
> > > > > Document how internal pads are used on source routes.
> > > > > 
> > > > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > > > ---
> > > > >    .../userspace-api/media/v4l/dev-subdev.rst     | 18 ++++++++++++++++++
> > > > >    .../media/v4l/vidioc-subdev-g-routing.rst      |  5 +++++
> > > > >    include/uapi/linux/v4l2-subdev.h               |  6 +++++-
> > > > >    3 files changed, 28 insertions(+), 1 deletion(-)
> > > > > 
> > > > > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > > > index a4f1df7093e8..395e06d2f0f2 100644
> > > > > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > > > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > > > @@ -551,6 +551,24 @@ A stream at a specific point in the media pipeline is identified by the
> > > > >    sub-device and a (pad, stream) pair. For sub-devices that do not support
> > > > >    multiplexed streams the 'stream' field is always 0.
> > > > > +.. _v4l2-subdev-source-routes:
> > > > > +
> > > > > +Source routes
> > > > > +^^^^^^^^^^^^^
> > > > > +
> > > > > +Cases where a single sub-device pad is a source of multiple streams are special
> > > > > +as there is no sink pad for such a route. In those cases, an internal pad is
> > > > > +used as the sink pad. Such pads have the :ref:`MEDIA_PAD_FL_INTERNAL_SOURCE
> > > > > +<MEDIA-PAD-FL-INTERNAL-SOURCE>` flag set.
> > > > > +
> > > > > +Internal pads have all the properties of a sink pad in such case, including
> > > > > +formats and selections. The format in this case is the source format of the
> > > > > +stream. An internal pad always has a single stream only (0).
> > > > > +
> > > > > +Generally source routes are not modifiable but they can be activated and
> > > > > +inactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > > > > +<v4l2-subdev-routing-flags>` flag, depending on driver capabilities.
> > > > > +
> > > > 
> > > > I find the above chapter a bit difficult to read, but I think we need to
> > > > conclude on the internal-pad vs internal-source-pad discussion first.
> > > > However, one point/question:
> > > > 
> > > > You write that there's only one stream in an internal pad. I wonder if
> > > > that's a good or a necessary limitation... Do you see that allowing multiple
> > > > streams brings extra complexity? It's still up to each driver to decide how
> > > > many streams they support (most would just support a single one), so each
> > > > driver can easily enforce that limit.
> > > 
> > > Good question. As we don't seem to have a tangible reason to allow it, I'd
> > > deny it until there's a use case.
> > > 
> > > Or did you have a use case in mind?
> > > 
> > > I thought indicating which streams will be bound together (i.e. either are
> > > all disabled or enabled) could be one, but I'm not sure we need that at the
> > > moment at least.
> > 
> > I don't have nothing concrete in mind. Maybe a TPG which also generates some
> > kind of metadata. But that could be implemented as two internal pads.
> > 
> > > > I'm fine with starting with only single-stream support, but if we later need
> > > > to loosen that restriction, I wonder if it'll be difficult if we have
> > > > specified that there can be only one stream. I.e. will the drivers and the
> > > > userspace depend on that limit.
> > > 
> > > We can always allow what wasn't allowed previously so that's a non-issue,
> > > really. But it needs to bring new functionality, otherwise there's no
> > > reason to do that.
> > 
> > It's not always that easy. If the drivers and the userspace expect that
> > there's a single route with ID 0, and then with a new kernel there are more
> > streams or the single stream is ID 1, things could go wrong.
> 
> Other drivers are a non-issue IMO as this isn't visible to other drivers.
> 
> I think for user space this could be an issue if it were entirely generic.
> But that's probably not going to be the case, is it? I don't mean test
> programs.

I could imagine this breaking libcamera for instance.

-- 
Regards,

Laurent Pinchart

Re: [RFC 4/7] media: mc: Check pad flag validity

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Sat, May 06, 2023 at 12:52:54AM +0300, Sakari Ailus wrote:
> Check the validity of pad flags on entity init. Exactly one of the flags
> must be set.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  drivers/media/mc/mc-entity.c | 12 +++++++++++-
>  1 file changed, 11 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> index ed99160a2487..c1bc48c4d239 100644
> --- a/drivers/media/mc/mc-entity.c
> +++ b/drivers/media/mc/mc-entity.c
> @@ -197,6 +197,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
>  	struct media_device *mdev = entity->graph_obj.mdev;
>  	struct media_pad *iter;
>  	unsigned int i = 0;
> +	int ret = 0;
>  
>  	if (num_pads >= MEDIA_ENTITY_MAX_PADS)
>  		return -E2BIG;
> @@ -210,6 +211,15 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
>  	media_entity_for_each_pad(entity, iter) {
>  		iter->entity = entity;
>  		iter->index = i++;
> +
> +		if (hweight32(iter->flags & (MEDIA_PAD_FL_SINK |
> +					     MEDIA_PAD_FL_SOURCE |
> +					     MEDIA_PAD_FL_INTERNAL_SOURCE))
> +		    != 1) {

I would add this check as patch 1/7 in this series, with FL_SINK an
FL_SOURCE only, and then extend it in the patch where you introduce
FL_INTERNAL_SOURCE.

> +			ret = -EINVAL;
> +			break;
> +		}
> +
>  		if (mdev)
>  			media_gobj_create(mdev, MEDIA_GRAPH_PAD,
>  					  &iter->graph_obj);
> @@ -218,7 +228,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
>  	if (mdev)
>  		mutex_unlock(&mdev->graph_mutex);
>  
> -	return 0;
> +	return ret;
>  }
>  EXPORT_SYMBOL_GPL(media_entity_pads_init);
>  

-- 
Regards,

Laurent Pinchart

Re: [RFC 5/7] media: uapi: Add generic serial metadata mbus formats

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Sat, May 06, 2023 at 12:52:55AM +0300, Sakari Ailus wrote:
> Add generic serial metadata mbus formats. These formats describe data
> width and packing but not the content itself. The reason for specifying
> such formats is that the formats as such are fairly device specific but
> they are still handled by CSI-2 receiver drivers that should not be aware
> of device specific formats. What makes generic metadata formats possible
> is that these formats are parsed by software only, after capturing the
> data to system memory.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../media/v4l/subdev-formats.rst              | 257 ++++++++++++++++++
>  include/uapi/linux/media-bus-format.h         |   9 +
>  2 files changed, 266 insertions(+)
> 
> diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> index a3a35eeed708..1492fff58426 100644
> --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
> +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> @@ -8234,3 +8234,260 @@ The following table lists the existing metadata formats.
>  	both sides of the link and the bus format is a fixed
>  	metadata format that is not configurable from userspace.
>  	Width and height will be set to 0 for this format.
> +
> +Generic Serial Metadata Formats
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +Generic serial metadata formats are used on serial busses where the actual data

s/busses/buses/

> +content is more or less device specific but the data is transmitted and received
> +by multiple devices that do not process the data in any way, simply writing
> +it to system memory for processing in software at the end of the pipeline.
> +
> +The more specific variant describing the actual data is used on the internal
> +source pad of the originating sub-device.

What do you mean by "more specific variant" here ? Please include an
example in the documentation.

I'm not sure I like mentioning internal source pads here, are we
guaranteed that metadata will always originate from an internal source
pad ?

> +
> +"b" in an array cell signifies a byte of data, followed by the number of byte
> +and finally the bit number in subscript. "p" indicates a padding bit.
> +
> +.. _media-bus-format-generic-meta:
> +
> +.. cssclass: longtable
> +
> +.. flat-table:: Generic Serial Metadata Formats
> +    :header-rows:  2
> +    :stub-columns: 0
> +
> +    * - Identifier
> +      - Code
> +      -
> +      - :cspan:`23` Data organization
> +    * -
> +      -
> +      - Bit
> +      - 23
> +      - 22
> +      - 21
> +      - 20
> +      - 19
> +      - 18
> +      - 17
> +      - 16
> +      - 15
> +      - 14
> +      - 13
> +      - 12
> +      - 11
> +      - 10
> +      - 9
> +      - 8
> +      - 7
> +      - 6
> +      - 5
> +      - 4
> +      - 3
> +      - 2
> +      - 1
> +      - 0
> +    * .. _MEDIA-BUS-FMT-META-1X8-8:
> +
> +      - MEDIA_BUS_FMT_META_1X8_8
> +      - 0x8001
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      - b0\ :sub:`7`
> +      - b0\ :sub:`6`
> +      - b0\ :sub:`5`
> +      - b0\ :sub:`4`
> +      - b0\ :sub:`3`
> +      - b0\ :sub:`2`
> +      - b0\ :sub:`1`
> +      - b0\ :sub:`0`
> +    * .. _MEDIA-BUS-FMT-META-1X8-10:
> +
> +      - MEDIA_BUS_FMT_META_1X8_10
> +      - 0x8002
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      - b0\ :sub:`7`
> +      - b0\ :sub:`6`
> +      - b0\ :sub:`5`
> +      - b0\ :sub:`4`
> +      - b0\ :sub:`3`
> +      - b0\ :sub:`2`
> +      - b0\ :sub:`1`
> +      - b0\ :sub:`0`
> +      - p
> +      - p
> +    * .. _MEDIA-BUS-FMT-META-1X8-12:
> +
> +      - MEDIA_BUS_FMT_META_1X8_12
> +      - 0x8003
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      - b0\ :sub:`7`
> +      - b0\ :sub:`6`
> +      - b0\ :sub:`5`
> +      - b0\ :sub:`4`
> +      - b0\ :sub:`3`
> +      - b0\ :sub:`2`
> +      - b0\ :sub:`1`
> +      - b0\ :sub:`0`
> +      - p
> +      - p
> +      - p
> +      - p
> +    * .. _MEDIA-BUS-FMT-META-1X8-14:
> +
> +      - MEDIA_BUS_FMT_META_1X8_14
> +      - 0x8004
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      - b0\ :sub:`7`
> +      - b0\ :sub:`6`
> +      - b0\ :sub:`5`
> +      - b0\ :sub:`4`
> +      - b0\ :sub:`3`
> +      - b0\ :sub:`2`
> +      - b0\ :sub:`1`
> +      - b0\ :sub:`0`
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +    * .. _MEDIA-BUS-FMT-META-1X8-16:
> +
> +      - MEDIA_BUS_FMT_META_1X8_16
> +      - 0x8005
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      - b0\ :sub:`7`
> +      - b0\ :sub:`6`
> +      - b0\ :sub:`5`
> +      - b0\ :sub:`4`
> +      - b0\ :sub:`3`
> +      - b0\ :sub:`2`
> +      - b0\ :sub:`1`
> +      - b0\ :sub:`0`
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +    * .. _MEDIA-BUS-FMT-META-1X8-20:
> +
> +      - MEDIA_BUS_FMT_META_1X8_20
> +      - 0x8007
> +      -
> +      -
> +      -
> +      -
> +      -
> +      - b0\ :sub:`7`
> +      - b0\ :sub:`6`
> +      - b0\ :sub:`5`
> +      - b0\ :sub:`4`
> +      - b0\ :sub:`3`
> +      - b0\ :sub:`2`
> +      - b0\ :sub:`1`
> +      - b0\ :sub:`0`
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +    * .. _MEDIA-BUS-FMT-META-1X8-24:
> +
> +      - MEDIA_BUS_FMT_META_1X8_24
> +      - 0x8009
> +      -
> +      - b0\ :sub:`7`
> +      - b0\ :sub:`6`
> +      - b0\ :sub:`5`
> +      - b0\ :sub:`4`
> +      - b0\ :sub:`3`
> +      - b0\ :sub:`2`
> +      - b0\ :sub:`1`
> +      - b0\ :sub:`0`
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
> index a03c543cb072..722463523bbd 100644
> --- a/include/uapi/linux/media-bus-format.h
> +++ b/include/uapi/linux/media-bus-format.h
> @@ -173,4 +173,13 @@
>   */
>  #define MEDIA_BUS_FMT_METADATA_FIXED		0x7001
>  
> +/* Generic line based metadata formats for serial buses. Next is 0x800b. */
> +#define MEDIA_BUS_FMT_META_1X8_8		0x8001
> +#define MEDIA_BUS_FMT_META_1X8_10		0x8002
> +#define MEDIA_BUS_FMT_META_1X8_12		0x8003
> +#define MEDIA_BUS_FMT_META_1X8_14		0x8004
> +#define MEDIA_BUS_FMT_META_1X8_16		0x8005
> +#define MEDIA_BUS_FMT_META_1X8_20		0x8007
> +#define MEDIA_BUS_FMT_META_1X8_24		0x8009

We've discussed this before privately, it's time to come to a conclusion
:-)

My preference would be 

#define MEDIA_BUS_FMT_META_RAW8			0x8001
#define MEDIA_BUS_FMT_META_RAW10		0x8002
#define MEDIA_BUS_FMT_META_RAW12		0x8003
#define MEDIA_BUS_FMT_META_RAW14		0x8004
#define MEDIA_BUS_FMT_META_RAW16		0x8005
#define MEDIA_BUS_FMT_META_RAW20		0x8007
#define MEDIA_BUS_FMT_META_RAW24		0x8009

without defining the contents of the data (that is, no mention of
padding bits).

> +
>  #endif /* __LINUX_MEDIA_BUS_FORMAT_H */

-- 
Regards,

Laurent Pinchart

Re: [RFC 7/7] media: v4l: Support line-based metadata capture

[Laurent Pinchart]

Hi Sakari,

Thank you for the patch.

On Sat, May 06, 2023 at 12:52:57AM +0300, Sakari Ailus wrote:
> many camera sensors, among other devices, transmit embedded data and image

s/many/Many/

> data for each CSI-2 frame. This embedded data typically contains register
> configuration of the sensor that has been used to capture the image data
> of the same frame.
> 
> The embedded data is received by the CSI-2 receiver and has the same
> properties as the image data, including that it is line based: it has
> width, height and bytesperline (stride).
> 
> Add these fields to struct v4l2_meta_format and document them.
> 
> Also add V4L2_FMT_FLAG_META_LINE_BASED to tell a given format is
> line-based i.e. these fields of struct v4l2_meta_format are valid for it.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../userspace-api/media/v4l/dev-meta.rst          | 15 +++++++++++++++
>  .../userspace-api/media/v4l/vidioc-enum-fmt.rst   |  7 +++++++
>  include/uapi/linux/videodev2.h                    | 10 ++++++++++
>  3 files changed, 32 insertions(+)
> 
> diff --git a/Documentation/userspace-api/media/v4l/dev-meta.rst b/Documentation/userspace-api/media/v4l/dev-meta.rst
> index 0e7e1ee1471a..7d3a64514db0 100644
> --- a/Documentation/userspace-api/media/v4l/dev-meta.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-meta.rst
> @@ -65,3 +65,18 @@ to 0.
>        - ``buffersize``
>        - Maximum buffer size in bytes required for data. The value is set by the
>          driver.
> +    * - __u32
> +      - ``width``
> +      - Width of a line of metadata in bytes. Valid when :c:type`v4l2_fmtdesc`

This departs from pixel formats, where the width is defined in pixels. I
wonder what the implications will be for userspace. Seeing one
implementation, both in a kernel driver and in libcamera, will help
validating the API.

> +	flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
> +	:c:func:`VIDIOC_ENUM_FMT`.
> +    * - __u32
> +      - ``height``
> +      - Height of a line of metadata in bytes. Valid when :c:type`v4l2_fmtdesc`

The "height of a line" seems like a weird concept, especially if the
height is expressed in bytes. I assume this is a bad copy&paste.

> +	flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
> +	:c:func:`VIDIOC_ENUM_FMT`.
> +    * - __u32
> +      - ``bytesperlines``
> +      - Offset in bytes between the beginning of two consecutive lines. Valid
> +	when :c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is
> +	set, otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> index 000c154b0f98..6d7664345a4e 100644
> --- a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> +++ b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> @@ -227,6 +227,13 @@ the ``mbus_code`` field is handled differently:
>  	The application can ask to configure the quantization of the capture
>  	device when calling the :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl with
>  	:ref:`V4L2_PIX_FMT_FLAG_SET_CSC <v4l2-pix-fmt-flag-set-csc>` set.
> +    * - ``V4L2_FMT_FLAG_META_LINE_BASED``
> +      - 0x0200
> +      - The metadata format is line-based. In this case the ``width``,
> +	``height`` and ``bytesperline`` fields of :c:type:`v4l2_meta_format` are
> +	valid. The buffer consists of ``height`` lines, each having ``width``
> +	bytes of data and offset between the beginning of each two consecutive
> +	lines is ``bytesperline``.

If we add width and height for metadata formats, does it mean that
drivers have to (or can) implement VIDIOC_ENUM_FRAMESIZES ? This should
be documented.

>  
>  Return Value
>  ============
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index adcbdc15dcdb..3681b2c15901 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -873,6 +873,7 @@ struct v4l2_fmtdesc {
>  #define V4L2_FMT_FLAG_CSC_YCBCR_ENC		0x0080
>  #define V4L2_FMT_FLAG_CSC_HSV_ENC		V4L2_FMT_FLAG_CSC_YCBCR_ENC
>  #define V4L2_FMT_FLAG_CSC_QUANTIZATION		0x0100
> +#define V4L2_FMT_FLAG_META_LINE_BASED		0x0200
>  
>  	/* Frame Size and frame rate enumeration */
>  /*
> @@ -2407,10 +2408,19 @@ struct v4l2_sdr_format {
>   * struct v4l2_meta_format - metadata format definition
>   * @dataformat:		little endian four character code (fourcc)
>   * @buffersize:		maximum size in bytes required for data
> + * @width:		number of bytes of data per line (valid for line based
> + *			formats only, see format documentation)
> + * @height:		number of lines of data per buffer (valid for line based
> + *			formats only)
> + * @bytesperline:	offset between the beginnings of two adjacent lines in
> + *			bytes (valid for line based formats only)
>   */
>  struct v4l2_meta_format {
>  	__u32				dataformat;
>  	__u32				buffersize;
> +	__u32				width;
> +	__u32				height;
> +	__u32				bytesperline;
>  } __attribute__ ((packed));
>  
>  /**

-- 
Regards,

Laurent Pinchart

Re: [RFC 0/7] Generic line based metadata support, internal pads

[Sakari Ailus]

Hi Naush,

On Fri, Jun 02, 2023 at 10:35:08AM +0100, Naushir Patuck wrote:
> Hi Sakari,
> 
> On Fri, 2 Jun 2023 at 09:46, Sakari Ailus <sakari.ailus@linux.intel.com> wrote:
> >
> > Hi Naushir,
> >
> > On Fri, Jun 02, 2023 at 08:54:35AM +0100, Naushir Patuck wrote:
> > > Hi Sakari,
> > >
> > > Thank you for working on this. Sensor metadata is something that
> > > Raspberry Pi do make extensive use of, and our downstream changes to
> > > support it, although a bit hacky, are not too dissimilar to your proposal
> > > here.
> > >
> > > On Fri, 5 May 2023 at 22:53, Sakari Ailus <sakari.ailus@linux.intel.com> wrote:
> > > >
> > > > Hi folks,
> > > >
> > > > Here are a few patches to add support generic, line based metadata as well
> > > > as internal source pads. While the amount of code is not very large, to
> > > > the contrary it is quite small actually IMO, I presume what this is about
> > > > and why it is being proposed requires some explaining.
> > > >
> > > > Metadata mbus codes and formats have existed for some time in V4L2. They
> > > > however have been only used by drivers that produce the data itself and
> > > > effectively this metadata has always been statistics of some sort (at
> > > > least when it comes to ISPs). What is different here is that we intend to
> > > > add support for metadata originating from camera sensors.
> > > >
> > > > Camera sensors produce different kinds of metadata, embedded data (usually
> > > > register address--value pairs used to capture the frame, in a more or less
> > > > sensor specific format), histograms (in a very sensor specific format),
> > > > dark pixels etc. The number of these formats is probably going to be about
> > > > as large as image data formats if not larger, as the image data formats
> > > > are much better standardised but a smaller subset of them will be
> > > > supported by V4L2, at least initially but possibly much more in the long
> > > > run.
> > > >
> > > > Having this many device specific formats would be a major problem for all
> > > > the other drivers along that pipeline (not to mention the users of those
> > > > drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2
> > > > receiver drivers that have DMA: the poor driver developer would not only
> > > > need to know camera sensor specific formats but to choose the specific
> > > > packing of that format suitable for the DMA used by the hardware. It is
> > > > unlikely many of these would ever get tested while being present on the
> > > > driver API. Also adding new sensors with new embedded data formats would
> > > > involve updating all bridge and CSI-2 receiver drivers. I don't expect
> > > > this to be a workable approach.
> > > >
> > > > Instead what I'm proposing is to use specific metadata formats on the
> > > > sensor devices only, on internal pads (more about those soon) of the
> > > > sensors, only visible in the UAPI, and then generic mbus formats along the
> > > > pipeline and finally generic V4L2 metadata formats on the DMAs (specific
> > > > to bit depth and packing). This would unsnarl the two, defining what data
> > > > there is (specific mbus code) and how that is transported and packed
> > > > (generic mbus codes and V4L2 formats).
> > > >
> > > > The user space would be required to "know" the path of that data from the
> > > > sensor's internal pad to the V4L2 video node. I do not see this as these
> > > > devices require at least some knowledge of the pipeline, i.e. hardware at
> > > > hand. Separating what the data means and how it is packed may even be
> > > > beneficial: it allows separating code that interprets the data (sensor
> > > > internal mbus code) from the code that accesses it (packing).
> > > >
> > > > These formats are in practice line based, meaning that there may be
> > > > padding at the end of the line, depending on the bus as well as the DMA.
> > > > If non-line based formats are needed, it is always possible to set the
> > > > "height" field to 1.
> > >
> > > One thing that may be worth considering or clarifying - for the case of
> > > the BCM2835 Unicam CSI-2 device, we only have 2x DMA output channels. So
> > > one will match image data packets, and the other will match "everything
> > > else". Typically "everything else" would only be CSI-2 embedded data, but
> > > in the case of the Raspberry Pi Camera v3 (IMX708), it includes embedded
> > > data, PDAF data, and HDR histogram data. Each of these outputs can be
> > > programmed to use a different packet ID in the sensor, but since Unicam
> > > only has a single DMA for "everything else", it all gets dumped into one
> > > metadata buffer. But given we know the exact structure of the data
> > > streams, it's trivial for useland to find the right bits in this buffer.
> > > Of course, other CSI-2 receivers with more DMA channels might allow these
> > > streams to end up in their own buffers.
> > >
> > > Nothing in your series seems to stop us operating Unicam in this way,
> > > particularly because there is no fixed definition of the data format for
> > > V4L2_META_FMT_GENERIC_8. So I don't think it's a problem, but perhaps
> > > it's worth documenting that the metadata might include multiple streams
> > > from the sensor?
> >
> > I believe this happens on other hardware, too, indeed. Currently the
> > documentation says that
> >
> >         Any number of routes from streams on sink pads towards streams on
> >         source pads is allowed, to the extent supported by drivers. For
> >         every stream on a source pad, however, only a single route is
> >         allowed.
> >
> >         (Documentation/userspace-api/media/v4l/dev-subdev.rst)
> >
> > This probably needs to be changed to allow what you'd need?
> 
> Yes, that last sentence sounds like it would (artificially wrt your
> series) limit metadata buffers to only handle a single output stream.
> However, I may have got the context of the paragraph wrong as well :)

That was exactly the purpose: I wanted to make sure we didn't allow this
without thinking what other implications it could have --- for instance
what you also mentioned, the V4L2 format of the related buffer.

-- 
Kind regards,

Sakari Ailus

Re: [RFC 2/7] media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs

[Sakari Ailus]

Heippa Tomi, Laurent,

On Fri, Jun 02, 2023 at 12:46:50PM +0300, Laurent Pinchart wrote:
> On Fri, Jun 02, 2023 at 12:44:12PM +0300, Laurent Pinchart wrote:
> > On Mon, May 08, 2023 at 03:24:10PM +0300, Sakari Ailus wrote:
> > > Moi,
> > 
> > こんにちは
> > 
> > > On Mon, May 08, 2023 at 01:14:07PM +0300, Tomi Valkeinen wrote:
> > > > On 06/05/2023 00:52, Sakari Ailus wrote:
> > > > > Take the new INTERNAL_SOURCE pad flag into account in validating routing
> > > > > IOCTL argument. Effectively this is a SINK pad in this respect. Due to the
> > > > > union there's no need to check for the particular name.
> > > > 
> > > > What union? The one you add in the next patch?
> > > 
> > > Oops. I think we can reorder the patches.
> > > 
> > > > As a concept the internal source pads sound good, and they work in practice
> > > > in my tests. But this union is what grates me a bit. We have a flag,
> > > > MEDIA_PAD_FL_INTERNAL_SOURCE, which tells which field of the union to use,
> > > > and then we go and do not use the new union field. Well, and also the
> > > > naming, as we normally have source and sink pads, but now we also have
> > > > internal source pads, which are actually identical to sink pads...
> > > 
> > > The union still should be used by the user space. We're testing flags here
> > > and those flags are the same independently of the INTERNAL_SOURCE flag.
> > > 
> > > I'm fine by not adding that union though, but for the user space I think
> > > it's better we have it: explaining that the sink_pad has a different
> > > meaning if that flag is set is harder than making it a union member.
> > > 
> > > > I understand the idea and reasoning, but the two points above do confuse me
> > > > and I'm sure would confuse others also.
> > > > 
> > > > I wonder if it would be less or more confusing to simplify this by just
> > > > adding a MEDIA_PAD_FL_INTERNAL, which could be applied to either a source or
> > > > a sink pad, and would prevent linking to it. The flag would indicate that
> > > > the stream from/to that pad is generated/consumed internally. (I don't know
> > > > when you would need an internal pad to consume data, but... who knows, the
> > > > need might pop up =).
> > > 
> > > This is another option. But I envision there will be more compatibility
> > > issues. Although... generally using such hardware requires knowledge of the
> > > device, and we haven't obviously had any support for devices needing this
> > > functionality in the tree. So in the end it might not matter much.
> > >
> > > > That would mean that an "internal sink pad" would generate a data stream,
> > > > i.e. it's a "source", but I think a normal sink pad is almost the same
> > > > anyway: when considering entity's internal routing, the normal sink pad is a
> > > > "source", and the same logic would apply with the internal pads too.
> > > > 
> > > > An internal sink pad that generates a stream is a bit more confusing than an
> > > > internal source pad, but I think that confusion is less than the rest of the
> > > > confusion in the code-side that comes with the internal source pads.
> > > 
> > > I prefer having the possible sources of the confusion in the framework than
> > > in the drivers. Therefore I think INTERNAL_SOURCE flag is a (slightly)
> > > better option.
> > > 
> > > I wonder what Llaurent thinks.
> > 
> > I like the idea of a MEDIA_PAD_FL_INTERNAL flag. That's actually how I
> > read patch 1/7, I didn't notice it used MEDIA_PAD_FL_INTERNAL*_SOURCE*
> > :-)
> > 
> > We could define MEDIA_PAD_FL_INTERNAL_SOURCE
> > 
> > #define MEDIA_PAD_FL_INTERNAL_SOURCE 	(MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL)
> 
> One option would be to find terms different from sink and source in this
> case. It would generate less confusion to map (e.g., not a really good
> name) MEDIA_PAD_FL_INTERNAL_PRODUCER to MEDIA_PAD_FL_SINK and to the
> sink_pad field than using MEDIA_PAD_FL_INTERNAL_SOURCE.

I don't think this helps as you'd still be accessing the sink pad related
fields that are there for another purpose.

Alternatively I'd have the (plain) INTERNAL flag and drop the union,
without masking or adding compound flags.

I can switch to that if you promise not to change your mind again. ;-)

> 
> > but I'm not sure it would be less confusing.
> > 
> > Regarding isolating the sources of confusion in the framework rather
> > than spreading them through drivers, I can't disagree, but I think that,
> > for raw camera sensors at least, the best option would be to create a
> > new camera sensor object with a much more tailored API than v4l2_subdev
> > (and of course wrapping that new object in a v4l2_subdev in the
> > framework). This won't address the other types of drivers, but I'm not
> > sure we'll get any in the foreseable future.
> 

-- 
Terveisin,

Sakari Ailus

Re: [RFC 1/7] media: mc: Add INTERNAL_SOURCE pad type flag

[Sakari Ailus]

Hi Laurent,

On Fri, Jun 02, 2023 at 12:18:44PM +0300, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.
> 
> On Sat, May 06, 2023 at 12:52:51AM +0300, Sakari Ailus wrote:
> > Internal source pads will be used as routing endpoints in V4L2
> > [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
> > 
> > Also prevent creating links to pads that have been flagged as internal.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  .../userspace-api/media/mediactl/media-types.rst          | 7 +++++++
> >  drivers/media/mc/mc-entity.c                              | 8 +++++++-
> >  include/uapi/linux/media.h                                | 1 +
> >  3 files changed, 15 insertions(+), 1 deletion(-)
> > 
> > diff --git a/Documentation/userspace-api/media/mediactl/media-types.rst b/Documentation/userspace-api/media/mediactl/media-types.rst
> > index 0ffeece1e0c8..c724139ad46c 100644
> > --- a/Documentation/userspace-api/media/mediactl/media-types.rst
> > +++ b/Documentation/userspace-api/media/mediactl/media-types.rst
> > @@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
> >  .. _MEDIA-PAD-FL-SINK:
> >  .. _MEDIA-PAD-FL-SOURCE:
> >  .. _MEDIA-PAD-FL-MUST-CONNECT:
> > +.. _MEDIA-PAD-FL-INTERNAL-SOURCE:
> >  
> >  .. flat-table:: Media pad flags
> >      :header-rows:  0
> > @@ -382,6 +383,12 @@ Types and flags used to represent the media graph elements
> >  	  when this flag isn't set; the absence of the flag doesn't imply
> >  	  there is none.
> >  
> > +    *  -  ``MEDIA_PAD_FL_INTERNAL_SOURCE``
> > +       -  This flag indicates an internal pad that has no external
> > +	  connections. Such a pad may not be connected with a link. The internal
> > +	  flag indicates that the stream either starts or ends in the
> 
> There's no mention of "stream" (with this meaning) anywhere in the MC
> API documentation, so you will need to explain streams first.

I'll add it to glossary and add a reference here.

> 
> Apart from that, and addressing Tomi's comment, this patch seems OK.
> 
> > +	  entity. For a given pad, the INTERNAL_SOURCE flag may not be set if
> > +	  either SINK or SOURCE flags is set.
> >  
> >  One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
> >  must be set for every pad.
> > diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> > index ef34ddd715c9..ed99160a2487 100644
> > --- a/drivers/media/mc/mc-entity.c
> > +++ b/drivers/media/mc/mc-entity.c
> > @@ -1062,7 +1062,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
> >  
> >  	for (i = 0; i < entity->num_pads; i++) {
> >  		if ((entity->pads[i].flags &
> > -		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
> > +		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
> > +		      MEDIA_PAD_FL_INTERNAL_SOURCE)) != pad_type)
> >  			continue;
> >  
> >  		if (entity->pads[i].sig_type == sig_type)
> > @@ -1087,6 +1088,11 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
> >  		return -EINVAL;
> >  	if (WARN_ON(!(sink->pads[sink_pad].flags & MEDIA_PAD_FL_SINK)))
> >  		return -EINVAL;
> > +	if (WARN_ON(sink->pads[sink_pad].flags &
> > +		    MEDIA_PAD_FL_INTERNAL_SOURCE) ||
> > +	    WARN_ON(source->pads[source_pad].flags &
> > +		    MEDIA_PAD_FL_INTERNAL_SOURCE))
> > +		return -EINVAL;
> >  
> >  	link = media_add_link(&source->links);
> >  	if (link == NULL)
> > diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
> > index edb8dfef9eba..0e2577e8b425 100644
> > --- a/include/uapi/linux/media.h
> > +++ b/include/uapi/linux/media.h
> > @@ -208,6 +208,7 @@ struct media_entity_desc {
> >  #define MEDIA_PAD_FL_SINK			(1U << 0)
> >  #define MEDIA_PAD_FL_SOURCE			(1U << 1)
> >  #define MEDIA_PAD_FL_MUST_CONNECT		(1U << 2)
> > +#define MEDIA_PAD_FL_INTERNAL_SOURCE		(1U << 3)
> >  
> >  struct media_pad_desc {
> >  	__u32 entity;		/* entity ID */
> 

-- 
Regards,

Sakari Ailus

Re: [RFC 2/7] media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs

[Laurent Pinchart]

Hi Sakari,

On Fri, Jun 02, 2023 at 01:10:40PM +0000, Sakari Ailus wrote:
> On Fri, Jun 02, 2023 at 12:46:50PM +0300, Laurent Pinchart wrote:
> > On Fri, Jun 02, 2023 at 12:44:12PM +0300, Laurent Pinchart wrote:
> > > On Mon, May 08, 2023 at 03:24:10PM +0300, Sakari Ailus wrote:
> > > > On Mon, May 08, 2023 at 01:14:07PM +0300, Tomi Valkeinen wrote:
> > > > > On 06/05/2023 00:52, Sakari Ailus wrote:
> > > > > > Take the new INTERNAL_SOURCE pad flag into account in validating routing
> > > > > > IOCTL argument. Effectively this is a SINK pad in this respect. Due to the
> > > > > > union there's no need to check for the particular name.
> > > > > 
> > > > > What union? The one you add in the next patch?
> > > > 
> > > > Oops. I think we can reorder the patches.
> > > > 
> > > > > As a concept the internal source pads sound good, and they work in practice
> > > > > in my tests. But this union is what grates me a bit. We have a flag,
> > > > > MEDIA_PAD_FL_INTERNAL_SOURCE, which tells which field of the union to use,
> > > > > and then we go and do not use the new union field. Well, and also the
> > > > > naming, as we normally have source and sink pads, but now we also have
> > > > > internal source pads, which are actually identical to sink pads...
> > > > 
> > > > The union still should be used by the user space. We're testing flags here
> > > > and those flags are the same independently of the INTERNAL_SOURCE flag.
> > > > 
> > > > I'm fine by not adding that union though, but for the user space I think
> > > > it's better we have it: explaining that the sink_pad has a different
> > > > meaning if that flag is set is harder than making it a union member.
> > > > 
> > > > > I understand the idea and reasoning, but the two points above do confuse me
> > > > > and I'm sure would confuse others also.
> > > > > 
> > > > > I wonder if it would be less or more confusing to simplify this by just
> > > > > adding a MEDIA_PAD_FL_INTERNAL, which could be applied to either a source or
> > > > > a sink pad, and would prevent linking to it. The flag would indicate that
> > > > > the stream from/to that pad is generated/consumed internally. (I don't know
> > > > > when you would need an internal pad to consume data, but... who knows, the
> > > > > need might pop up =).
> > > > 
> > > > This is another option. But I envision there will be more compatibility
> > > > issues. Although... generally using such hardware requires knowledge of the
> > > > device, and we haven't obviously had any support for devices needing this
> > > > functionality in the tree. So in the end it might not matter much.
> > > >
> > > > > That would mean that an "internal sink pad" would generate a data stream,
> > > > > i.e. it's a "source", but I think a normal sink pad is almost the same
> > > > > anyway: when considering entity's internal routing, the normal sink pad is a
> > > > > "source", and the same logic would apply with the internal pads too.
> > > > > 
> > > > > An internal sink pad that generates a stream is a bit more confusing than an
> > > > > internal source pad, but I think that confusion is less than the rest of the
> > > > > confusion in the code-side that comes with the internal source pads.
> > > > 
> > > > I prefer having the possible sources of the confusion in the framework than
> > > > in the drivers. Therefore I think INTERNAL_SOURCE flag is a (slightly)
> > > > better option.
> > > > 
> > > > I wonder what Llaurent thinks.
> > > 
> > > I like the idea of a MEDIA_PAD_FL_INTERNAL flag. That's actually how I
> > > read patch 1/7, I didn't notice it used MEDIA_PAD_FL_INTERNAL*_SOURCE*
> > > :-)
> > > 
> > > We could define MEDIA_PAD_FL_INTERNAL_SOURCE
> > > 
> > > #define MEDIA_PAD_FL_INTERNAL_SOURCE 	(MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL)
> > 
> > One option would be to find terms different from sink and source in this
> > case. It would generate less confusion to map (e.g., not a really good
> > name) MEDIA_PAD_FL_INTERNAL_PRODUCER to MEDIA_PAD_FL_SINK and to the
> > sink_pad field than using MEDIA_PAD_FL_INTERNAL_SOURCE.
> 
> I don't think this helps as you'd still be accessing the sink pad related
> fields that are there for another purpose.
> 
> Alternatively I'd have the (plain) INTERNAL flag and drop the union,
> without masking or adding compound flags.
> 
> I can switch to that if you promise not to change your mind again. ;-)

I'll do my best :-) Could you show the impact (if any) it would have on
drivers when accessing the routing fields ?

> > > but I'm not sure it would be less confusing.
> > > 
> > > Regarding isolating the sources of confusion in the framework rather
> > > than spreading them through drivers, I can't disagree, but I think that,
> > > for raw camera sensors at least, the best option would be to create a
> > > new camera sensor object with a much more tailored API than v4l2_subdev
> > > (and of course wrapping that new object in a v4l2_subdev in the
> > > framework). This won't address the other types of drivers, but I'm not
> > > sure we'll get any in the foreseable future.

-- 
Regards,

Laurent Pinchart

Re: [RFC 2/7] media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs

[Sakari Ailus]

Hi Laurent,

On Sun, Jun 04, 2023 at 05:26:26PM +0300, Laurent Pinchart wrote:
> Hi Sakari,
> 
> On Fri, Jun 02, 2023 at 01:10:40PM +0000, Sakari Ailus wrote:
> > On Fri, Jun 02, 2023 at 12:46:50PM +0300, Laurent Pinchart wrote:
> > > On Fri, Jun 02, 2023 at 12:44:12PM +0300, Laurent Pinchart wrote:
> > > > On Mon, May 08, 2023 at 03:24:10PM +0300, Sakari Ailus wrote:
> > > > > On Mon, May 08, 2023 at 01:14:07PM +0300, Tomi Valkeinen wrote:
> > > > > > On 06/05/2023 00:52, Sakari Ailus wrote:
> > > > > > > Take the new INTERNAL_SOURCE pad flag into account in validating routing
> > > > > > > IOCTL argument. Effectively this is a SINK pad in this respect. Due to the
> > > > > > > union there's no need to check for the particular name.
> > > > > > 
> > > > > > What union? The one you add in the next patch?
> > > > > 
> > > > > Oops. I think we can reorder the patches.
> > > > > 
> > > > > > As a concept the internal source pads sound good, and they work in practice
> > > > > > in my tests. But this union is what grates me a bit. We have a flag,
> > > > > > MEDIA_PAD_FL_INTERNAL_SOURCE, which tells which field of the union to use,
> > > > > > and then we go and do not use the new union field. Well, and also the
> > > > > > naming, as we normally have source and sink pads, but now we also have
> > > > > > internal source pads, which are actually identical to sink pads...
> > > > > 
> > > > > The union still should be used by the user space. We're testing flags here
> > > > > and those flags are the same independently of the INTERNAL_SOURCE flag.
> > > > > 
> > > > > I'm fine by not adding that union though, but for the user space I think
> > > > > it's better we have it: explaining that the sink_pad has a different
> > > > > meaning if that flag is set is harder than making it a union member.
> > > > > 
> > > > > > I understand the idea and reasoning, but the two points above do confuse me
> > > > > > and I'm sure would confuse others also.
> > > > > > 
> > > > > > I wonder if it would be less or more confusing to simplify this by just
> > > > > > adding a MEDIA_PAD_FL_INTERNAL, which could be applied to either a source or
> > > > > > a sink pad, and would prevent linking to it. The flag would indicate that
> > > > > > the stream from/to that pad is generated/consumed internally. (I don't know
> > > > > > when you would need an internal pad to consume data, but... who knows, the
> > > > > > need might pop up =).
> > > > > 
> > > > > This is another option. But I envision there will be more compatibility
> > > > > issues. Although... generally using such hardware requires knowledge of the
> > > > > device, and we haven't obviously had any support for devices needing this
> > > > > functionality in the tree. So in the end it might not matter much.
> > > > >
> > > > > > That would mean that an "internal sink pad" would generate a data stream,
> > > > > > i.e. it's a "source", but I think a normal sink pad is almost the same
> > > > > > anyway: when considering entity's internal routing, the normal sink pad is a
> > > > > > "source", and the same logic would apply with the internal pads too.
> > > > > > 
> > > > > > An internal sink pad that generates a stream is a bit more confusing than an
> > > > > > internal source pad, but I think that confusion is less than the rest of the
> > > > > > confusion in the code-side that comes with the internal source pads.
> > > > > 
> > > > > I prefer having the possible sources of the confusion in the framework than
> > > > > in the drivers. Therefore I think INTERNAL_SOURCE flag is a (slightly)
> > > > > better option.
> > > > > 
> > > > > I wonder what Llaurent thinks.
> > > > 
> > > > I like the idea of a MEDIA_PAD_FL_INTERNAL flag. That's actually how I
> > > > read patch 1/7, I didn't notice it used MEDIA_PAD_FL_INTERNAL*_SOURCE*
> > > > :-)
> > > > 
> > > > We could define MEDIA_PAD_FL_INTERNAL_SOURCE
> > > > 
> > > > #define MEDIA_PAD_FL_INTERNAL_SOURCE 	(MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL)
> > > 
> > > One option would be to find terms different from sink and source in this
> > > case. It would generate less confusion to map (e.g., not a really good
> > > name) MEDIA_PAD_FL_INTERNAL_PRODUCER to MEDIA_PAD_FL_SINK and to the
> > > sink_pad field than using MEDIA_PAD_FL_INTERNAL_SOURCE.
> > 
> > I don't think this helps as you'd still be accessing the sink pad related
> > fields that are there for another purpose.
> > 
> > Alternatively I'd have the (plain) INTERNAL flag and drop the union,
> > without masking or adding compound flags.
> > 
> > I can switch to that if you promise not to change your mind again. ;-)
> 
> I'll do my best :-) Could you show the impact (if any) it would have on
> drivers when accessing the routing fields ?

I don't think there's much of an impact for the drivers. It's mainly
affecting setting up pads for the entities. Tomi?

-- 
Sakari Ailus

Re: [RFC 2/7] media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs

[Laurent Pinchart]

On Mon, Jun 05, 2023 at 08:06:58AM +0000, Sakari Ailus wrote:
> On Sun, Jun 04, 2023 at 05:26:26PM +0300, Laurent Pinchart wrote:
> > On Fri, Jun 02, 2023 at 01:10:40PM +0000, Sakari Ailus wrote:
> > > On Fri, Jun 02, 2023 at 12:46:50PM +0300, Laurent Pinchart wrote:
> > > > On Fri, Jun 02, 2023 at 12:44:12PM +0300, Laurent Pinchart wrote:
> > > > > On Mon, May 08, 2023 at 03:24:10PM +0300, Sakari Ailus wrote:
> > > > > > On Mon, May 08, 2023 at 01:14:07PM +0300, Tomi Valkeinen wrote:
> > > > > > > On 06/05/2023 00:52, Sakari Ailus wrote:
> > > > > > > > Take the new INTERNAL_SOURCE pad flag into account in validating routing
> > > > > > > > IOCTL argument. Effectively this is a SINK pad in this respect. Due to the
> > > > > > > > union there's no need to check for the particular name.
> > > > > > > 
> > > > > > > What union? The one you add in the next patch?
> > > > > > 
> > > > > > Oops. I think we can reorder the patches.
> > > > > > 
> > > > > > > As a concept the internal source pads sound good, and they work in practice
> > > > > > > in my tests. But this union is what grates me a bit. We have a flag,
> > > > > > > MEDIA_PAD_FL_INTERNAL_SOURCE, which tells which field of the union to use,
> > > > > > > and then we go and do not use the new union field. Well, and also the
> > > > > > > naming, as we normally have source and sink pads, but now we also have
> > > > > > > internal source pads, which are actually identical to sink pads...
> > > > > > 
> > > > > > The union still should be used by the user space. We're testing flags here
> > > > > > and those flags are the same independently of the INTERNAL_SOURCE flag.
> > > > > > 
> > > > > > I'm fine by not adding that union though, but for the user space I think
> > > > > > it's better we have it: explaining that the sink_pad has a different
> > > > > > meaning if that flag is set is harder than making it a union member.
> > > > > > 
> > > > > > > I understand the idea and reasoning, but the two points above do confuse me
> > > > > > > and I'm sure would confuse others also.
> > > > > > > 
> > > > > > > I wonder if it would be less or more confusing to simplify this by just
> > > > > > > adding a MEDIA_PAD_FL_INTERNAL, which could be applied to either a source or
> > > > > > > a sink pad, and would prevent linking to it. The flag would indicate that
> > > > > > > the stream from/to that pad is generated/consumed internally. (I don't know
> > > > > > > when you would need an internal pad to consume data, but... who knows, the
> > > > > > > need might pop up =).
> > > > > > 
> > > > > > This is another option. But I envision there will be more compatibility
> > > > > > issues. Although... generally using such hardware requires knowledge of the
> > > > > > device, and we haven't obviously had any support for devices needing this
> > > > > > functionality in the tree. So in the end it might not matter much.
> > > > > >
> > > > > > > That would mean that an "internal sink pad" would generate a data stream,
> > > > > > > i.e. it's a "source", but I think a normal sink pad is almost the same
> > > > > > > anyway: when considering entity's internal routing, the normal sink pad is a
> > > > > > > "source", and the same logic would apply with the internal pads too.
> > > > > > > 
> > > > > > > An internal sink pad that generates a stream is a bit more confusing than an
> > > > > > > internal source pad, but I think that confusion is less than the rest of the
> > > > > > > confusion in the code-side that comes with the internal source pads.
> > > > > > 
> > > > > > I prefer having the possible sources of the confusion in the framework than
> > > > > > in the drivers. Therefore I think INTERNAL_SOURCE flag is a (slightly)
> > > > > > better option.
> > > > > > 
> > > > > > I wonder what Llaurent thinks.
> > > > > 
> > > > > I like the idea of a MEDIA_PAD_FL_INTERNAL flag. That's actually how I
> > > > > read patch 1/7, I didn't notice it used MEDIA_PAD_FL_INTERNAL*_SOURCE*
> > > > > :-)
> > > > > 
> > > > > We could define MEDIA_PAD_FL_INTERNAL_SOURCE
> > > > > 
> > > > > #define MEDIA_PAD_FL_INTERNAL_SOURCE 	(MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL)
> > > > 
> > > > One option would be to find terms different from sink and source in this
> > > > case. It would generate less confusion to map (e.g., not a really good
> > > > name) MEDIA_PAD_FL_INTERNAL_PRODUCER to MEDIA_PAD_FL_SINK and to the
> > > > sink_pad field than using MEDIA_PAD_FL_INTERNAL_SOURCE.
> > > 
> > > I don't think this helps as you'd still be accessing the sink pad related
> > > fields that are there for another purpose.
> > > 
> > > Alternatively I'd have the (plain) INTERNAL flag and drop the union,
> > > without masking or adding compound flags.
> > > 
> > > I can switch to that if you promise not to change your mind again. ;-)
> > 
> > I'll do my best :-) Could you show the impact (if any) it would have on
> > drivers when accessing the routing fields ?
> 
> I don't think there's much of an impact for the drivers. It's mainly
> affecting setting up pads for the entities. Tomi?

I think it would be useful to discuss it with an actual sensor driver
using the API.

-- 
Regards,

Laurent Pinchart

Re: [RFC 1/7] media: mc: Add INTERNAL_SOURCE pad type flag

[Hans Verkuil]

On 05/05/2023 23:52, Sakari Ailus wrote:
> Internal source pads will be used as routing endpoints in V4L2
> [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
> 
> Also prevent creating links to pads that have been flagged as internal.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../userspace-api/media/mediactl/media-types.rst          | 7 +++++++
>  drivers/media/mc/mc-entity.c                              | 8 +++++++-
>  include/uapi/linux/media.h                                | 1 +
>  3 files changed, 15 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/userspace-api/media/mediactl/media-types.rst b/Documentation/userspace-api/media/mediactl/media-types.rst
> index 0ffeece1e0c8..c724139ad46c 100644
> --- a/Documentation/userspace-api/media/mediactl/media-types.rst
> +++ b/Documentation/userspace-api/media/mediactl/media-types.rst
> @@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
>  .. _MEDIA-PAD-FL-SINK:
>  .. _MEDIA-PAD-FL-SOURCE:
>  .. _MEDIA-PAD-FL-MUST-CONNECT:
> +.. _MEDIA-PAD-FL-INTERNAL-SOURCE:
>  
>  .. flat-table:: Media pad flags
>      :header-rows:  0
> @@ -382,6 +383,12 @@ Types and flags used to represent the media graph elements
>  	  when this flag isn't set; the absence of the flag doesn't imply
>  	  there is none.
>  
> +    *  -  ``MEDIA_PAD_FL_INTERNAL_SOURCE``
> +       -  This flag indicates an internal pad that has no external

'internal pad' or 'internal source pad'?

> +	  connections. Such a pad may not be connected with a link. The internal
> +	  flag indicates that the stream either starts or ends in the
> +	  entity.

This sentence makes no sense. The flag is for a pad, but then this sentence
talks about streams and entities and doesn't relate that to the pad. Also,
the flag is INTERNAL_SOURCE, and reading 'stream ends' suggests that it
should be INTERNAL_SINK in that case.

I think this needs to be more elaborate and ideally add some examples
of the use-case(s) (or refer to an example elsewhere).

 For a given pad, the INTERNAL_SOURCE flag may not be set if
> +	  either SINK or SOURCE flags is set.
>  
>  One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``

and 'MEDIA_PAD_FL_INTERNAL_SOURCE' as well?

>  must be set for every pad.
> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> index ef34ddd715c9..ed99160a2487 100644
> --- a/drivers/media/mc/mc-entity.c
> +++ b/drivers/media/mc/mc-entity.c
> @@ -1062,7 +1062,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
>  
>  	for (i = 0; i < entity->num_pads; i++) {
>  		if ((entity->pads[i].flags &
> -		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
> +		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
> +		      MEDIA_PAD_FL_INTERNAL_SOURCE)) != pad_type)
>  			continue;
>  
>  		if (entity->pads[i].sig_type == sig_type)
> @@ -1087,6 +1088,11 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
>  		return -EINVAL;
>  	if (WARN_ON(!(sink->pads[sink_pad].flags & MEDIA_PAD_FL_SINK)))
>  		return -EINVAL;
> +	if (WARN_ON(sink->pads[sink_pad].flags &
> +		    MEDIA_PAD_FL_INTERNAL_SOURCE) ||
> +	    WARN_ON(source->pads[source_pad].flags &
> +		    MEDIA_PAD_FL_INTERNAL_SOURCE))
> +		return -EINVAL;
>  
>  	link = media_add_link(&source->links);
>  	if (link == NULL)
> diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
> index edb8dfef9eba..0e2577e8b425 100644
> --- a/include/uapi/linux/media.h
> +++ b/include/uapi/linux/media.h
> @@ -208,6 +208,7 @@ struct media_entity_desc {
>  #define MEDIA_PAD_FL_SINK			(1U << 0)
>  #define MEDIA_PAD_FL_SOURCE			(1U << 1)
>  #define MEDIA_PAD_FL_MUST_CONNECT		(1U << 2)
> +#define MEDIA_PAD_FL_INTERNAL_SOURCE		(1U << 3)
>  
>  struct media_pad_desc {
>  	__u32 entity;		/* entity ID */

Regards,

	Hans

Re: [RFC 2/7] media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs

[Hans Verkuil]

On 05/05/2023 23:52, Sakari Ailus wrote:
> Take the new INTERNAL_SOURCE pad flag into account in validating routing
> IOCTL argument. Effectively this is a SINK pad in this respect. Due to the
> union there's no need to check for the particular name.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 2ec179cd1264..6312fc2bf1f0 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -906,7 +906,8 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>  				return -EINVAL;
>  
>  			if (!(pads[route->sink_pad].flags &
> -			      MEDIA_PAD_FL_SINK))
> +			      (MEDIA_PAD_FL_SINK |
> +			       MEDIA_PAD_FL_INTERNAL_SOURCE)))
>  				return -EINVAL;
>  
>  			if (route->source_pad >= sd->entity.num_pads)
> @@ -1787,7 +1788,8 @@ int v4l2_subdev_routing_validate(struct v4l2_subdev *sd,
>  
>  		/* Validate the sink and source pad numbers. */
>  		if (route->sink_pad >= sd->entity.num_pads ||
> -		    !(sd->entity.pads[route->sink_pad].flags & MEDIA_PAD_FL_SINK)) {
> +		    !(sd->entity.pads[route->sink_pad].flags &
> +		      (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL_SOURCE))) {

So this relates to my reply to the previous patch: calling a sink pad
an INTERNAL_SOURCE is really confusing.

Some suggestions for alternative names:

1) MEDIA_PAD_FL_SINK_INTERNAL
2) MEDIA_PAD_FL_SINK_INTERNAL_SOURCE (or _SRC)
3) MEDIA_PAD_FL_SINK_FROM_INTERNAL_SOURCE (or _SRC)

IMHO the name *has* to start with MEDIA_PAD_FL_SINK_ to indicate
that it is a sink pad, followed by something that indicates that
it is fed from an internal source.

Regards,

	Hans

>  			dev_dbg(sd->dev, "route %u sink (%u) is not a sink pad\n",
>  				i, route->sink_pad);
>  			goto out;

Re: [RFC 3/7] media: uapi: v4l: Document source routes

[Hans Verkuil]

On 05/05/2023 23:52, Sakari Ailus wrote:
> Document how internal pads are used on source routes.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../userspace-api/media/v4l/dev-subdev.rst     | 18 ++++++++++++++++++
>  .../media/v4l/vidioc-subdev-g-routing.rst      |  5 +++++
>  include/uapi/linux/v4l2-subdev.h               |  6 +++++-
>  3 files changed, 28 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> index a4f1df7093e8..395e06d2f0f2 100644
> --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> @@ -551,6 +551,24 @@ A stream at a specific point in the media pipeline is identified by the
>  sub-device and a (pad, stream) pair. For sub-devices that do not support
>  multiplexed streams the 'stream' field is always 0.
>  
> +.. _v4l2-subdev-source-routes:
> +
> +Source routes
> +^^^^^^^^^^^^^
> +
> +Cases where a single sub-device pad is a source of multiple streams are special
> +as there is no sink pad for such a route. In those cases, an internal pad is
> +used as the sink pad. Such pads have the :ref:`MEDIA_PAD_FL_INTERNAL_SOURCE
> +<MEDIA-PAD-FL-INTERNAL-SOURCE>` flag set.

add: instead of the <MEDIA-PAD-FL-SINK> flag.

> +
> +Internal pads have all the properties of a sink pad in such case, including
> +formats and selections. The format in this case is the source format of the
> +stream. An internal pad always has a single stream only (0).
> +
> +Generally source routes are not modifiable but they can be activated and
> +inactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE

inactivated -> deactivated

> +<v4l2-subdev-routing-flags>` flag, depending on driver capabilities.
> +

Again, this really needs one or more examples.

>  Interaction between routes, streams, formats and selections
>  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>  
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> index 68ca343c3b44..e00622071b64 100644
> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> @@ -94,6 +94,11 @@ for all the route entries and call ``VIDIOC_SUBDEV_G_ROUTING`` again.
>      * - __u32
>        - ``sink_pad``
>        - Sink pad number.
> +    * - __u32
> +      - ``internal_source_pad``
> +      - Internal source pad number. For pads with :ref:`internal source pad flag
> +	<MEDIA-PAD-FL-INTERNAL-SOURCE>`, for use with :ref:`source routes
> +	<v4l2-subdev-source-routes>`.
>      * - __u32
>        - ``sink_stream``
>        - Sink pad stream number.
> diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
> index 4a195b68f28f..703e3a1f199b 100644
> --- a/include/uapi/linux/v4l2-subdev.h
> +++ b/include/uapi/linux/v4l2-subdev.h
> @@ -203,6 +203,7 @@ struct v4l2_subdev_capability {
>   * struct v4l2_subdev_route - A route inside a subdev
>   *
>   * @sink_pad: the sink pad index
> + * @internal_source_pad: the internal source pad
>   * @sink_stream: the sink stream identifier
>   * @source_pad: the source pad index
>   * @source_stream: the source stream identifier
> @@ -210,7 +211,10 @@ struct v4l2_subdev_capability {
>   * @reserved: drivers and applications must zero this array
>   */
>  struct v4l2_subdev_route {
> -	__u32 sink_pad;
> +	union {
> +		__u32 sink_pad;
> +		__u32 internal_source_pad;

Nobody will understand this. I think this should still be kept as
sink_pad. If the INTERNAL_SOURCE flag is changed to SINK_FROM_INTERNAL_SOURCE
(or something along those lines), then 'sink_pad' still makes sense
and you don't need this change.

It's still a sink pad, just a sink pad with a special property in that it
receives the data from an internal source.

> +	};
>  	__u32 sink_stream;
>  	__u32 source_pad;
>  	__u32 source_stream;

Regards,

	Hans

Re: [RFC 5/7] media: uapi: Add generic serial metadata mbus formats

[Hans Verkuil]

On 05/05/2023 23:52, Sakari Ailus wrote:
> Add generic serial metadata mbus formats. These formats describe data
> width and packing but not the content itself. The reason for specifying
> such formats is that the formats as such are fairly device specific but
> they are still handled by CSI-2 receiver drivers that should not be aware
> of device specific formats. What makes generic metadata formats possible
> is that these formats are parsed by software only, after capturing the
> data to system memory.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../media/v4l/subdev-formats.rst              | 257 ++++++++++++++++++
>  include/uapi/linux/media-bus-format.h         |   9 +
>  2 files changed, 266 insertions(+)
> 
> diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> index a3a35eeed708..1492fff58426 100644
> --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
> +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> @@ -8234,3 +8234,260 @@ The following table lists the existing metadata formats.
>  	both sides of the link and the bus format is a fixed
>  	metadata format that is not configurable from userspace.
>  	Width and height will be set to 0 for this format.
> +
> +Generic Serial Metadata Formats
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +Generic serial metadata formats are used on serial busses where the actual data
> +content is more or less device specific but the data is transmitted and received
> +by multiple devices that do not process the data in any way, simply writing
> +it to system memory for processing in software at the end of the pipeline.
> +
> +The more specific variant describing the actual data is used on the internal
> +source pad of the originating sub-device.

As Laurent said as well, the mention of 'internal source pad' is confusing.

Aren't there two possibilities here? Either the entity has just a source pad
that delivers the metadata, and that source pad has the precise metadata
format. Or the entity supports routing and then there is a sink pad connected
to an internal source that is routed to a source pad. In that case that sink
pad has the precise metadata format.

Apologies if I am wrong, it's hard to keep track of all the possibilities,
especially if you do not use this regularly.

> +
> +"b" in an array cell signifies a byte of data, followed by the number of byte
> +and finally the bit number in subscript. "p" indicates a padding bit.
> +
> +.. _media-bus-format-generic-meta:
> +
> +.. cssclass: longtable
> +
> +.. flat-table:: Generic Serial Metadata Formats
> +    :header-rows:  2
> +    :stub-columns: 0
> +
> +    * - Identifier
> +      - Code
> +      -
> +      - :cspan:`23` Data organization
> +    * -
> +      -
> +      - Bit
> +      - 23
> +      - 22
> +      - 21
> +      - 20
> +      - 19
> +      - 18
> +      - 17
> +      - 16
> +      - 15
> +      - 14
> +      - 13
> +      - 12
> +      - 11
> +      - 10
> +      - 9
> +      - 8
> +      - 7
> +      - 6
> +      - 5
> +      - 4
> +      - 3
> +      - 2
> +      - 1
> +      - 0
> +    * .. _MEDIA-BUS-FMT-META-1X8-8:
> +
> +      - MEDIA_BUS_FMT_META_1X8_8
> +      - 0x8001
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      - b0\ :sub:`7`
> +      - b0\ :sub:`6`
> +      - b0\ :sub:`5`
> +      - b0\ :sub:`4`
> +      - b0\ :sub:`3`
> +      - b0\ :sub:`2`
> +      - b0\ :sub:`1`
> +      - b0\ :sub:`0`
> +    * .. _MEDIA-BUS-FMT-META-1X8-10:
> +
> +      - MEDIA_BUS_FMT_META_1X8_10
> +      - 0x8002
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      - b0\ :sub:`7`
> +      - b0\ :sub:`6`
> +      - b0\ :sub:`5`
> +      - b0\ :sub:`4`
> +      - b0\ :sub:`3`
> +      - b0\ :sub:`2`
> +      - b0\ :sub:`1`
> +      - b0\ :sub:`0`
> +      - p
> +      - p
> +    * .. _MEDIA-BUS-FMT-META-1X8-12:
> +
> +      - MEDIA_BUS_FMT_META_1X8_12
> +      - 0x8003
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      - b0\ :sub:`7`
> +      - b0\ :sub:`6`
> +      - b0\ :sub:`5`
> +      - b0\ :sub:`4`
> +      - b0\ :sub:`3`
> +      - b0\ :sub:`2`
> +      - b0\ :sub:`1`
> +      - b0\ :sub:`0`
> +      - p
> +      - p
> +      - p
> +      - p
> +    * .. _MEDIA-BUS-FMT-META-1X8-14:
> +
> +      - MEDIA_BUS_FMT_META_1X8_14
> +      - 0x8004
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      - b0\ :sub:`7`
> +      - b0\ :sub:`6`
> +      - b0\ :sub:`5`
> +      - b0\ :sub:`4`
> +      - b0\ :sub:`3`
> +      - b0\ :sub:`2`
> +      - b0\ :sub:`1`
> +      - b0\ :sub:`0`
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +    * .. _MEDIA-BUS-FMT-META-1X8-16:
> +
> +      - MEDIA_BUS_FMT_META_1X8_16
> +      - 0x8005
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      - b0\ :sub:`7`
> +      - b0\ :sub:`6`
> +      - b0\ :sub:`5`
> +      - b0\ :sub:`4`
> +      - b0\ :sub:`3`
> +      - b0\ :sub:`2`
> +      - b0\ :sub:`1`
> +      - b0\ :sub:`0`
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +    * .. _MEDIA-BUS-FMT-META-1X8-20:
> +
> +      - MEDIA_BUS_FMT_META_1X8_20
> +      - 0x8007
> +      -
> +      -
> +      -
> +      -
> +      -
> +      - b0\ :sub:`7`
> +      - b0\ :sub:`6`
> +      - b0\ :sub:`5`
> +      - b0\ :sub:`4`
> +      - b0\ :sub:`3`
> +      - b0\ :sub:`2`
> +      - b0\ :sub:`1`
> +      - b0\ :sub:`0`
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +    * .. _MEDIA-BUS-FMT-META-1X8-24:
> +
> +      - MEDIA_BUS_FMT_META_1X8_24
> +      - 0x8009
> +      -
> +      - b0\ :sub:`7`
> +      - b0\ :sub:`6`
> +      - b0\ :sub:`5`
> +      - b0\ :sub:`4`
> +      - b0\ :sub:`3`
> +      - b0\ :sub:`2`
> +      - b0\ :sub:`1`
> +      - b0\ :sub:`0`
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
> index a03c543cb072..722463523bbd 100644
> --- a/include/uapi/linux/media-bus-format.h
> +++ b/include/uapi/linux/media-bus-format.h
> @@ -173,4 +173,13 @@
>   */
>  #define MEDIA_BUS_FMT_METADATA_FIXED		0x7001
>  
> +/* Generic line based metadata formats for serial buses. Next is 0x800b. */
> +#define MEDIA_BUS_FMT_META_1X8_8		0x8001
> +#define MEDIA_BUS_FMT_META_1X8_10		0x8002
> +#define MEDIA_BUS_FMT_META_1X8_12		0x8003
> +#define MEDIA_BUS_FMT_META_1X8_14		0x8004
> +#define MEDIA_BUS_FMT_META_1X8_16		0x8005

You skip 0x8006 here in case there will be a _18 variant?

> +#define MEDIA_BUS_FMT_META_1X8_20		0x8007
> +#define MEDIA_BUS_FMT_META_1X8_24		0x8009
> +
>  #endif /* __LINUX_MEDIA_BUS_FORMAT_H */

Regards,

	Hans

Re: [RFC 5/7] media: uapi: Add generic serial metadata mbus formats

[Hans Verkuil]

On 05/05/2023 23:52, Sakari Ailus wrote:
> Add generic serial metadata mbus formats. These formats describe data
> width and packing but not the content itself. The reason for specifying
> such formats is that the formats as such are fairly device specific but
> they are still handled by CSI-2 receiver drivers that should not be aware
> of device specific formats. What makes generic metadata formats possible
> is that these formats are parsed by software only, after capturing the
> data to system memory.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../media/v4l/subdev-formats.rst              | 257 ++++++++++++++++++
>  include/uapi/linux/media-bus-format.h         |   9 +
>  2 files changed, 266 insertions(+)
> 
> diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> index a3a35eeed708..1492fff58426 100644
> --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
> +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> @@ -8234,3 +8234,260 @@ The following table lists the existing metadata formats.
>  	both sides of the link and the bus format is a fixed
>  	metadata format that is not configurable from userspace.
>  	Width and height will be set to 0 for this format.
> +
> +Generic Serial Metadata Formats
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +Generic serial metadata formats are used on serial busses where the actual data
> +content is more or less device specific but the data is transmitted and received
> +by multiple devices that do not process the data in any way, simply writing
> +it to system memory for processing in software at the end of the pipeline.
> +
> +The more specific variant describing the actual data is used on the internal
> +source pad of the originating sub-device.

I'm a bit confused here: on which pads are these generic mbus formats set? And who
sets this?

I.e. the internal source pad has e.g. MEDIA_BUS_FMT_META_FOOBAR_HIST as specific
format. It's routed to a source pad + stream pair for transfer over CSI. Does
that source pad + stream pair use the generic mediabus format?

Regards,

	Hans

> +
> +"b" in an array cell signifies a byte of data, followed by the number of byte
> +and finally the bit number in subscript. "p" indicates a padding bit.
> +
> +.. _media-bus-format-generic-meta:
> +
> +.. cssclass: longtable
> +
> +.. flat-table:: Generic Serial Metadata Formats
> +    :header-rows:  2
> +    :stub-columns: 0
> +
> +    * - Identifier
> +      - Code
> +      -
> +      - :cspan:`23` Data organization
> +    * -
> +      -
> +      - Bit
> +      - 23
> +      - 22
> +      - 21
> +      - 20
> +      - 19
> +      - 18
> +      - 17
> +      - 16
> +      - 15
> +      - 14
> +      - 13
> +      - 12
> +      - 11
> +      - 10
> +      - 9
> +      - 8
> +      - 7
> +      - 6
> +      - 5
> +      - 4
> +      - 3
> +      - 2
> +      - 1
> +      - 0
> +    * .. _MEDIA-BUS-FMT-META-1X8-8:
> +
> +      - MEDIA_BUS_FMT_META_1X8_8
> +      - 0x8001
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      - b0\ :sub:`7`
> +      - b0\ :sub:`6`
> +      - b0\ :sub:`5`
> +      - b0\ :sub:`4`
> +      - b0\ :sub:`3`
> +      - b0\ :sub:`2`
> +      - b0\ :sub:`1`
> +      - b0\ :sub:`0`
> +    * .. _MEDIA-BUS-FMT-META-1X8-10:
> +
> +      - MEDIA_BUS_FMT_META_1X8_10
> +      - 0x8002
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      - b0\ :sub:`7`
> +      - b0\ :sub:`6`
> +      - b0\ :sub:`5`
> +      - b0\ :sub:`4`
> +      - b0\ :sub:`3`
> +      - b0\ :sub:`2`
> +      - b0\ :sub:`1`
> +      - b0\ :sub:`0`
> +      - p
> +      - p
> +    * .. _MEDIA-BUS-FMT-META-1X8-12:
> +
> +      - MEDIA_BUS_FMT_META_1X8_12
> +      - 0x8003
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      - b0\ :sub:`7`
> +      - b0\ :sub:`6`
> +      - b0\ :sub:`5`
> +      - b0\ :sub:`4`
> +      - b0\ :sub:`3`
> +      - b0\ :sub:`2`
> +      - b0\ :sub:`1`
> +      - b0\ :sub:`0`
> +      - p
> +      - p
> +      - p
> +      - p
> +    * .. _MEDIA-BUS-FMT-META-1X8-14:
> +
> +      - MEDIA_BUS_FMT_META_1X8_14
> +      - 0x8004
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      - b0\ :sub:`7`
> +      - b0\ :sub:`6`
> +      - b0\ :sub:`5`
> +      - b0\ :sub:`4`
> +      - b0\ :sub:`3`
> +      - b0\ :sub:`2`
> +      - b0\ :sub:`1`
> +      - b0\ :sub:`0`
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +    * .. _MEDIA-BUS-FMT-META-1X8-16:
> +
> +      - MEDIA_BUS_FMT_META_1X8_16
> +      - 0x8005
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      -
> +      - b0\ :sub:`7`
> +      - b0\ :sub:`6`
> +      - b0\ :sub:`5`
> +      - b0\ :sub:`4`
> +      - b0\ :sub:`3`
> +      - b0\ :sub:`2`
> +      - b0\ :sub:`1`
> +      - b0\ :sub:`0`
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +    * .. _MEDIA-BUS-FMT-META-1X8-20:
> +
> +      - MEDIA_BUS_FMT_META_1X8_20
> +      - 0x8007
> +      -
> +      -
> +      -
> +      -
> +      -
> +      - b0\ :sub:`7`
> +      - b0\ :sub:`6`
> +      - b0\ :sub:`5`
> +      - b0\ :sub:`4`
> +      - b0\ :sub:`3`
> +      - b0\ :sub:`2`
> +      - b0\ :sub:`1`
> +      - b0\ :sub:`0`
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +    * .. _MEDIA-BUS-FMT-META-1X8-24:
> +
> +      - MEDIA_BUS_FMT_META_1X8_24
> +      - 0x8009
> +      -
> +      - b0\ :sub:`7`
> +      - b0\ :sub:`6`
> +      - b0\ :sub:`5`
> +      - b0\ :sub:`4`
> +      - b0\ :sub:`3`
> +      - b0\ :sub:`2`
> +      - b0\ :sub:`1`
> +      - b0\ :sub:`0`
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> +      - p
> diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
> index a03c543cb072..722463523bbd 100644
> --- a/include/uapi/linux/media-bus-format.h
> +++ b/include/uapi/linux/media-bus-format.h
> @@ -173,4 +173,13 @@
>   */
>  #define MEDIA_BUS_FMT_METADATA_FIXED		0x7001
>  
> +/* Generic line based metadata formats for serial buses. Next is 0x800b. */
> +#define MEDIA_BUS_FMT_META_1X8_8		0x8001
> +#define MEDIA_BUS_FMT_META_1X8_10		0x8002
> +#define MEDIA_BUS_FMT_META_1X8_12		0x8003
> +#define MEDIA_BUS_FMT_META_1X8_14		0x8004
> +#define MEDIA_BUS_FMT_META_1X8_16		0x8005
> +#define MEDIA_BUS_FMT_META_1X8_20		0x8007
> +#define MEDIA_BUS_FMT_META_1X8_24		0x8009
> +
>  #endif /* __LINUX_MEDIA_BUS_FORMAT_H */

Re: [RFC 6/7] media: uapi: Add generic 8-bit metadata format definitions

[Hans Verkuil]

On 05/05/2023 23:52, Sakari Ailus wrote:
> Generic 8-bit metadata formats define the in-memory data layout but not
> the format of the data itself. The reasoning for having such formats is to
> allow CSI-2 receiver drivers to receive and DMA drivers to write the data
> to memory without knowing a large number of device specific formats.
> 
> These formats may be used only in conjunction of a Media controller
> pipeline where the internal pad of the source sub-device defines the
> specific format of the data (using an mbus code).
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
>  .../userspace-api/media/v4l/meta-formats.rst  |   1 +
>  .../media/v4l/metafmt-generic.rst             | 317 ++++++++++++++++++
>  drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
>  include/uapi/linux/videodev2.h                |   9 +
>  4 files changed, 335 insertions(+)
>  create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
> 
> diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
> index 0bb61fc5bc00..919f595576b9 100644
> --- a/Documentation/userspace-api/media/v4l/meta-formats.rst
> +++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
> @@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only.
>      metafmt-vsp1-hgo
>      metafmt-vsp1-hgt
>      metafmt-vivid
> +    metafmt-generic
> diff --git a/Documentation/userspace-api/media/v4l/metafmt-generic.rst b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> new file mode 100644
> index 000000000000..09d3fabb1e94
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> @@ -0,0 +1,317 @@
> +.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
> +
> +**************************************************************************************************************************************************************************************************************************************************************************************************************************
> +V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O'), V4L2_META_FMT_GENERIC_CSI2_2_24 ('MC2O')
> +**************************************************************************************************************************************************************************************************************************************************************************************************************************
> +
> +
> +Generic line-based metadata formats
> +
> +
> +Description
> +===========
> +
> +These generic line-based metadata formats define the memory layout of the data
> +without defining the format or meaning of the metadata itself. These formats may
> +only be used with a Media controller pipeline where the more specific format is
> +defined in an :ref:`internal source pad <MEDIA-PAD-FL-INTERNAL-SOURCE>` of the
> +source sub-device. See also :ref:`source routes <v4l2-subdev-source-routes>`.
> +
> +.. _v4l2-meta-fmt-generic-8:
> +
> +V4L2_META_FMT_GENERIC_8
> +-----------------------
> +
> +The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format whereas the
> +rest of the formats have CSI-2 packing that are typically used on camera sensors
> +in conjunction with pixel data of the same bit depth.

You keep referring to CSI-2 packing without ever defining it.

I'm not sure you should include CSI2 in the name either. It is OK to say something
here like: "The way the data is stored follows the CSI-2 specification."

But it is not limited to CSI-2 and can be used with other buses as well.

> +
> +This format is also used on CSI-2 on both 8 bits per sample as well as on
> +16 bits per sample when two bytes of metadata are packed into one sample.

Little or big endian?

Regards,

	Hans

> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_8.**
> +Each cell is one byte. "M" denotes a byte of metadata.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|
> +
> +.. flat-table::
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - M\ :sub:`10`
> +      - M\ :sub:`20`
> +      - M\ :sub:`30`
> +    * - start + 4:
> +      - M\ :sub:`01`
> +      - M\ :sub:`11`
> +      - M\ :sub:`21`
> +      - M\ :sub:`31`
> +
> +.. _v4l2-meta-fmt-generic-csi2-10:
> +
> +V4L2_META_FMT_GENERIC_CSI2_10
> +-----------------------------
> +
> +V4L2_META_FMT_GENERIC_CSI2_10 contains 8-bit generic metadata with CSI-2
> +packing, 10 bits for each 8 bits of data. Every four bytes of metadata is
> +followed by a single byte of padding.
> +
> +This format is also used on CSI2 on 20 bits per sample format that packs two
> +bytes of metadata into one sample.
> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
> +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
> +
> +.. flat-table::
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - M\ :sub:`10`
> +      - M\ :sub:`20`
> +      - M\ :sub:`30`
> +      - p
> +    * - start + 5:
> +      - M\ :sub:`01`
> +      - M\ :sub:`11`
> +      - M\ :sub:`21`
> +      - M\ :sub:`31`
> +      - p
> +
> +.. _v4l2-meta-fmt-generic-csi2-12:
> +
> +V4L2_META_FMT_GENERIC_CSI2_12
> +-----------------------------
> +
> +V4L2_META_FMT_GENERIC_CSI2_12 contains 8-bit generic metadata with CSI-2
> +packing, 12 bits for each 8 bits of data. Every four bytes of metadata is
> +followed by two bytes of padding.
> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_12.**
> +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> +
> +.. flat-table::
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - M\ :sub:`10`
> +      - M\ :sub:`20`
> +      - M\ :sub:`30`
> +      - p
> +      - p
> +    * - start + 6:
> +      - M\ :sub:`01`
> +      - M\ :sub:`11`
> +      - M\ :sub:`21`
> +      - M\ :sub:`31`
> +      - p
> +      - p
> +
> +.. _v4l2-meta-fmt-generic-csi2-14:
> +
> +V4L2_META_FMT_GENERIC_CSI2_14
> +-----------------------------
> +
> +V4L2_META_FMT_GENERIC_CSI2_14 contains 8-bit generic metadata with CSI-2
> +packing, 14 bits for each 8 bits of data. Every four bytes of metadata is
> +followed by three bytes of padding.
> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_14.**
> +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{.8cm}|
> +
> +.. flat-table::
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - M\ :sub:`10`
> +      - M\ :sub:`20`
> +      - M\ :sub:`30`
> +      - p
> +      - p
> +      - p
> +    * - start + 7:
> +      - M\ :sub:`01`
> +      - M\ :sub:`11`
> +      - M\ :sub:`21`
> +      - M\ :sub:`31`
> +      - p
> +      - p
> +      - p
> +
> +.. _v4l2-meta-fmt-generic-csi2-16:
> +
> +V4L2_META_FMT_GENERIC_CSI2_16
> +-----------------------------
> +
> +V4L2_META_FMT_GENERIC_CSI2_16 contains 8-bit generic metadata with CSI-2
> +packing, 16 bits for each 8 bits of data. Every byte of metadata is
> +followed by one byte of padding.
> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_16.**
> +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|
> +
> +.. flat-table::
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - p
> +      - M\ :sub:`10`
> +      - p
> +      - M\ :sub:`20`
> +      - p
> +      - M\ :sub:`30`
> +      - p
> +    * - start + 8:
> +      - M\ :sub:`01`
> +      - p
> +      - M\ :sub:`11`
> +      - p
> +      - M\ :sub:`21`
> +      - p
> +      - M\ :sub:`31`
> +      - p
> +
> +.. _v4l2-meta-fmt-generic-csi2-20:
> +
> +V4L2_META_FMT_GENERIC_CSI2_20
> +-----------------------------
> +
> +V4L2_META_FMT_GENERIC_CSI2_20 contains 8-bit generic metadata with CSI-2
> +packing, 20 bits for each 8 bits of data. Every byte of metadata is followed by
> +alternating one and two bytes of padding.
> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_20.**
> +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> +
> +.. flat-table::
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8 8 8 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - p
> +      - M\ :sub:`10`
> +      - p
> +      - p
> +      - M\ :sub:`20`
> +      - p
> +      - M\ :sub:`30`
> +      - p
> +      - p
> +    * - start + 10:
> +      - M\ :sub:`01`
> +      - p
> +      - M\ :sub:`11`
> +      - p
> +      - p
> +      - M\ :sub:`21`
> +      - p
> +      - M\ :sub:`31`
> +      - p
> +      - p
> +
> +.. _v4l2-meta-fmt-generic-csi2-24:
> +
> +V4L2_META_FMT_GENERIC_CSI2_24
> +-----------------------------
> +
> +V4L2_META_FMT_GENERIC_CSI2_24 contains 8-bit generic metadata with CSI-2
> +packing, 24 bits for each 8 bits of data. Every byte of metadata is followed by
> +alternating one and two bytes of padding.
> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_24.**
> +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
> +
> +.. flat-table::
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8 8 8 8 8 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - p
> +      - p
> +      - M\ :sub:`10`
> +      - p
> +      - p
> +      - M\ :sub:`20`
> +      - p
> +      - p
> +      - M\ :sub:`30`
> +      - p
> +      - p
> +    * - start + 12:
> +      - M\ :sub:`01`
> +      - p
> +      - p
> +      - M\ :sub:`11`
> +      - p
> +      - p
> +      - M\ :sub:`21`
> +      - p
> +      - p
> +      - M\ :sub:`31`
> +      - p
> +      - p
> +
> +.. _v4l2-meta-fmt-generic-csi2-2-24:
> +
> +V4L2_META_FMT_GENERIC_CSI2_2_24
> +-------------------------------
> +
> +V4L2_META_FMT_GENERIC_CSI2_2_24 contains 8-bit generic metadata with CSI-2
> +packing, 24 bits for each two times 8 bits of data. Every two bytes of metadata
> +are followed by one byte of padding.
> +
> +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_2_24.**
> +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
> +
> +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
> +
> +.. flat-table::
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths: 12 8 8 8 8 8 8
> +
> +    * - start + 0:
> +      - M\ :sub:`00`
> +      - M\ :sub:`10`
> +      - p
> +      - M\ :sub:`20`
> +      - M\ :sub:`30`
> +      - p
> +    * - start + 6:
> +      - M\ :sub:`01`
> +      - M\ :sub:`11`
> +      - p
> +      - M\ :sub:`21`
> +      - M\ :sub:`31`
> +      - p
> +
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index a858acea6547..7781b0bd3e7d 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -1451,6 +1451,14 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
>  	case V4L2_PIX_FMT_Y210:		descr = "10-bit YUYV Packed"; break;
>  	case V4L2_PIX_FMT_Y212:		descr = "12-bit YUYV Packed"; break;
>  	case V4L2_PIX_FMT_Y216:		descr = "16-bit YUYV Packed"; break;
> +	case V4L2_META_FMT_GENERIC_8:	descr = "8-bit Generic Metadata"; break;
> +	case V4L2_META_FMT_GENERIC_CSI2_10:	descr = "8b Generic Meta, 10b CSI-2"; break;
> +	case V4L2_META_FMT_GENERIC_CSI2_12:	descr = "8b Generic Meta, 12b CSI-2"; break;
> +	case V4L2_META_FMT_GENERIC_CSI2_14:	descr = "8b Generic Meta, 14b CSI-2"; break;
> +	case V4L2_META_FMT_GENERIC_CSI2_16:	descr = "8b Generic Meta, 16b CSI-2"; break;
> +	case V4L2_META_FMT_GENERIC_CSI2_20:	descr = "8b Generic Meta, 20b CSI-2"; break;
> +	case V4L2_META_FMT_GENERIC_CSI2_24:	descr = "8b Generic Meta, 24b CSI-2"; break;
> +	case V4L2_META_FMT_GENERIC_CSI2_2_24:	descr = "2x8b Generic Meta, 24b CSI-2"; break;
>  
>  	default:
>  		/* Compressed formats */
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index aee75eb9e686..adcbdc15dcdb 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -834,6 +834,15 @@ struct v4l2_pix_format {
>  #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
>  #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
>  
> +#define V4L2_META_FMT_GENERIC_8		v4l2_fourcc('M', 'E', 'T', '8') /* Generic 8-bit metadata */
> +#define V4L2_META_FMT_GENERIC_CSI2_10	v4l2_fourcc('M', 'C', '1', 'A') /* 10-bit CSI-2 packed 8-bit metadata */
> +#define V4L2_META_FMT_GENERIC_CSI2_12	v4l2_fourcc('M', 'C', '1', 'C') /* 12-bit CSI-2 packed 8-bit metadata */
> +#define V4L2_META_FMT_GENERIC_CSI2_14	v4l2_fourcc('M', 'C', '1', 'E') /* 14-bit CSI-2 packed 8-bit metadata */
> +#define V4L2_META_FMT_GENERIC_CSI2_16	v4l2_fourcc('M', 'C', '1', 'G') /* 16-bit CSI-2 packed 8-bit metadata */
> +#define V4L2_META_FMT_GENERIC_CSI2_20	v4l2_fourcc('M', 'C', '1', 'K') /* 20-bit CSI-2 packed 8-bit metadata */
> +#define V4L2_META_FMT_GENERIC_CSI2_24	v4l2_fourcc('M', 'C', '1', 'O') /* 24-bit CSI-2 packed 8-bit metadata */
> +#define V4L2_META_FMT_GENERIC_CSI2_2_24	v4l2_fourcc('M', 'C', '2', 'O') /* 2 bytes of 8-bit metadata, 24-bit CSI-2 packed */
> +
>  /* priv field value to indicates that subsequent fields are valid. */
>  #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
>  

Re: [RFC 1/7] media: mc: Add INTERNAL_SOURCE pad type flag

[Sakari Ailus]

Hi Hans,

Thanks for the review.

On Thu, Jun 08, 2023 at 09:59:48AM +0200, Hans Verkuil wrote:
> On 05/05/2023 23:52, Sakari Ailus wrote:
> > Internal source pads will be used as routing endpoints in V4L2
> > [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
> > 
> > Also prevent creating links to pads that have been flagged as internal.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  .../userspace-api/media/mediactl/media-types.rst          | 7 +++++++
> >  drivers/media/mc/mc-entity.c                              | 8 +++++++-
> >  include/uapi/linux/media.h                                | 1 +
> >  3 files changed, 15 insertions(+), 1 deletion(-)
> > 
> > diff --git a/Documentation/userspace-api/media/mediactl/media-types.rst b/Documentation/userspace-api/media/mediactl/media-types.rst
> > index 0ffeece1e0c8..c724139ad46c 100644
> > --- a/Documentation/userspace-api/media/mediactl/media-types.rst
> > +++ b/Documentation/userspace-api/media/mediactl/media-types.rst
> > @@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
> >  .. _MEDIA-PAD-FL-SINK:
> >  .. _MEDIA-PAD-FL-SOURCE:
> >  .. _MEDIA-PAD-FL-MUST-CONNECT:
> > +.. _MEDIA-PAD-FL-INTERNAL-SOURCE:
> >  
> >  .. flat-table:: Media pad flags
> >      :header-rows:  0
> > @@ -382,6 +383,12 @@ Types and flags used to represent the media graph elements
> >  	  when this flag isn't set; the absence of the flag doesn't imply
> >  	  there is none.
> >  
> > +    *  -  ``MEDIA_PAD_FL_INTERNAL_SOURCE``
> > +       -  This flag indicates an internal pad that has no external
> 
> 'internal pad' or 'internal source pad'?
> 
> > +	  connections. Such a pad may not be connected with a link. The internal
> > +	  flag indicates that the stream either starts or ends in the
> > +	  entity.
> 
> This sentence makes no sense. The flag is for a pad, but then this sentence
> talks about streams and entities and doesn't relate that to the pad. Also,

Streams are between pads, but are visible on V4L2 only while this is MC
related. Documentation-wise this is a bit awkward.

> the flag is INTERNAL_SOURCE, and reading 'stream ends' suggests that it
> should be INTERNAL_SINK in that case.

When it comes to routing, it uses fields related to the sink pad, but this
is a source.

Based on review of the other patches, it seems both Tomi and Laurent would
prefer a plain INTERNAL flag which then would be used in conjunction with
the existing SINK flag, to note an internal source. Routing-wise this is a
cleaner option as this is associated with the sink related fields in any
case.

> 
> I think this needs to be more elaborate and ideally add some examples
> of the use-case(s) (or refer to an example elsewhere).
> 
>  For a given pad, the INTERNAL_SOURCE flag may not be set if
> > +	  either SINK or SOURCE flags is set.
> >  
> >  One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
> 
> and 'MEDIA_PAD_FL_INTERNAL_SOURCE' as well?
> 
> >  must be set for every pad.
> > diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> > index ef34ddd715c9..ed99160a2487 100644
> > --- a/drivers/media/mc/mc-entity.c
> > +++ b/drivers/media/mc/mc-entity.c
> > @@ -1062,7 +1062,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
> >  
> >  	for (i = 0; i < entity->num_pads; i++) {
> >  		if ((entity->pads[i].flags &
> > -		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
> > +		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
> > +		      MEDIA_PAD_FL_INTERNAL_SOURCE)) != pad_type)
> >  			continue;
> >  
> >  		if (entity->pads[i].sig_type == sig_type)
> > @@ -1087,6 +1088,11 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
> >  		return -EINVAL;
> >  	if (WARN_ON(!(sink->pads[sink_pad].flags & MEDIA_PAD_FL_SINK)))
> >  		return -EINVAL;
> > +	if (WARN_ON(sink->pads[sink_pad].flags &
> > +		    MEDIA_PAD_FL_INTERNAL_SOURCE) ||
> > +	    WARN_ON(source->pads[source_pad].flags &
> > +		    MEDIA_PAD_FL_INTERNAL_SOURCE))
> > +		return -EINVAL;
> >  
> >  	link = media_add_link(&source->links);
> >  	if (link == NULL)
> > diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
> > index edb8dfef9eba..0e2577e8b425 100644
> > --- a/include/uapi/linux/media.h
> > +++ b/include/uapi/linux/media.h
> > @@ -208,6 +208,7 @@ struct media_entity_desc {
> >  #define MEDIA_PAD_FL_SINK			(1U << 0)
> >  #define MEDIA_PAD_FL_SOURCE			(1U << 1)
> >  #define MEDIA_PAD_FL_MUST_CONNECT		(1U << 2)
> > +#define MEDIA_PAD_FL_INTERNAL_SOURCE		(1U << 3)
> >  
> >  struct media_pad_desc {
> >  	__u32 entity;		/* entity ID */
> 
> Regards,
> 
> 	Hans

-- 
Kind regards,

Sakari Ailus

Re: [RFC 3/7] media: uapi: v4l: Document source routes

[Sakari Ailus]

Hi Laurent,

On Fri, Jun 02, 2023 at 12:56:00PM +0300, Laurent Pinchart wrote:
> Hello,
> 
> On Mon, May 08, 2023 at 07:35:04PM +0300, Tomi Valkeinen wrote:
> > On 08/05/2023 19:26, Sakari Ailus wrote:
> > > On Mon, May 08, 2023 at 01:33:24PM +0300, Tomi Valkeinen wrote:
> > >> On 06/05/2023 00:52, Sakari Ailus wrote:
> > >>> Document how internal pads are used on source routes.
> > >>>
> > >>> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > >>> ---
> > >>>    .../userspace-api/media/v4l/dev-subdev.rst     | 18 ++++++++++++++++++
> > >>>    .../media/v4l/vidioc-subdev-g-routing.rst      |  5 +++++
> > >>>    include/uapi/linux/v4l2-subdev.h               |  6 +++++-
> > >>>    3 files changed, 28 insertions(+), 1 deletion(-)
> > >>>
> > >>> diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > >>> index a4f1df7093e8..395e06d2f0f2 100644
> > >>> --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > >>> +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > >>> @@ -551,6 +551,24 @@ A stream at a specific point in the media pipeline is identified by the
> > >>>    sub-device and a (pad, stream) pair. For sub-devices that do not support
> > >>>    multiplexed streams the 'stream' field is always 0.
> > >>> +.. _v4l2-subdev-source-routes:
> > >>> +
> > >>> +Source routes
> > >>> +^^^^^^^^^^^^^
> > >>> +
> > >>> +Cases where a single sub-device pad is a source of multiple streams are special
> > >>> +as there is no sink pad for such a route. In those cases, an internal pad is
> > >>> +used as the sink pad. Such pads have the :ref:`MEDIA_PAD_FL_INTERNAL_SOURCE
> > >>> +<MEDIA-PAD-FL-INTERNAL-SOURCE>` flag set.
> > >>> +
> > >>> +Internal pads have all the properties of a sink pad in such case, including
> > >>> +formats and selections. The format in this case is the source format of the
> > >>> +stream. An internal pad always has a single stream only (0).
> > >>> +
> > >>> +Generally source routes are not modifiable but they can be activated and
> > >>> +inactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > >>> +<v4l2-subdev-routing-flags>` flag, depending on driver capabilities.
> > >>> +
> > >>
> > >> I find the above chapter a bit difficult to read, but I think we need to
> > >> conclude on the internal-pad vs internal-source-pad discussion first.
> > >> However, one point/question:
> 
> Agreed, it's far from being clear :-( Even the first sentence, "Cases
> where a single sub-device pad is a source of multiple streams are
> special" is confusing, having a subdev source pad carrying multiple
> streams isn't special, it is found in, for instance, FPD-Link or GMSL
> receivers that transmit multiple streams from different cameras over a
> single CSI-2 output. This may not be what Sakari meant here, but it can
> be understood that way.

The sentence continues on the next line. I'll rework this part of the
documentation in any case while moving to a plain INTERNAL flag.

> 
> We need more than 3 paragraphs here, and we need a very clear example
> with a diagram. I'd recommend using a camera sensor that produces image
> data and embedded data for the example, as that's a common real-life use
> case. The text should present the example, explain what the problem is,
> and then explain how internal pads fix it. Then it can expand on the
> features and usage of internal pads in a more generic way.

It's fine to have examples but I'd add them to where the other examples are
located.

> 
> > >> You write that there's only one stream in an internal pad. I wonder if
> > >> that's a good or a necessary limitation... Do you see that allowing multiple
> > >> streams brings extra complexity? It's still up to each driver to decide how
> > >> many streams they support (most would just support a single one), so each
> > >> driver can easily enforce that limit.
> > > 
> > > Good question. As we don't seem to have a tangible reason to allow it, I'd
> > > deny it until there's a use case.
> > > 
> > > Or did you have a use case in mind?
> > > 
> > > I thought indicating which streams will be bound together (i.e. either are
> > > all disabled or enabled) could be one, but I'm not sure we need that at the
> > > moment at least.
> > 
> > I don't have nothing concrete in mind. Maybe a TPG which also generates 
> > some kind of metadata. But that could be implemented as two internal pads.
> > 
> > >> I'm fine with starting with only single-stream support, but if we later need
> > >> to loosen that restriction, I wonder if it'll be difficult if we have
> > >> specified that there can be only one stream. I.e. will the drivers and the
> > >> userspace depend on that limit.
> > > 
> > > We can always allow what wasn't allowed previously so that's a non-issue,
> > > really. But it needs to bring new functionality, otherwise there's no
> > > reason to do that.
> > 
> > It's not always that easy. If the drivers and the userspace expect that 
> > there's a single route with ID 0, and then with a new kernel there are 
> > more streams or the single stream is ID 1, things could go wrong.
> 
> I agree with Tomi here. On the kernel side it should be fine (with an
> unknown amount of pain), but I'd consider this as a userspace API
> breakage. If the specifications indicates that only a single stream can
> be used, applications may rely on that, and things could go wrong if new
> streams are added.
> 
> We can restrict the kernel implementation to a single stream, but the
> userspace API has to indicate that multiple streams can co-exist if we
> want to allow that in the future.

I doubt this ever could be an actual problem. Allowing multiple streams to
be merged on a source pad requires the user to first configure them.

I find it hard to believe there would be an application that tried to
configure routes this way (which would have always failed) and then
proceeded anyway, and somehow returning success from route setup leads to
routing configuration that the application couldn't handle anymore.

-- 
Kind regards,

Sakari Ailus

Re: [RFC 0/7] Generic line based metadata support, internal pads

[Sakari Ailus]

Hi Laurent,

On Fri, Jun 02, 2023 at 12:12:26PM +0300, Laurent Pinchart wrote:
> Hello,
> 
> On Fri, Jun 02, 2023 at 08:54:35AM +0100, Naushir Patuck wrote:
> > Hi Sakari,
> > 
> > Thank you for working on this.  Sensor metadata is something that Raspberry Pi
> > do make extensive use of, and our downstream changes to support it, although a
> > bit hacky, are not too dissimilar to your proposal here.
> > 
> > On Fri, 5 May 2023 at 22:53, Sakari Ailus wrote:
> > >
> > > Hi folks,
> > >
> > > Here are a few patches to add support generic, line based metadata as well
> > > as internal source pads. While the amount of code is not very large, to
> > > the contrary it is quite small actually IMO, I presume what this is about
> > > and why it is being proposed requires some explaining.
> > >
> > > Metadata mbus codes and formats have existed for some time in V4L2. They
> > > however have been only used by drivers that produce the data itself and
> > > effectively this metadata has always been statistics of some sort (at
> > > least when it comes to ISPs). What is different here is that we intend to
> > > add support for metadata originating from camera sensors.
> > >
> > > Camera sensors produce different kinds of metadata, embedded data (usually
> > > register address--value pairs used to capture the frame, in a more or less
> > > sensor specific format), histograms (in a very sensor specific format),
> > > dark pixels etc.
> 
> Optical dark pixels are image data, I wouldn't include them in the
> "metadata" category. They can of course be transmitted over a different
> stream, so they're relevant to the API being designed.

Good point, I'll address this for v2.

> 
> > > The number of these formats is probably going to be about
> > > as large as image data formats if not larger, as the image data formats
> > > are much better standardised but a smaller subset of them will be
> > > supported by V4L2, at least initially but possibly much more in the long
> > > run.
> 
> Strictly speaking, the number of metadata formats depends on how we
> define "format". Today, we can use the GREY pixel format to capture
> greyscale images in the visible spectrum, but also IR images, thermal
> images, or even depth images. They're all one pixel format. On the other
> hand, we have Y16 for greyscale visible and IR images, and Z16 for depth
> maps. It's already a mess, even without metadata :-)

Yes.

> 
> > > Having this many device specific formats would be a major problem for all
> > > the other drivers along that pipeline (not to mention the users of those
> > > drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2
> > > receiver drivers that have DMA: the poor driver developer would not only
> > > need to know camera sensor specific formats but to choose the specific
> > > packing of that format suitable for the DMA used by the hardware. It is
> > > unlikely many of these would ever get tested while being present on the
> > > driver API. Also adding new sensors with new embedded data formats would
> > > involve updating all bridge and CSI-2 receiver drivers. I don't expect
> > > this to be a workable approach.
> 
> I'm glad we agree on this.

Well, there are alternatives, but this is probably easier in the long run.

We'll just need more documentation than usually for the metadata mbus
codes.

> 
> > > Instead what I'm proposing is to use specific metadata formats on the
> > > sensor devices only, on internal pads (more about those soon) of the
> > > sensors, only visible in the UAPI, and then generic mbus formats along the
> > > pipeline and finally generic V4L2 metadata formats on the DMAs (specific
> > > to bit depth and packing). This would unsnarl the two, defining what data
> > > there is (specific mbus code) and how that is transported and packed
> > > (generic mbus codes and V4L2 formats).
> 
> Decoupling the information needed by the kernel (e.g. are we
> transporting RAW8 or RAW10 from the sensor through the pipeline) from
> the information useful for userspace only (e.g. the sensor embedded data
> is encoding using register + value pairs, based on the IMX708 registers
> set) is a good idea. I expect the main question to be where to draw the
> line between those two categories. Some pieces of information may be
> useless to any processing block in the pipeline except for an odd block
> in the middle.
> 
> This is, I believe, a problem similar to the CFA pattern. That
> information is useless for most devices, but required by the demosaicing
> block and some other blocks along the pipeline (colour gains for
> instance, or some Bayer statistics engines). We currently convey the CFA
> pattern in the media bus codes and pixel formats (e.g. SGRBG8 vs.
> SRGGB8) through the whole pipeline, while it could be conveyed out of
> band (e.g. exposed by the sensor using a control, and set on the devices
> that need it using a control as well).

Cases where another device in the pipeline would need to know the exact
metadata format are exceedingly rare. I actually don't have any I could
name right now.

> 
> If we come up with a good solution for metadata (and I hope we will),
> maybe we'll be able to use a similar mechanism for CFA patterns,
> simplifying new drivers and userspace. Or maybe this will remain a pipe
> dream given the backward compatibility implications.
> 
> > > The user space would be required to "know" the path of that data from the
> > > sensor's internal pad to the V4L2 video node. I do not see this as these
> > > devices require at least some knowledge of the pipeline, i.e. hardware at
> > > hand. Separating what the data means and how it is packed may even be
> > > beneficial: it allows separating code that interprets the data (sensor
> > > internal mbus code) from the code that accesses it (packing).
> > >
> > > These formats are in practice line based, meaning that there may be
> > > padding at the end of the line, depending on the bus as well as the DMA.
> > > If non-line based formats are needed, it is always possible to set the
> > > "height" field to 1.
> > 
> > One thing that may be worth considering or clarifying - for the case of the
> > BCM2835 Unicam CSI-2 device, we only have 2x DMA output channels.  So one will
> > match image data packets, and the other will match "everything else".  Typically
> > "everything else" would only be CSI-2 embedded data, but in the case of the
> > Raspberry Pi Camera v3 (IMX708), it includes embedded data, PDAF data, and
> > HDR histogram data.  Each of these outputs can be programmed to use a different
> > packet ID in the sensor, but since Unicam only has a single DMA for "everything
> > else", it all gets dumped into one metadata buffer.  But given we know the exact
> > structure of the data streams, it's trivial for useland to find the right bits
> > in this buffer.  Of course, other CSI-2 receivers with more DMA channels might
> > allow these streams to end up in their own buffers.
> > 
> > Nothing in your series seems to stop us operating Unicam in this way,
> > particularly because there is no fixed definition of the data format for
> > V4L2_META_FMT_GENERIC_8. So I don't think it's a problem, but perhaps it's worth
> > documenting that the metadata might include multiple streams from the sensor?
> 
> Thanks for your feedback Naush. Would you consider reviewing the
> individual patches in the series ? :-)
> 
> > > The internal (source) pads are an alternative to source routes [1]. The
> > > source routes were not universally liked and I do have to say I like
> > > re-using existing interface concepts (pads and everything you can do with
> > > pads, including access format, selections etc.) wherever it makes sense,
> > > instead of duplicating functionality.
> > >
> > > Effectively internal source pads behave mostly just like sink pads, but
> > > they describe a flow of data that originates from a sub-device instead of
> > > arriving to a sub-device. The SUBDEV_S_ROUTING IOCTLs are used to enable
> > > and disable routes from internal source pads to sub-device's source pads.
> > > The subdev format IOCTLs are usable, too, so one can find which subdev
> > > format is available on given internal source pad.
> > >
> > > This set depends on these patches:
> > >
> > > <URL:https://lore.kernel.org/linux-media/20230505205416.55002-1-sakari.ailus@linux.intel.com/T/#t>
> > >
> > > I've also pushed these here and I'll keep updating the branch:
> > >
> > > <URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=metadata>
> > >
> > > Questions and comments are most welcome.
> > >
> > >
> > > [1] <URL:https://lore.kernel.org/linux-media/20220831141357.1396081-20-tomi.valkeinen@ideasonboard.com/>
> > >
> > > Sakari Ailus (7):
> > >   media: mc: Add INTERNAL_SOURCE pad type flag
> > >   media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs
> > >   media: uapi: v4l: Document source routes
> > >   media: mc: Check pad flag validity
> > >   media: uapi: Add generic serial metadata mbus formats
> > >   media: uapi: Add generic 8-bit metadata format definitions
> > >   media: v4l: Support line-based metadata capture
> > >
> > >  .../media/mediactl/media-types.rst            |   7 +
> > >  .../userspace-api/media/v4l/dev-meta.rst      |  15 +
> > >  .../userspace-api/media/v4l/dev-subdev.rst    |  18 +
> > >  .../userspace-api/media/v4l/meta-formats.rst  |   1 +
> > >  .../media/v4l/metafmt-generic.rst             | 317 ++++++++++++++++++
> > >  .../media/v4l/subdev-formats.rst              | 257 ++++++++++++++
> > >  .../media/v4l/vidioc-enum-fmt.rst             |   7 +
> > >  .../media/v4l/vidioc-subdev-g-routing.rst     |   5 +
> > >  drivers/media/mc/mc-entity.c                  |  20 +-
> > >  drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
> > >  drivers/media/v4l2-core/v4l2-subdev.c         |   6 +-
> > >  include/uapi/linux/media-bus-format.h         |   9 +
> > >  include/uapi/linux/media.h                    |   1 +
> > >  include/uapi/linux/v4l2-subdev.h              |   6 +-
> > >  include/uapi/linux/videodev2.h                |  19 ++
> > >  15 files changed, 691 insertions(+), 5 deletions(-)
> > >  create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst

-- 
Kind regards,

Sakari Ailus

Re: [RFC 5/7] media: uapi: Add generic serial metadata mbus formats

[Sakari Ailus]

Hi Hans,

On Thu, Jun 08, 2023 at 10:35:38AM +0200, Hans Verkuil wrote:
> On 05/05/2023 23:52, Sakari Ailus wrote:
> > Add generic serial metadata mbus formats. These formats describe data
> > width and packing but not the content itself. The reason for specifying
> > such formats is that the formats as such are fairly device specific but
> > they are still handled by CSI-2 receiver drivers that should not be aware
> > of device specific formats. What makes generic metadata formats possible
> > is that these formats are parsed by software only, after capturing the
> > data to system memory.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  .../media/v4l/subdev-formats.rst              | 257 ++++++++++++++++++
> >  include/uapi/linux/media-bus-format.h         |   9 +
> >  2 files changed, 266 insertions(+)
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > index a3a35eeed708..1492fff58426 100644
> > --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > @@ -8234,3 +8234,260 @@ The following table lists the existing metadata formats.
> >  	both sides of the link and the bus format is a fixed
> >  	metadata format that is not configurable from userspace.
> >  	Width and height will be set to 0 for this format.
> > +
> > +Generic Serial Metadata Formats
> > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > +
> > +Generic serial metadata formats are used on serial busses where the actual data
> > +content is more or less device specific but the data is transmitted and received
> > +by multiple devices that do not process the data in any way, simply writing
> > +it to system memory for processing in software at the end of the pipeline.
> > +
> > +The more specific variant describing the actual data is used on the internal
> > +source pad of the originating sub-device.
> 
> As Laurent said as well, the mention of 'internal source pad' is confusing.
> 
> Aren't there two possibilities here? Either the entity has just a source pad
> that delivers the metadata, and that source pad has the precise metadata
> format. Or the entity supports routing and then there is a sink pad connected
> to an internal source that is routed to a source pad. In that case that sink
> pad has the precise metadata format.
> 
> Apologies if I am wrong, it's hard to keep track of all the possibilities,
> especially if you do not use this regularly.

It's required the generic metadata format is present only on internal pads,
otherwise link validation will fail --- the next device won't be aware of
the specific mbus code.

> 
> > +
> > +"b" in an array cell signifies a byte of data, followed by the number of byte
> > +and finally the bit number in subscript. "p" indicates a padding bit.
> > +
> > +.. _media-bus-format-generic-meta:
> > +
> > +.. cssclass: longtable
> > +
> > +.. flat-table:: Generic Serial Metadata Formats
> > +    :header-rows:  2
> > +    :stub-columns: 0
> > +
> > +    * - Identifier
> > +      - Code
> > +      -
> > +      - :cspan:`23` Data organization
> > +    * -
> > +      -
> > +      - Bit
> > +      - 23
> > +      - 22
> > +      - 21
> > +      - 20
> > +      - 19
> > +      - 18
> > +      - 17
> > +      - 16
> > +      - 15
> > +      - 14
> > +      - 13
> > +      - 12
> > +      - 11
> > +      - 10
> > +      - 9
> > +      - 8
> > +      - 7
> > +      - 6
> > +      - 5
> > +      - 4
> > +      - 3
> > +      - 2
> > +      - 1
> > +      - 0
> > +    * .. _MEDIA-BUS-FMT-META-1X8-8:
> > +
> > +      - MEDIA_BUS_FMT_META_1X8_8
> > +      - 0x8001
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      - b0\ :sub:`7`
> > +      - b0\ :sub:`6`
> > +      - b0\ :sub:`5`
> > +      - b0\ :sub:`4`
> > +      - b0\ :sub:`3`
> > +      - b0\ :sub:`2`
> > +      - b0\ :sub:`1`
> > +      - b0\ :sub:`0`
> > +    * .. _MEDIA-BUS-FMT-META-1X8-10:
> > +
> > +      - MEDIA_BUS_FMT_META_1X8_10
> > +      - 0x8002
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      - b0\ :sub:`7`
> > +      - b0\ :sub:`6`
> > +      - b0\ :sub:`5`
> > +      - b0\ :sub:`4`
> > +      - b0\ :sub:`3`
> > +      - b0\ :sub:`2`
> > +      - b0\ :sub:`1`
> > +      - b0\ :sub:`0`
> > +      - p
> > +      - p
> > +    * .. _MEDIA-BUS-FMT-META-1X8-12:
> > +
> > +      - MEDIA_BUS_FMT_META_1X8_12
> > +      - 0x8003
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      - b0\ :sub:`7`
> > +      - b0\ :sub:`6`
> > +      - b0\ :sub:`5`
> > +      - b0\ :sub:`4`
> > +      - b0\ :sub:`3`
> > +      - b0\ :sub:`2`
> > +      - b0\ :sub:`1`
> > +      - b0\ :sub:`0`
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +    * .. _MEDIA-BUS-FMT-META-1X8-14:
> > +
> > +      - MEDIA_BUS_FMT_META_1X8_14
> > +      - 0x8004
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      - b0\ :sub:`7`
> > +      - b0\ :sub:`6`
> > +      - b0\ :sub:`5`
> > +      - b0\ :sub:`4`
> > +      - b0\ :sub:`3`
> > +      - b0\ :sub:`2`
> > +      - b0\ :sub:`1`
> > +      - b0\ :sub:`0`
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +    * .. _MEDIA-BUS-FMT-META-1X8-16:
> > +
> > +      - MEDIA_BUS_FMT_META_1X8_16
> > +      - 0x8005
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      - b0\ :sub:`7`
> > +      - b0\ :sub:`6`
> > +      - b0\ :sub:`5`
> > +      - b0\ :sub:`4`
> > +      - b0\ :sub:`3`
> > +      - b0\ :sub:`2`
> > +      - b0\ :sub:`1`
> > +      - b0\ :sub:`0`
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +    * .. _MEDIA-BUS-FMT-META-1X8-20:
> > +
> > +      - MEDIA_BUS_FMT_META_1X8_20
> > +      - 0x8007
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      - b0\ :sub:`7`
> > +      - b0\ :sub:`6`
> > +      - b0\ :sub:`5`
> > +      - b0\ :sub:`4`
> > +      - b0\ :sub:`3`
> > +      - b0\ :sub:`2`
> > +      - b0\ :sub:`1`
> > +      - b0\ :sub:`0`
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +    * .. _MEDIA-BUS-FMT-META-1X8-24:
> > +
> > +      - MEDIA_BUS_FMT_META_1X8_24
> > +      - 0x8009
> > +      -
> > +      - b0\ :sub:`7`
> > +      - b0\ :sub:`6`
> > +      - b0\ :sub:`5`
> > +      - b0\ :sub:`4`
> > +      - b0\ :sub:`3`
> > +      - b0\ :sub:`2`
> > +      - b0\ :sub:`1`
> > +      - b0\ :sub:`0`
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
> > index a03c543cb072..722463523bbd 100644
> > --- a/include/uapi/linux/media-bus-format.h
> > +++ b/include/uapi/linux/media-bus-format.h
> > @@ -173,4 +173,13 @@
> >   */
> >  #define MEDIA_BUS_FMT_METADATA_FIXED		0x7001
> >  
> > +/* Generic line based metadata formats for serial buses. Next is 0x800b. */
> > +#define MEDIA_BUS_FMT_META_1X8_8		0x8001
> > +#define MEDIA_BUS_FMT_META_1X8_10		0x8002
> > +#define MEDIA_BUS_FMT_META_1X8_12		0x8003
> > +#define MEDIA_BUS_FMT_META_1X8_14		0x8004
> > +#define MEDIA_BUS_FMT_META_1X8_16		0x8005
> 
> You skip 0x8006 here in case there will be a _18 variant?

I removed duplicate entries but forgot to change the values. I'll address
that for v2.

There are different packings defined for these formats, sensor generally
using the pixel data equivalent format. With 16 and more bits per sample,
multiple octets of metadata can be transported in that sample, and those
formats are equivalent to the formats at half (or third) of the sample
size. Say, alternative format for 16 bpp image data to 1X8_16 is 1X8_8.

-- 
Kind regards,

Sakari Ailus

Re: [RFC 5/7] media: uapi: Add generic serial metadata mbus formats

[Sakari Ailus]

Hi Hans,

On Thu, Jun 08, 2023 at 10:46:28AM +0200, Hans Verkuil wrote:
> On 05/05/2023 23:52, Sakari Ailus wrote:
> > Add generic serial metadata mbus formats. These formats describe data
> > width and packing but not the content itself. The reason for specifying
> > such formats is that the formats as such are fairly device specific but
> > they are still handled by CSI-2 receiver drivers that should not be aware
> > of device specific formats. What makes generic metadata formats possible
> > is that these formats are parsed by software only, after capturing the
> > data to system memory.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  .../media/v4l/subdev-formats.rst              | 257 ++++++++++++++++++
> >  include/uapi/linux/media-bus-format.h         |   9 +
> >  2 files changed, 266 insertions(+)
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > index a3a35eeed708..1492fff58426 100644
> > --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > @@ -8234,3 +8234,260 @@ The following table lists the existing metadata formats.
> >  	both sides of the link and the bus format is a fixed
> >  	metadata format that is not configurable from userspace.
> >  	Width and height will be set to 0 for this format.
> > +
> > +Generic Serial Metadata Formats
> > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > +
> > +Generic serial metadata formats are used on serial busses where the actual data
> > +content is more or less device specific but the data is transmitted and received
> > +by multiple devices that do not process the data in any way, simply writing
> > +it to system memory for processing in software at the end of the pipeline.
> > +
> > +The more specific variant describing the actual data is used on the internal
> > +source pad of the originating sub-device.
> 
> I'm a bit confused here: on which pads are these generic mbus formats
> set? And who sets this?

As with other pads, the user can set the format. In most cases only a
single option is available whenever the device ourputs metadata over a
stream (and thus uses on of the generic line based metadata formats).

> 
> I.e. the internal source pad has e.g. MEDIA_BUS_FMT_META_FOOBAR_HIST as
> specific format. It's routed to a source pad + stream pair for transfer
> over CSI. Does that source pad + stream pair use the generic mediabus
> format?

Correct.

-- 
Sakari Ailus

Re: [RFC 7/7] media: v4l: Support line-based metadata capture

[Sakari Ailus]

Hi Hans,

On Fri, Jun 02, 2023 at 01:50:54PM +0300, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.
> 
> On Sat, May 06, 2023 at 12:52:57AM +0300, Sakari Ailus wrote:
> > many camera sensors, among other devices, transmit embedded data and image
> 
> s/many/Many/
> 
> > data for each CSI-2 frame. This embedded data typically contains register
> > configuration of the sensor that has been used to capture the image data
> > of the same frame.
> > 
> > The embedded data is received by the CSI-2 receiver and has the same
> > properties as the image data, including that it is line based: it has
> > width, height and bytesperline (stride).
> > 
> > Add these fields to struct v4l2_meta_format and document them.
> > 
> > Also add V4L2_FMT_FLAG_META_LINE_BASED to tell a given format is
> > line-based i.e. these fields of struct v4l2_meta_format are valid for it.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  .../userspace-api/media/v4l/dev-meta.rst          | 15 +++++++++++++++
> >  .../userspace-api/media/v4l/vidioc-enum-fmt.rst   |  7 +++++++
> >  include/uapi/linux/videodev2.h                    | 10 ++++++++++
> >  3 files changed, 32 insertions(+)
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/dev-meta.rst b/Documentation/userspace-api/media/v4l/dev-meta.rst
> > index 0e7e1ee1471a..7d3a64514db0 100644
> > --- a/Documentation/userspace-api/media/v4l/dev-meta.rst
> > +++ b/Documentation/userspace-api/media/v4l/dev-meta.rst
> > @@ -65,3 +65,18 @@ to 0.
> >        - ``buffersize``
> >        - Maximum buffer size in bytes required for data. The value is set by the
> >          driver.
> > +    * - __u32
> > +      - ``width``
> > +      - Width of a line of metadata in bytes. Valid when :c:type`v4l2_fmtdesc`
> 
> This departs from pixel formats, where the width is defined in pixels. I
> wonder what the implications will be for userspace. Seeing one
> implementation, both in a kernel driver and in libcamera, will help
> validating the API.

I thought of bytes when writing this but pixels (or samples) are probably a
better term for this.

> 
> > +	flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
> > +	:c:func:`VIDIOC_ENUM_FMT`.
> > +    * - __u32
> > +      - ``height``
> > +      - Height of a line of metadata in bytes. Valid when :c:type`v4l2_fmtdesc`
> 
> The "height of a line" seems like a weird concept, especially if the
> height is expressed in bytes. I assume this is a bad copy&paste.

Yes. I'll address these for v2.

> 
> > +	flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
> > +	:c:func:`VIDIOC_ENUM_FMT`.
> > +    * - __u32
> > +      - ``bytesperlines``
> > +      - Offset in bytes between the beginning of two consecutive lines. Valid
> > +	when :c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is
> > +	set, otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
> > diff --git a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> > index 000c154b0f98..6d7664345a4e 100644
> > --- a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> > +++ b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> > @@ -227,6 +227,13 @@ the ``mbus_code`` field is handled differently:
> >  	The application can ask to configure the quantization of the capture
> >  	device when calling the :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl with
> >  	:ref:`V4L2_PIX_FMT_FLAG_SET_CSC <v4l2-pix-fmt-flag-set-csc>` set.
> > +    * - ``V4L2_FMT_FLAG_META_LINE_BASED``
> > +      - 0x0200
> > +      - The metadata format is line-based. In this case the ``width``,
> > +	``height`` and ``bytesperline`` fields of :c:type:`v4l2_meta_format` are
> > +	valid. The buffer consists of ``height`` lines, each having ``width``
> > +	bytes of data and offset between the beginning of each two consecutive
> > +	lines is ``bytesperline``.
> 
> If we add width and height for metadata formats, does it mean that
> drivers have to (or can) implement VIDIOC_ENUM_FRAMESIZES ? This should
> be documented.

Good point.

I don't think it's meaningful to implement that as the pipeline
configuration determines this in any case.

But I wonder whether this would better be documented for V4L2_CAP_IO_MC
rather than for metadata formats. There's currently only a reference to
this in ENUM_FMT documentation.

> 
> >  
> >  Return Value
> >  ============
> > diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> > index adcbdc15dcdb..3681b2c15901 100644
> > --- a/include/uapi/linux/videodev2.h
> > +++ b/include/uapi/linux/videodev2.h
> > @@ -873,6 +873,7 @@ struct v4l2_fmtdesc {
> >  #define V4L2_FMT_FLAG_CSC_YCBCR_ENC		0x0080
> >  #define V4L2_FMT_FLAG_CSC_HSV_ENC		V4L2_FMT_FLAG_CSC_YCBCR_ENC
> >  #define V4L2_FMT_FLAG_CSC_QUANTIZATION		0x0100
> > +#define V4L2_FMT_FLAG_META_LINE_BASED		0x0200
> >  
> >  	/* Frame Size and frame rate enumeration */
> >  /*
> > @@ -2407,10 +2408,19 @@ struct v4l2_sdr_format {
> >   * struct v4l2_meta_format - metadata format definition
> >   * @dataformat:		little endian four character code (fourcc)
> >   * @buffersize:		maximum size in bytes required for data
> > + * @width:		number of bytes of data per line (valid for line based
> > + *			formats only, see format documentation)
> > + * @height:		number of lines of data per buffer (valid for line based
> > + *			formats only)
> > + * @bytesperline:	offset between the beginnings of two adjacent lines in
> > + *			bytes (valid for line based formats only)
> >   */
> >  struct v4l2_meta_format {
> >  	__u32				dataformat;
> >  	__u32				buffersize;
> > +	__u32				width;
> > +	__u32				height;
> > +	__u32				bytesperline;
> >  } __attribute__ ((packed));
> >  
> >  /**

-- 
Regards,

Sakari Ailus

Re: [RFC 0/7] Generic line based metadata support, internal pads

[Dave Stevenson]

Hi Sakari

Sorry to be late to the party.

On Fri, 5 May 2023 at 22:53, Sakari Ailus <sakari.ailus@linux.intel.com> wrote:
>
> Hi folks,
>
> Here are a few patches to add support generic, line based metadata as well
> as internal source pads. While the amount of code is not very large, to
> the contrary it is quite small actually IMO, I presume what this is about
> and why it is being proposed requires some explaining.
>
> Metadata mbus codes and formats have existed for some time in V4L2. They
> however have been only used by drivers that produce the data itself and
> effectively this metadata has always been statistics of some sort (at
> least when it comes to ISPs). What is different here is that we intend to
> add support for metadata originating from camera sensors.
>
> Camera sensors produce different kinds of metadata, embedded data (usually
> register address--value pairs used to capture the frame, in a more or less
> sensor specific format), histograms (in a very sensor specific format),
> dark pixels etc. The number of these formats is probably going to be about
> as large as image data formats if not larger, as the image data formats
> are much better standardised but a smaller subset of them will be
> supported by V4L2, at least initially but possibly much more in the long
> run.
>
> Having this many device specific formats would be a major problem for all
> the other drivers along that pipeline (not to mention the users of those
> drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2
> receiver drivers that have DMA: the poor driver developer would not only
> need to know camera sensor specific formats but to choose the specific
> packing of that format suitable for the DMA used by the hardware. It is
> unlikely many of these would ever get tested while being present on the
> driver API. Also adding new sensors with new embedded data formats would
> involve updating all bridge and CSI-2 receiver drivers. I don't expect
> this to be a workable approach.
>
> Instead what I'm proposing is to use specific metadata formats on the
> sensor devices only, on internal pads (more about those soon) of the
> sensors, only visible in the UAPI, and then generic mbus formats along the
> pipeline and finally generic V4L2 metadata formats on the DMAs (specific
> to bit depth and packing). This would unsnarl the two, defining what data
> there is (specific mbus code) and how that is transported and packed
> (generic mbus codes and V4L2 formats).
>
> The user space would be required to "know" the path of that data from the
> sensor's internal pad to the V4L2 video node. I do not see this as these
> devices require at least some knowledge of the pipeline, i.e. hardware at
> hand. Separating what the data means and how it is packed may even be
> beneficial: it allows separating code that interprets the data (sensor
> internal mbus code) from the code that accesses it (packing).
>
> These formats are in practice line based, meaning that there may be
> padding at the end of the line, depending on the bus as well as the DMA.
> If non-line based formats are needed, it is always possible to set the
> "height" field to 1.
>
> The internal (source) pads are an alternative to source routes [1]. The
> source routes were not universally liked and I do have to say I like
> re-using existing interface concepts (pads and everything you can do with
> pads, including access format, selections etc.) wherever it makes sense,
> instead of duplicating functionality.
>
> Effectively internal source pads behave mostly just like sink pads, but
> they describe a flow of data that originates from a sub-device instead of
> arriving to a sub-device. The SUBDEV_S_ROUTING IOCTLs are used to enable
> and disable routes from internal source pads to sub-device's source pads.
> The subdev format IOCTLs are usable, too, so one can find which subdev
> format is available on given internal source pad.

Just to confirm my own understanding, the muxed streams API currently
copes with situations such as the FPD-Link devices combining a single
stream from each of a number of subdevs into one multiplexed stream
using virtual channels, but it doesn't handle multiple data types from
the same subdev. You're addressing that omission.

All seems reasonable.

One detail that I haven't seen covered and think ought to be clarified.
For raw sensors where you have image data and metadata, in my
experience the line width of that metadata and the packing format
(8/10/12/14/16/20/24 bit) are all dictated by the image data
configuration. You can't have eg 640 pixel wide _SBGGR10_1X10 image
data and 320 pixel wide _META_1X8_8 metadata. (it has to be 640 pixel
wide _META_1X10_10)

Which layer would be responsible for validating the configuration, and when?
a) The driver validates in set_stream, and fails if mismatched. That
potentially results in lots of duplication between drivers.
b) set_pad_format on the image pad updates the formats and widths of
the metadata pads automatically. Is there a way of notifying clients
that the formats on potentially unrelated pads has changed, or is it
something that just gets documented?
c) something else.

Either is workable, but IMHO it needs to be specified as to which is
the expected behaviour.

Thanks
  Dave

> This set depends on these patches:
>
> <URL:https://lore.kernel.org/linux-media/20230505205416.55002-1-sakari.ailus@linux.intel.com/T/#t>
>
> I've also pushed these here and I'll keep updating the branch:
>
> <URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=metadata>
>
> Questions and comments are most welcome.
>
>
> [1] <URL:https://lore.kernel.org/linux-media/20220831141357.1396081-20-tomi.valkeinen@ideasonboard.com/>
>
> Sakari Ailus (7):
>   media: mc: Add INTERNAL_SOURCE pad type flag
>   media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs
>   media: uapi: v4l: Document source routes
>   media: mc: Check pad flag validity
>   media: uapi: Add generic serial metadata mbus formats
>   media: uapi: Add generic 8-bit metadata format definitions
>   media: v4l: Support line-based metadata capture
>
>  .../media/mediactl/media-types.rst            |   7 +
>  .../userspace-api/media/v4l/dev-meta.rst      |  15 +
>  .../userspace-api/media/v4l/dev-subdev.rst    |  18 +
>  .../userspace-api/media/v4l/meta-formats.rst  |   1 +
>  .../media/v4l/metafmt-generic.rst             | 317 ++++++++++++++++++
>  .../media/v4l/subdev-formats.rst              | 257 ++++++++++++++
>  .../media/v4l/vidioc-enum-fmt.rst             |   7 +
>  .../media/v4l/vidioc-subdev-g-routing.rst     |   5 +
>  drivers/media/mc/mc-entity.c                  |  20 +-
>  drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
>  drivers/media/v4l2-core/v4l2-subdev.c         |   6 +-
>  include/uapi/linux/media-bus-format.h         |   9 +
>  include/uapi/linux/media.h                    |   1 +
>  include/uapi/linux/v4l2-subdev.h              |   6 +-
>  include/uapi/linux/videodev2.h                |  19 ++
>  15 files changed, 691 insertions(+), 5 deletions(-)
>  create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
>
> --
> 2.30.2
>

Re: [RFC 6/7] media: uapi: Add generic 8-bit metadata format definitions

[Sakari Ailus]

Hi Laurent,

On Thu, Jun 08, 2023 at 10:54:43AM +0200, Hans Verkuil wrote:
> On 05/05/2023 23:52, Sakari Ailus wrote:
> > Generic 8-bit metadata formats define the in-memory data layout but not
> > the format of the data itself. The reasoning for having such formats is to
> > allow CSI-2 receiver drivers to receive and DMA drivers to write the data
> > to memory without knowing a large number of device specific formats.
> > 
> > These formats may be used only in conjunction of a Media controller
> > pipeline where the internal pad of the source sub-device defines the
> > specific format of the data (using an mbus code).
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  .../userspace-api/media/v4l/meta-formats.rst  |   1 +
> >  .../media/v4l/metafmt-generic.rst             | 317 ++++++++++++++++++
> >  drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
> >  include/uapi/linux/videodev2.h                |   9 +
> >  4 files changed, 335 insertions(+)
> >  create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
> > index 0bb61fc5bc00..919f595576b9 100644
> > --- a/Documentation/userspace-api/media/v4l/meta-formats.rst
> > +++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
> > @@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only.
> >      metafmt-vsp1-hgo
> >      metafmt-vsp1-hgt
> >      metafmt-vivid
> > +    metafmt-generic
> > diff --git a/Documentation/userspace-api/media/v4l/metafmt-generic.rst b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > new file mode 100644
> > index 000000000000..09d3fabb1e94
> > --- /dev/null
> > +++ b/Documentation/userspace-api/media/v4l/metafmt-generic.rst
> > @@ -0,0 +1,317 @@
> > +.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
> > +
> > +**************************************************************************************************************************************************************************************************************************************************************************************************************************
> > +V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O'), V4L2_META_FMT_GENERIC_CSI2_2_24 ('MC2O')
> > +**************************************************************************************************************************************************************************************************************************************************************************************************************************
> > +
> > +
> > +Generic line-based metadata formats
> > +
> > +
> > +Description
> > +===========
> > +
> > +These generic line-based metadata formats define the memory layout of the data
> > +without defining the format or meaning of the metadata itself. These formats may
> > +only be used with a Media controller pipeline where the more specific format is
> > +defined in an :ref:`internal source pad <MEDIA-PAD-FL-INTERNAL-SOURCE>` of the
> > +source sub-device. See also :ref:`source routes <v4l2-subdev-source-routes>`.
> > +
> > +.. _v4l2-meta-fmt-generic-8:
> > +
> > +V4L2_META_FMT_GENERIC_8
> > +-----------------------
> > +
> > +The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format whereas the
> > +rest of the formats have CSI-2 packing that are typically used on camera sensors
> > +in conjunction with pixel data of the same bit depth.
> 
> You keep referring to CSI-2 packing without ever defining it.
> 
> I'm not sure you should include CSI2 in the name either. It is OK to say something
> here like: "The way the data is stored follows the CSI-2 specification."
> 
> But it is not limited to CSI-2 and can be used with other buses as well.

That's true, certainly, but where this packing is most commonly used is
CSI-2 (or derives from CSI-2).

> 
> > +
> > +This format is also used on CSI-2 on both 8 bits per sample as well as on
> > +16 bits per sample when two bytes of metadata are packed into one sample.
> 
> Little or big endian?

Little endian. I'll address these for v2.

-- 
Kind regards,

Sakari Ailus

Re: [RFC 0/7] Generic line based metadata support, internal pads

[Sakari Ailus]

Hi Dave,

On Fri, Jun 09, 2023 at 02:59:20PM +0100, Dave Stevenson wrote:
> Hi Sakari
> 
> Sorry to be late to the party.
> 
> On Fri, 5 May 2023 at 22:53, Sakari Ailus <sakari.ailus@linux.intel.com> wrote:
> >
> > Hi folks,
> >
> > Here are a few patches to add support generic, line based metadata as well
> > as internal source pads. While the amount of code is not very large, to
> > the contrary it is quite small actually IMO, I presume what this is about
> > and why it is being proposed requires some explaining.
> >
> > Metadata mbus codes and formats have existed for some time in V4L2. They
> > however have been only used by drivers that produce the data itself and
> > effectively this metadata has always been statistics of some sort (at
> > least when it comes to ISPs). What is different here is that we intend to
> > add support for metadata originating from camera sensors.
> >
> > Camera sensors produce different kinds of metadata, embedded data (usually
> > register address--value pairs used to capture the frame, in a more or less
> > sensor specific format), histograms (in a very sensor specific format),
> > dark pixels etc. The number of these formats is probably going to be about
> > as large as image data formats if not larger, as the image data formats
> > are much better standardised but a smaller subset of them will be
> > supported by V4L2, at least initially but possibly much more in the long
> > run.
> >
> > Having this many device specific formats would be a major problem for all
> > the other drivers along that pipeline (not to mention the users of those
> > drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2
> > receiver drivers that have DMA: the poor driver developer would not only
> > need to know camera sensor specific formats but to choose the specific
> > packing of that format suitable for the DMA used by the hardware. It is
> > unlikely many of these would ever get tested while being present on the
> > driver API. Also adding new sensors with new embedded data formats would
> > involve updating all bridge and CSI-2 receiver drivers. I don't expect
> > this to be a workable approach.
> >
> > Instead what I'm proposing is to use specific metadata formats on the
> > sensor devices only, on internal pads (more about those soon) of the
> > sensors, only visible in the UAPI, and then generic mbus formats along the
> > pipeline and finally generic V4L2 metadata formats on the DMAs (specific
> > to bit depth and packing). This would unsnarl the two, defining what data
> > there is (specific mbus code) and how that is transported and packed
> > (generic mbus codes and V4L2 formats).
> >
> > The user space would be required to "know" the path of that data from the
> > sensor's internal pad to the V4L2 video node. I do not see this as these
> > devices require at least some knowledge of the pipeline, i.e. hardware at
> > hand. Separating what the data means and how it is packed may even be
> > beneficial: it allows separating code that interprets the data (sensor
> > internal mbus code) from the code that accesses it (packing).
> >
> > These formats are in practice line based, meaning that there may be
> > padding at the end of the line, depending on the bus as well as the DMA.
> > If non-line based formats are needed, it is always possible to set the
> > "height" field to 1.
> >
> > The internal (source) pads are an alternative to source routes [1]. The
> > source routes were not universally liked and I do have to say I like
> > re-using existing interface concepts (pads and everything you can do with
> > pads, including access format, selections etc.) wherever it makes sense,
> > instead of duplicating functionality.
> >
> > Effectively internal source pads behave mostly just like sink pads, but
> > they describe a flow of data that originates from a sub-device instead of
> > arriving to a sub-device. The SUBDEV_S_ROUTING IOCTLs are used to enable
> > and disable routes from internal source pads to sub-device's source pads.
> > The subdev format IOCTLs are usable, too, so one can find which subdev
> > format is available on given internal source pad.
> 
> Just to confirm my own understanding, the muxed streams API currently
> copes with situations such as the FPD-Link devices combining a single
> stream from each of a number of subdevs into one multiplexed stream
> using virtual channels, but it doesn't handle multiple data types from
> the same subdev. You're addressing that omission.

Correct.

> 
> All seems reasonable.
> 
> One detail that I haven't seen covered and think ought to be clarified.
> For raw sensors where you have image data and metadata, in my
> experience the line width of that metadata and the packing format
> (8/10/12/14/16/20/24 bit) are all dictated by the image data
> configuration. You can't have eg 640 pixel wide _SBGGR10_1X10 image
> data and 320 pixel wide _META_1X8_8 metadata. (it has to be 640 pixel
> wide _META_1X10_10)

That corresponds to my experience and is probably related to sensor
implementation rather than CSI-2 itself.

> 
> Which layer would be responsible for validating the configuration, and when?
> a) The driver validates in set_stream, and fails if mismatched. That
> potentially results in lots of duplication between drivers.
> b) set_pad_format on the image pad updates the formats and widths of
> the metadata pads automatically. Is there a way of notifying clients
> that the formats on potentially unrelated pads has changed, or is it
> something that just gets documented?
> c) something else.
> 
> Either is workable, but IMHO it needs to be specified as to which is
> the expected behaviour.

I left that out of the spec as in the end this depends on the hardware
constraints and how the driver author decided to map that to the API.

I expect sensor drivers to let the user configure the image format freely
(as it's been so far) and if there's anything to configure in the metadata
format that does not affect the image data format, then that
configurability is available on the metadata internal sink pad via
SUBDEV_S_FMT. So from the above options b appears a good fit. SUBDEV_S_FMT
does not have notification ability and probably shouldn't, instead the
order of configuration should be followed.

I think this would be good to be documented, perhaps with the first sensor
driver that uses it. Likely to be merged with this set later on.

-- 
Kind regards,

Sakari Ailus

Re: [RFC 4/7] media: mc: Check pad flag validity

[Sakari Ailus]

On Fri, Jun 02, 2023 at 12:58:21PM +0300, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.
> 
> On Sat, May 06, 2023 at 12:52:54AM +0300, Sakari Ailus wrote:
> > Check the validity of pad flags on entity init. Exactly one of the flags
> > must be set.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  drivers/media/mc/mc-entity.c | 12 +++++++++++-
> >  1 file changed, 11 insertions(+), 1 deletion(-)
> > 
> > diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> > index ed99160a2487..c1bc48c4d239 100644
> > --- a/drivers/media/mc/mc-entity.c
> > +++ b/drivers/media/mc/mc-entity.c
> > @@ -197,6 +197,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
> >  	struct media_device *mdev = entity->graph_obj.mdev;
> >  	struct media_pad *iter;
> >  	unsigned int i = 0;
> > +	int ret = 0;
> >  
> >  	if (num_pads >= MEDIA_ENTITY_MAX_PADS)
> >  		return -E2BIG;
> > @@ -210,6 +211,15 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
> >  	media_entity_for_each_pad(entity, iter) {
> >  		iter->entity = entity;
> >  		iter->index = i++;
> > +
> > +		if (hweight32(iter->flags & (MEDIA_PAD_FL_SINK |
> > +					     MEDIA_PAD_FL_SOURCE |
> > +					     MEDIA_PAD_FL_INTERNAL_SOURCE))
> > +		    != 1) {
> 
> I would add this check as patch 1/7 in this series, with FL_SINK an
> FL_SOURCE only, and then extend it in the patch where you introduce
> FL_INTERNAL_SOURCE.

Sounds reasonable.

> 
> > +			ret = -EINVAL;
> > +			break;
> > +		}
> > +
> >  		if (mdev)
> >  			media_gobj_create(mdev, MEDIA_GRAPH_PAD,
> >  					  &iter->graph_obj);
> > @@ -218,7 +228,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
> >  	if (mdev)
> >  		mutex_unlock(&mdev->graph_mutex);
> >  
> > -	return 0;
> > +	return ret;
> >  }
> >  EXPORT_SYMBOL_GPL(media_entity_pads_init);
> >  
> 
> -- 
> Regards,
> 
> Laurent Pinchart

-- 
Sakari Ailus

Re: [RFC 5/7] media: uapi: Add generic serial metadata mbus formats

[Sakari Ailus]

Hi Laurent,

On Fri, Jun 02, 2023 at 01:36:41PM +0300, Laurent Pinchart wrote:
> Hi Sakari,
> 
> Thank you for the patch.
> 
> On Sat, May 06, 2023 at 12:52:55AM +0300, Sakari Ailus wrote:
> > Add generic serial metadata mbus formats. These formats describe data
> > width and packing but not the content itself. The reason for specifying
> > such formats is that the formats as such are fairly device specific but
> > they are still handled by CSI-2 receiver drivers that should not be aware
> > of device specific formats. What makes generic metadata formats possible
> > is that these formats are parsed by software only, after capturing the
> > data to system memory.
> > 
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> >  .../media/v4l/subdev-formats.rst              | 257 ++++++++++++++++++
> >  include/uapi/linux/media-bus-format.h         |   9 +
> >  2 files changed, 266 insertions(+)
> > 
> > diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > index a3a35eeed708..1492fff58426 100644
> > --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> > @@ -8234,3 +8234,260 @@ The following table lists the existing metadata formats.
> >  	both sides of the link and the bus format is a fixed
> >  	metadata format that is not configurable from userspace.
> >  	Width and height will be set to 0 for this format.
> > +
> > +Generic Serial Metadata Formats
> > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > +
> > +Generic serial metadata formats are used on serial busses where the actual data
> 
> s/busses/buses/
> 
> > +content is more or less device specific but the data is transmitted and received
> > +by multiple devices that do not process the data in any way, simply writing
> > +it to system memory for processing in software at the end of the pipeline.
> > +
> > +The more specific variant describing the actual data is used on the internal
> > +source pad of the originating sub-device.
> 
> What do you mean by "more specific variant" here ? Please include an
> example in the documentation.
> 
> I'm not sure I like mentioning internal source pads here, are we
> guaranteed that metadata will always originate from an internal source
> pad ?

We'll have to, otherwise the specific format will be visible on the link,
and link validator on the sink end won't (and shouldn't) know about it (and
that would systematically lead to a failure to validate it).

> 
> > +
> > +"b" in an array cell signifies a byte of data, followed by the number of byte
> > +and finally the bit number in subscript. "p" indicates a padding bit.
> > +
> > +.. _media-bus-format-generic-meta:
> > +
> > +.. cssclass: longtable
> > +
> > +.. flat-table:: Generic Serial Metadata Formats
> > +    :header-rows:  2
> > +    :stub-columns: 0
> > +
> > +    * - Identifier
> > +      - Code
> > +      -
> > +      - :cspan:`23` Data organization
> > +    * -
> > +      -
> > +      - Bit
> > +      - 23
> > +      - 22
> > +      - 21
> > +      - 20
> > +      - 19
> > +      - 18
> > +      - 17
> > +      - 16
> > +      - 15
> > +      - 14
> > +      - 13
> > +      - 12
> > +      - 11
> > +      - 10
> > +      - 9
> > +      - 8
> > +      - 7
> > +      - 6
> > +      - 5
> > +      - 4
> > +      - 3
> > +      - 2
> > +      - 1
> > +      - 0
> > +    * .. _MEDIA-BUS-FMT-META-1X8-8:
> > +
> > +      - MEDIA_BUS_FMT_META_1X8_8
> > +      - 0x8001
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      - b0\ :sub:`7`
> > +      - b0\ :sub:`6`
> > +      - b0\ :sub:`5`
> > +      - b0\ :sub:`4`
> > +      - b0\ :sub:`3`
> > +      - b0\ :sub:`2`
> > +      - b0\ :sub:`1`
> > +      - b0\ :sub:`0`
> > +    * .. _MEDIA-BUS-FMT-META-1X8-10:
> > +
> > +      - MEDIA_BUS_FMT_META_1X8_10
> > +      - 0x8002
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      - b0\ :sub:`7`
> > +      - b0\ :sub:`6`
> > +      - b0\ :sub:`5`
> > +      - b0\ :sub:`4`
> > +      - b0\ :sub:`3`
> > +      - b0\ :sub:`2`
> > +      - b0\ :sub:`1`
> > +      - b0\ :sub:`0`
> > +      - p
> > +      - p
> > +    * .. _MEDIA-BUS-FMT-META-1X8-12:
> > +
> > +      - MEDIA_BUS_FMT_META_1X8_12
> > +      - 0x8003
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      - b0\ :sub:`7`
> > +      - b0\ :sub:`6`
> > +      - b0\ :sub:`5`
> > +      - b0\ :sub:`4`
> > +      - b0\ :sub:`3`
> > +      - b0\ :sub:`2`
> > +      - b0\ :sub:`1`
> > +      - b0\ :sub:`0`
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +    * .. _MEDIA-BUS-FMT-META-1X8-14:
> > +
> > +      - MEDIA_BUS_FMT_META_1X8_14
> > +      - 0x8004
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      - b0\ :sub:`7`
> > +      - b0\ :sub:`6`
> > +      - b0\ :sub:`5`
> > +      - b0\ :sub:`4`
> > +      - b0\ :sub:`3`
> > +      - b0\ :sub:`2`
> > +      - b0\ :sub:`1`
> > +      - b0\ :sub:`0`
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +    * .. _MEDIA-BUS-FMT-META-1X8-16:
> > +
> > +      - MEDIA_BUS_FMT_META_1X8_16
> > +      - 0x8005
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      - b0\ :sub:`7`
> > +      - b0\ :sub:`6`
> > +      - b0\ :sub:`5`
> > +      - b0\ :sub:`4`
> > +      - b0\ :sub:`3`
> > +      - b0\ :sub:`2`
> > +      - b0\ :sub:`1`
> > +      - b0\ :sub:`0`
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +    * .. _MEDIA-BUS-FMT-META-1X8-20:
> > +
> > +      - MEDIA_BUS_FMT_META_1X8_20
> > +      - 0x8007
> > +      -
> > +      -
> > +      -
> > +      -
> > +      -
> > +      - b0\ :sub:`7`
> > +      - b0\ :sub:`6`
> > +      - b0\ :sub:`5`
> > +      - b0\ :sub:`4`
> > +      - b0\ :sub:`3`
> > +      - b0\ :sub:`2`
> > +      - b0\ :sub:`1`
> > +      - b0\ :sub:`0`
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +    * .. _MEDIA-BUS-FMT-META-1X8-24:
> > +
> > +      - MEDIA_BUS_FMT_META_1X8_24
> > +      - 0x8009
> > +      -
> > +      - b0\ :sub:`7`
> > +      - b0\ :sub:`6`
> > +      - b0\ :sub:`5`
> > +      - b0\ :sub:`4`
> > +      - b0\ :sub:`3`
> > +      - b0\ :sub:`2`
> > +      - b0\ :sub:`1`
> > +      - b0\ :sub:`0`
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > +      - p
> > diff --git a/include/uapi/linux/media-bus-format.h b/include/uapi/linux/media-bus-format.h
> > index a03c543cb072..722463523bbd 100644
> > --- a/include/uapi/linux/media-bus-format.h
> > +++ b/include/uapi/linux/media-bus-format.h
> > @@ -173,4 +173,13 @@
> >   */
> >  #define MEDIA_BUS_FMT_METADATA_FIXED		0x7001
> >  
> > +/* Generic line based metadata formats for serial buses. Next is 0x800b. */
> > +#define MEDIA_BUS_FMT_META_1X8_8		0x8001
> > +#define MEDIA_BUS_FMT_META_1X8_10		0x8002
> > +#define MEDIA_BUS_FMT_META_1X8_12		0x8003
> > +#define MEDIA_BUS_FMT_META_1X8_14		0x8004
> > +#define MEDIA_BUS_FMT_META_1X8_16		0x8005
> > +#define MEDIA_BUS_FMT_META_1X8_20		0x8007
> > +#define MEDIA_BUS_FMT_META_1X8_24		0x8009
> 
> We've discussed this before privately, it's time to come to a conclusion
> :-)
> 
> My preference would be 
> 
> #define MEDIA_BUS_FMT_META_RAW8			0x8001
> #define MEDIA_BUS_FMT_META_RAW10		0x8002
> #define MEDIA_BUS_FMT_META_RAW12		0x8003
> #define MEDIA_BUS_FMT_META_RAW14		0x8004
> #define MEDIA_BUS_FMT_META_RAW16		0x8005
> #define MEDIA_BUS_FMT_META_RAW20		0x8007
> #define MEDIA_BUS_FMT_META_RAW24		0x8009
> 
> without defining the contents of the data (that is, no mention of
> padding bits).

I don't object but I'd like to know what others think.

If we ever need formats with padding (or other depths than 8) we'll need to
change the naming. This may still be unlikely.

-- 
Regards,

Sakari Ailus

Re: [RFC 0/7] Generic line based metadata support, internal pads

[Laurent Pinchart]

On Fri, Jun 09, 2023 at 02:41:22PM +0000, Sakari Ailus wrote:
> On Fri, Jun 09, 2023 at 02:59:20PM +0100, Dave Stevenson wrote:
> > On Fri, 5 May 2023 at 22:53, Sakari Ailus wrote:
> > >
> > > Hi folks,
> > >
> > > Here are a few patches to add support generic, line based metadata as well
> > > as internal source pads. While the amount of code is not very large, to
> > > the contrary it is quite small actually IMO, I presume what this is about
> > > and why it is being proposed requires some explaining.
> > >
> > > Metadata mbus codes and formats have existed for some time in V4L2. They
> > > however have been only used by drivers that produce the data itself and
> > > effectively this metadata has always been statistics of some sort (at
> > > least when it comes to ISPs). What is different here is that we intend to
> > > add support for metadata originating from camera sensors.
> > >
> > > Camera sensors produce different kinds of metadata, embedded data (usually
> > > register address--value pairs used to capture the frame, in a more or less
> > > sensor specific format), histograms (in a very sensor specific format),
> > > dark pixels etc. The number of these formats is probably going to be about
> > > as large as image data formats if not larger, as the image data formats
> > > are much better standardised but a smaller subset of them will be
> > > supported by V4L2, at least initially but possibly much more in the long
> > > run.
> > >
> > > Having this many device specific formats would be a major problem for all
> > > the other drivers along that pipeline (not to mention the users of those
> > > drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2
> > > receiver drivers that have DMA: the poor driver developer would not only
> > > need to know camera sensor specific formats but to choose the specific
> > > packing of that format suitable for the DMA used by the hardware. It is
> > > unlikely many of these would ever get tested while being present on the
> > > driver API. Also adding new sensors with new embedded data formats would
> > > involve updating all bridge and CSI-2 receiver drivers. I don't expect
> > > this to be a workable approach.
> > >
> > > Instead what I'm proposing is to use specific metadata formats on the
> > > sensor devices only, on internal pads (more about those soon) of the
> > > sensors, only visible in the UAPI, and then generic mbus formats along the
> > > pipeline and finally generic V4L2 metadata formats on the DMAs (specific
> > > to bit depth and packing). This would unsnarl the two, defining what data
> > > there is (specific mbus code) and how that is transported and packed
> > > (generic mbus codes and V4L2 formats).
> > >
> > > The user space would be required to "know" the path of that data from the
> > > sensor's internal pad to the V4L2 video node. I do not see this as these
> > > devices require at least some knowledge of the pipeline, i.e. hardware at
> > > hand. Separating what the data means and how it is packed may even be
> > > beneficial: it allows separating code that interprets the data (sensor
> > > internal mbus code) from the code that accesses it (packing).
> > >
> > > These formats are in practice line based, meaning that there may be
> > > padding at the end of the line, depending on the bus as well as the DMA.
> > > If non-line based formats are needed, it is always possible to set the
> > > "height" field to 1.
> > >
> > > The internal (source) pads are an alternative to source routes [1]. The
> > > source routes were not universally liked and I do have to say I like
> > > re-using existing interface concepts (pads and everything you can do with
> > > pads, including access format, selections etc.) wherever it makes sense,
> > > instead of duplicating functionality.
> > >
> > > Effectively internal source pads behave mostly just like sink pads, but
> > > they describe a flow of data that originates from a sub-device instead of
> > > arriving to a sub-device. The SUBDEV_S_ROUTING IOCTLs are used to enable
> > > and disable routes from internal source pads to sub-device's source pads.
> > > The subdev format IOCTLs are usable, too, so one can find which subdev
> > > format is available on given internal source pad.
> > 
> > Just to confirm my own understanding, the muxed streams API currently
> > copes with situations such as the FPD-Link devices combining a single
> > stream from each of a number of subdevs into one multiplexed stream
> > using virtual channels, but it doesn't handle multiple data types from
> > the same subdev. You're addressing that omission.
> 
> Correct.
> 
> > All seems reasonable.
> > 
> > One detail that I haven't seen covered and think ought to be clarified.
> > For raw sensors where you have image data and metadata, in my
> > experience the line width of that metadata and the packing format
> > (8/10/12/14/16/20/24 bit) are all dictated by the image data
> > configuration. You can't have eg 640 pixel wide _SBGGR10_1X10 image
> > data and 320 pixel wide _META_1X8_8 metadata. (it has to be 640 pixel
> > wide _META_1X10_10)
> 
> That corresponds to my experience and is probably related to sensor
> implementation rather than CSI-2 itself.

I agree, in theory the above could be implemented by a CSI-2 source, but
in practice that would be very uncommon.

> > Which layer would be responsible for validating the configuration, and when?
> > a) The driver validates in set_stream, and fails if mismatched. That
> > potentially results in lots of duplication between drivers.
> > b) set_pad_format on the image pad updates the formats and widths of
> > the metadata pads automatically. Is there a way of notifying clients
> > that the formats on potentially unrelated pads has changed, or is it
> > something that just gets documented?
> > c) something else.
> > 
> > Either is workable, but IMHO it needs to be specified as to which is
> > the expected behaviour.
> 
> I left that out of the spec as in the end this depends on the hardware
> constraints and how the driver author decided to map that to the API.
> 
> I expect sensor drivers to let the user configure the image format freely
> (as it's been so far) and if there's anything to configure in the metadata
> format that does not affect the image data format, then that
> configurability is available on the metadata internal sink pad via
> SUBDEV_S_FMT. So from the above options b appears a good fit. SUBDEV_S_FMT
> does not have notification ability and probably shouldn't, instead the
> order of configuration should be followed.
> 
> I think this would be good to be documented, perhaps with the first sensor
> driver that uses it. Likely to be merged with this set later on.

I agree with the above. The behaviour depends on the hardware
constraints, and is thus device-dependent, but we can standardize it for
classes of devices. For raw sensors that produce image and embedded data
streams, option B should be required in the documentation.

-- 
Regards,

Laurent Pinchart