[RFC v3 1/2] v4l2: add support for colorspace conversion for video capture

[RFC v3 1/2] v4l2: add support for colorspace conversion for video capture

[Dafna Hirschfeld]

From: Philipp Zabel <p.zabel@pengutronix.de>

For video capture it is the driver that reports the colorspace,
Y'CbCr/HSV encoding, quantization range and transfer function
used by the video, and there is no way to request something
different, even though many HDTV receivers have some sort of
colorspace conversion capabilities.

For output video this feature already exists since the application
specifies this information for the video format it will send out, and
the transmitter will enable any available CSC if a format conversion has
to be performed in order to match the capabilities of the sink.

For video capture we propose adding new pix_format flag:
V4L2_PIX_FMT_FLAG_HAS_CSC. The flag is set by the application,
the driver will interpret the ycbcr_enc/hsv_enc, and quantization fields
as the requested colorspace information and will attempt to
do the conversion it supports.

Drivers set the flags
V4L2_FMT_FLAG_CSC_YCBCR_ENC,
V4L2_FMT_FLAG_CSC_HSV_ENC,
V4L2_FMT_FLAG_CSC_HSV_QUANTIZATION,
in the flags field of the struct v4l2_fmtdesc during enumeration to
indicate that they support colorspace conversion for the respective field.
Currently the conversion of the fields colorspace, xfer_func is not
supported since there are no drivers that support it.

The same API is added for the subdevices. With the flag V4L2_MBUS_FRAMEFMT_HAS_CSC
set by the application in the VIDIOC_SUBDEV_S_FMT ioctl and the flags
V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC, V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION
set by the driver in the VIDIOC_SUBDEV_ENUM_MBUS_CODE ioctl.

For subdevices, new 'flags' fields were added to the structs
v4l2_subdev_mbus_code_enum, v4l2_mbus_framefmt which are borrowed from the 'reserved' field

Drivers do not have to actually look at the flags: if the flags are not
set, then the colorspace, ycbcr_enc and quantization fields are set to
the default values by the core, i.e. just pass on the received format
without conversion.

Signed-off-by: Hans Verkuil <Hans Verkuil@cisco.com>
Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
---
This is v3 of the RFC suggested originaly by Hans Verkuil:

 https://patchwork.linuxtv.org/patch/28847/

And then a v2 from Philipp Zabel:

 https://patchwork.kernel.org/project/linux-media/list/?series=15483

changes in v3:
I added the API to subdevices as well and added fixes according to
comments from Hans.
I also added a usecase for the new API for the rkisp1 driver.

changes in v2 (reported by Philipp Zabel):
 - convert to rst
 - split V4L2_PIX_FMT_FLAG_REQUEST_CSC into four separate flags for
   colorspace, ycbcr_enc/hsv_enc, quantization, and xfer_func
 - let driver set flags to indicate supported features

[1] https://patchwork.linuxtv.org/patch/28847/

 .../media/v4l/pixfmt-reserved.rst             |  6 +++
 .../media/v4l/pixfmt-v4l2-mplane.rst          | 16 ++-----
 .../userspace-api/media/v4l/pixfmt-v4l2.rst   | 36 +++++++++++++---
 .../media/v4l/subdev-formats.rst              | 42 +++++++++++++++++--
 .../media/v4l/vidioc-enum-fmt.rst             | 18 ++++++++
 .../v4l/vidioc-subdev-enum-mbus-code.rst      | 23 ++++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c          | 19 ++++++++-
 drivers/media/v4l2-core/v4l2-subdev.c         |  7 ++++
 include/uapi/linux/v4l2-mediabus.h            |  5 ++-
 include/uapi/linux/v4l2-subdev.h              |  5 ++-
 include/uapi/linux/videodev2.h                |  4 ++
 11 files changed, 158 insertions(+), 23 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst b/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst
index 59b9e7238f90..fa8dada69f8c 100644
--- a/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst
+++ b/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst
@@ -280,3 +280,9 @@ please make a proposal on the linux-media mailing list.
 	by RGBA values (128, 192, 255, 128), the same pixel described with
 	premultiplied colors would be described by RGBA values (64, 96,
 	128, 128)
+    * - ``V4L2_PIX_FMT_FLAG_HAS_CSC``
+      - 0x00000002
+      - Set by the application. It is only used for capture and is
+	ignored for output streams. If set, then request the driver to do
+	colorspace conversion from the received colorspace, only conversions
+	of Ycbcr encoding, HSV encoding and quantization are supported.
diff --git a/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst b/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst
index 444b4082684c..66f3365d7b72 100644
--- a/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst
+++ b/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst
@@ -105,29 +105,21 @@ describing all planes of that format.
     * - __u8
       - ``ycbcr_enc``
       - Y'CbCr encoding, from enum :c:type:`v4l2_ycbcr_encoding`.
-        This information supplements the ``colorspace`` and must be set by
-	the driver for capture streams and by the application for output
-	streams, see :ref:`colorspaces`.
+	See struct :c:type:`v4l2_pix_format`.
     * - __u8
       - ``hsv_enc``
       - HSV encoding, from enum :c:type:`v4l2_hsv_encoding`.
-        This information supplements the ``colorspace`` and must be set by
-	the driver for capture streams and by the application for output
-	streams, see :ref:`colorspaces`.
+	See struct :c:type:`v4l2_pix_format`.
     * - }
       -
     * - __u8
       - ``quantization``
       - Quantization range, from enum :c:type:`v4l2_quantization`.
-        This information supplements the ``colorspace`` and must be set by
-	the driver for capture streams and by the application for output
-	streams, see :ref:`colorspaces`.
+	See struct :c:type:`v4l2_pix_format`.
     * - __u8
       - ``xfer_func``
       - Transfer function, from enum :c:type:`v4l2_xfer_func`.
-        This information supplements the ``colorspace`` and must be set by
-	the driver for capture streams and by the application for output
-	streams, see :ref:`colorspaces`.
+	See struct :c:type:`v4l2_pix_format`.
     * - __u8
       - ``reserved[7]``
       - Reserved for future extensions. Should be zeroed by drivers and
diff --git a/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst b/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst
index 759420a872d6..ce57718cd66b 100644
--- a/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst
+++ b/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst
@@ -110,8 +110,8 @@ Single-planar format structure
       - ``colorspace``
       - Image colorspace, from enum :c:type:`v4l2_colorspace`.
         This information supplements the ``pixelformat`` and must be set
-	by the driver for capture streams and by the application for
-	output streams, see :ref:`colorspaces`.
+	by the driver for capture streams and by the application for output
+	streams, see :ref:`colorspace`.
     * - __u32
       - ``priv``
       - This field indicates whether the remaining fields of the
@@ -148,13 +148,31 @@ Single-planar format structure
       - Y'CbCr encoding, from enum :c:type:`v4l2_ycbcr_encoding`.
         This information supplements the ``colorspace`` and must be set by
 	the driver for capture streams and by the application for output
-	streams, see :ref:`colorspaces`.
+	streams, see :ref:`colorspaces`. If the application sets the
+	flag ``V4L2_PIX_FMT_FLAG_HAS_CSC`` then the application can set
+	this field for a capture stream to request a specific Y'CbCr encoding
+	for the captured image data. The driver will attempt to do the
+	conversion to the specified Y'CbCr encoding or return the encoding it
+	will use if it can't do the conversion. This field is ignored for
+	non-Y'CbCr pixelformats. The driver indicates that ycbcr_enc conversion
+	is supported by setting the flag V4L2_FMT_FLAG_CSC_YCBCR_ENC on the
+	on the corresponding struct :c:type:`v4l2_fmtdesc` during enumeration.
+	See :ref:`fmtdesc-flags`
     * - __u32
       - ``hsv_enc``
       - HSV encoding, from enum :c:type:`v4l2_hsv_encoding`.
         This information supplements the ``colorspace`` and must be set by
 	the driver for capture streams and by the application for output
-	streams, see :ref:`colorspaces`.
+	streams, see :ref:`colorspaces`. If the application sets the flag
+	``V4L2_PIX_FMT_FLAG_HAS_CSC`` then the application can set this
+	field for a capture stream to request a specific HSV encoding for the
+	captured image data. The driver will attempt to do the conversion to
+	the specified HSV encoding or return the encoding it will use if it
+	can't do the conversion. This field is ignored for non-HSV
+	pixelformats. The driver indicates that hsv_enc conversion
+	is supported by setting the flag V4L2_FMT_FLAG_CSC_HSV_ENC on the
+	on the corresponding struct :c:type:`v4l2_fmtdesc` during enumeration.
+	See :ref:`fmtdesc-flags`
     * - }
       -
     * - __u32
@@ -162,7 +180,15 @@ Single-planar format structure
       - Quantization range, from enum :c:type:`v4l2_quantization`.
         This information supplements the ``colorspace`` and must be set by
 	the driver for capture streams and by the application for output
-	streams, see :ref:`colorspaces`.
+	streams, see :ref:`colorspaces`. If the application sets the flag
+	``V4L2_PIX_FMT_FLAG_HAS_CSC`` then the application can set
+	this field for a capture stream to request a specific quantization
+	range for the captured image data. The driver will attempt to do the
+	conversion to the specified quantization range or return the
+	quantization it will use if it can't do the conversion.  The driver
+	indicates that quantization conversion is supported by setting the flag
+	V4L2_FMT_FLAG_CSC_QUANTIZATION on the on the corresponding struct
+	:c:type:`v4l2_fmtdesc` during enumeration. See :ref:`fmtdesc-flags`
     * - __u32
       - ``xfer_func``
       - Transfer function, from enum :c:type:`v4l2_xfer_func`.
diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
index 9a4d61b0d76f..f1d4ca29a3e8 100644
--- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
+++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
@@ -49,13 +49,33 @@ Media Bus Formats
       - Y'CbCr encoding, from enum :c:type:`v4l2_ycbcr_encoding`.
         This information supplements the ``colorspace`` and must be set by
 	the driver for capture streams and by the application for output
-	streams, see :ref:`colorspaces`.
+	streams, see :ref:`colorspaces`. If the application sets the
+	flag ``V4L2_MBUS_FRAMEFMT_HAS_CSC`` the the application can set
+	this field for a capture stream to request a specific Y'CbCr encoding
+	for the mbus data. The driver will attempt to do the
+	conversion to the specified Y'CbCr encoding or return the encoding it
+	will use if it can't do the conversion. This field is ignored for
+	non-Y'CbCr pixelformats. The driver indicates that ycbcr_enc conversion
+	is supported by setting the flag V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC on
+	the corresponding struct c:type:`v4l2_subdev_mbus_code_enum` during
+	enumeration. See :ref:`vidioc-subdev-enum-mbus-code.rst <v4l2-subdev-mbus-code-flags>`
+
     * - __u16
       - ``quantization``
       - Quantization range, from enum :c:type:`v4l2_quantization`.
         This information supplements the ``colorspace`` and must be set by
 	the driver for capture streams and by the application for output
-	streams, see :ref:`colorspaces`.
+	streams, see :ref:`colorspaces`. If the application sets the
+	flag ``V4L2_MBUS_FRAMEFMT_HAS_CSC`` the the application can set
+	this field for a capture stream to request a specific quantization
+	encoding for the mbus data. The driver will attempt to do the
+	conversion to the specified quantization or return the quantization it
+	will use if it can't do the conversion. The driver indicates that
+	quantization conversion is supported by setting the flag
+	V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION on the corresponding struct
+	c:type:`v4l2_subdev_mbus_code_enum` during enumeration.
+	See :ref:`vidioc-subdev-enum-mbus-code.rst <v4l2-subdev-mbus-code-flags>`
+
     * - __u16
       - ``xfer_func``
       - Transfer function, from enum :c:type:`v4l2_xfer_func`.
@@ -63,10 +83,26 @@ Media Bus Formats
 	the driver for capture streams and by the application for output
 	streams, see :ref:`colorspaces`.
     * - __u16
-      - ``reserved``\ [11]
+      - ``flags``
+      - flags see:  :ref:v4l2-mbus-framefmt-flags
+    * - __u16
+      - ``reserved``\ [10]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
+.. _v4l2-mbus-framefmt-flags:
+
+.. flat-table:: v4l2_mbus_framefmt Flags
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       3 1 4
+
+    * - ``V4L2_MBUS_FRAMEFMT_HAS_CSC``
+      - 0x0001
+      - Set by the application. It is only used for capture and is
+	ignored for output streams. If set, then request the subdevice to do
+	colorspace conversion from the received colorspace, only conversions
+	of Ycbcr encoding, and quantization are supported.
 
 
 .. _v4l2-mbus-pixelcode:
diff --git a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
index 7e3142e11d77..125f074543af 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
@@ -145,6 +145,24 @@ formats in preference order, where preferred formats are returned before
 	parameters are detected. This flag can only be used in combination
 	with the ``V4L2_FMT_FLAG_COMPRESSED`` flag, since this applies to
 	compressed formats only. It is also only applies to stateful codecs.
+    * - ``V4L2_FMT_FLAG_CSC_YCBCR_ENC``
+      - 0x0010
+      - The driver allows the application to try to change the default
+	ycbcr encoding. This flag is relevant only for capture devices.
+	The application can ask to configure the ycbcr_enc of the capture device
+	when calling the :c:func:`VIDIOC_S_FMT` ioctl.
+    * - ``V4L2_FMT_FLAG_CSC_HSV_ENC``
+      - 0x0010
+      - The driver allows the application to try to change the default
+	hsv encoding. This flag is relevant only for capture devices.
+	The application can ask to configure the hsv_enc of the capture device
+	when calling the :c:func:`VIDIOC_S_FMT` ioctl.
+    * - ``V4L2_FMT_FLAG_CSC_QUANTIZATION``
+      - 0x0020
+      - The driver allows the application to try to change the default
+	quantization. This flag is relevant only for capture devices.
+	The application can ask to configure the quantization of the capture device
+	when calling the :c:func:`VIDIOC_S_FMT` ioctl.
 
 
 Return Value
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
index 35b8607203a4..4ad87cb74f57 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
@@ -78,11 +78,34 @@ information about the try formats.
       - ``which``
       - Media bus format codes to be enumerated, from enum
 	:ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
+    * - __u32
+      - ``flags``
+      - see :ref:`v4l2-subdev-mbus-code-flags`
     * - __u32
       - ``reserved``\ [8]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
+.. _v4l2-subdev-mbus-code-flags:
+
+
+.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{7.7cm}|
+
+.. flat-table:: flags in struct v4l2_subdev_mbus_code_enum
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC
+      - 0x00000001
+      - The driver supports setting the ycbcr encoding by the application
+	when calling the VIDIOC_SUBDEV_S_FMT ioctl. see :ref:`v4l2-mbus-format`
+	on how to do this.
+    * - V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION
+      - 0x00000002
+      - The driver supports setting the quantization by the application
+	when calling the VIDIOC_SUBDEV_S_FMT ioctl. see :ref:`v4l2-mbus-format`
+	on how to do this.
 
 Return Value
 ============
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index b2ef8e60ea7d..3c7ffb6b15cb 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -1029,6 +1029,15 @@ static void v4l_sanitize_format(struct v4l2_format *fmt)
 		fmt->fmt.pix_mp.num_planes = min_t(u32, fmt->fmt.pix_mp.num_planes,
 					       VIDEO_MAX_PLANES);
 
+	if (fmt->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE) {
+		if (fmt->fmt.pix_mp.flags & V4L2_PIX_FMT_FLAG_HAS_CSC)
+			return;
+		fmt->fmt.pix_mp.colorspace = V4L2_COLORSPACE_DEFAULT;
+		fmt->fmt.pix_mp.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
+		fmt->fmt.pix_mp.quantization = V4L2_QUANTIZATION_DEFAULT;
+		fmt->fmt.pix_mp.xfer_func = V4L2_XFER_FUNC_DEFAULT;
+	}
+
 	/*
 	 * The v4l2_pix_format structure has been extended with fields that were
 	 * not previously required to be set to zero by applications. The priv
@@ -1043,8 +1052,16 @@ static void v4l_sanitize_format(struct v4l2_format *fmt)
 	    fmt->type != V4L2_BUF_TYPE_VIDEO_OUTPUT)
 		return;
 
-	if (fmt->fmt.pix.priv == V4L2_PIX_FMT_PRIV_MAGIC)
+	if (fmt->fmt.pix.priv == V4L2_PIX_FMT_PRIV_MAGIC) {
+		if (fmt->type != V4L2_BUF_TYPE_VIDEO_CAPTURE ||
+		    fmt->fmt.pix.flags & V4L2_PIX_FMT_FLAG_HAS_CSC)
+			return;
+		fmt->fmt.pix.colorspace = V4L2_COLORSPACE_DEFAULT;
+		fmt->fmt.pix.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
+		fmt->fmt.pix.quantization = V4L2_QUANTIZATION_DEFAULT;
+		fmt->fmt.pix.xfer_func = V4L2_XFER_FUNC_DEFAULT;
 		return;
+	}
 
 	fmt->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
 
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index a376b351135f..51e009936aad 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -477,6 +477,13 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 	case VIDIOC_SUBDEV_S_FMT: {
 		struct v4l2_subdev_format *format = arg;
 
+		if (!(format->format.flags & V4L2_MBUS_FRAMEFMT_HAS_CSC)) {
+			format->format.colorspace = V4L2_COLORSPACE_DEFAULT;
+			format->format.xfer_func = V4L2_XFER_FUNC_DEFAULT;
+			format->format.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
+			format->format.quantization = V4L2_QUANTIZATION_DEFAULT;
+		}
+
 		memset(format->reserved, 0, sizeof(format->reserved));
 		memset(format->format.reserved, 0, sizeof(format->format.reserved));
 		return v4l2_subdev_call(sd, pad, set_fmt, subdev_fh->pad, format);
diff --git a/include/uapi/linux/v4l2-mediabus.h b/include/uapi/linux/v4l2-mediabus.h
index 123a231001a8..89ff0ad6a631 100644
--- a/include/uapi/linux/v4l2-mediabus.h
+++ b/include/uapi/linux/v4l2-mediabus.h
@@ -16,6 +16,8 @@
 #include <linux/types.h>
 #include <linux/videodev2.h>
 
+#define V4L2_MBUS_FRAMEFMT_HAS_CSC	0x0001
+
 /**
  * struct v4l2_mbus_framefmt - frame format on the media bus
  * @width:	image width
@@ -36,7 +38,8 @@ struct v4l2_mbus_framefmt {
 	__u16			ycbcr_enc;
 	__u16			quantization;
 	__u16			xfer_func;
-	__u16			reserved[11];
+	__u16			flags;
+	__u16			reserved[10];
 };
 
 #ifndef __KERNEL__
diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
index 03970ce30741..4410b26a7158 100644
--- a/include/uapi/linux/v4l2-subdev.h
+++ b/include/uapi/linux/v4l2-subdev.h
@@ -65,6 +65,8 @@ struct v4l2_subdev_crop {
 	__u32 reserved[8];
 };
 
+#define V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC	0x00000001
+#define V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION	0x00000002
 /**
  * struct v4l2_subdev_mbus_code_enum - Media bus format enumeration
  * @pad: pad number, as reported by the media API
@@ -77,7 +79,8 @@ struct v4l2_subdev_mbus_code_enum {
 	__u32 index;
 	__u32 code;
 	__u32 which;
-	__u32 reserved[8];
+	__u32 flags;
+	__u32 reserved[7];
 };
 
 /**
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 9817b7e2c968..adc9dd1080b8 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -772,6 +772,7 @@ struct v4l2_pix_format {
 
 /* Flags */
 #define V4L2_PIX_FMT_FLAG_PREMUL_ALPHA	0x00000001
+#define V4L2_PIX_FMT_FLAG_HAS_CSC	0x00000002
 
 /*
  *	F O R M A T   E N U M E R A T I O N
@@ -789,6 +790,9 @@ struct v4l2_fmtdesc {
 #define V4L2_FMT_FLAG_EMULATED			0x0002
 #define V4L2_FMT_FLAG_CONTINUOUS_BYTESTREAM	0x0004
 #define V4L2_FMT_FLAG_DYN_RESOLUTION		0x0008
+#define V4L2_FMT_FLAG_CSC_YCBCR_ENC		0x0010
+#define V4L2_FMT_FLAG_CSC_HSV_ENC		0x0010
+#define V4L2_FMT_FLAG_CSC_QUANTIZATION		0x0020
 
 	/* Frame Size and frame rate enumeration */
 /*
-- 
2.17.1

[RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad

[Dafna Hirschfeld]

The isp entity has a hardware support to force full range quantization
for YUV formats. Use the new API to indicate userspace that
quantization conversion is supported by adding the flag
V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION during media code enumeration.
Then uppon s_fmt on the video source pad, we assign the
quantization from userspace for YUV formats.
Also in the capture and resizer entities we retrieve the colorspace
from the isp entity.

Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
---
 drivers/staging/media/rkisp1/rkisp1-capture.c | 28 ++++++-------
 drivers/staging/media/rkisp1/rkisp1-common.h  |  2 +
 drivers/staging/media/rkisp1/rkisp1-isp.c     | 39 +++++++++++++++++--
 drivers/staging/media/rkisp1/rkisp1-resizer.c | 13 +++++++
 4 files changed, 65 insertions(+), 17 deletions(-)

diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c
index fbf62399fe3d..aca0f93bc772 100644
--- a/drivers/staging/media/rkisp1/rkisp1-capture.c
+++ b/drivers/staging/media/rkisp1/rkisp1-capture.c
@@ -1066,14 +1066,13 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
 			   const struct v4l2_format_info **fmt_info)
 {
 	const struct rkisp1_capture_config *config = cap->config;
-	struct rkisp1_capture *other_cap =
-			&cap->rkisp1->capture_devs[cap->id ^ 1];
 	const struct rkisp1_capture_fmt_cfg *fmt;
 	const struct v4l2_format_info *info;
 	const unsigned int max_widths[] = { RKISP1_RSZ_MP_SRC_MAX_WIDTH,
 					    RKISP1_RSZ_SP_SRC_MAX_WIDTH };
 	const unsigned int max_heights[] = { RKISP1_RSZ_MP_SRC_MAX_HEIGHT,
 					     RKISP1_RSZ_SP_SRC_MAX_HEIGHT};
+	struct v4l2_subdev_format isp_sd_fmt;
 
 	fmt = rkisp1_find_fmt_cfg(cap, pixm->pixelformat);
 	if (!fmt) {
@@ -1081,24 +1080,27 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
 		pixm->pixelformat = fmt->fourcc;
 	}
 
+	rkisp1_get_isp_src_fmt(cap->rkisp1, &isp_sd_fmt);
+	pixm->field = isp_sd_fmt.format.field;
+	pixm->colorspace = isp_sd_fmt.format.colorspace;
+	pixm->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
+	pixm->xfer_func = isp_sd_fmt.format.xfer_func;
+
+	/*
+	 * isp has a feature to set full range quantization for yuv formats.
+	 * so we need to get the format from the isp.
+	 */
+	pixm->quantization = isp_sd_fmt.format.quantization;
+	if (!v4l2_is_format_yuv(cap->pix.info))
+		pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
+
 	pixm->width = clamp_t(u32, pixm->width,
 			      RKISP1_RSZ_SRC_MIN_WIDTH, max_widths[cap->id]);
 	pixm->height = clamp_t(u32, pixm->height,
 			       RKISP1_RSZ_SRC_MIN_HEIGHT, max_heights[cap->id]);
 
-	pixm->field = V4L2_FIELD_NONE;
-	pixm->colorspace = V4L2_COLORSPACE_DEFAULT;
-	pixm->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
-
 	info = rkisp1_fill_pixfmt(pixm, cap->id);
 
-	/* can not change quantization when stream-on */
-	if (other_cap->is_streaming)
-		pixm->quantization = other_cap->pix.fmt.quantization;
-	/* output full range by default, take effect in params */
-	else if (!pixm->quantization ||
-		 pixm->quantization > V4L2_QUANTIZATION_LIM_RANGE)
-		pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
 
 	if (fmt_cfg)
 		*fmt_cfg = fmt;
diff --git a/drivers/staging/media/rkisp1/rkisp1-common.h b/drivers/staging/media/rkisp1/rkisp1-common.h
index 2d7b7e078636..7a5576fa14c9 100644
--- a/drivers/staging/media/rkisp1/rkisp1-common.h
+++ b/drivers/staging/media/rkisp1/rkisp1-common.h
@@ -300,6 +300,8 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
 			struct v4l2_device *v4l2_dev);
 void rkisp1_isp_unregister(struct rkisp1_device *rkisp1);
 
+int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
+			   struct v4l2_subdev_format *sd_fmt);
 const struct rkisp1_isp_mbus_info *rkisp1_isp_mbus_info_get(u32 mbus_code);
 
 irqreturn_t rkisp1_isp_isr(struct rkisp1_device *rkisp1);
diff --git a/drivers/staging/media/rkisp1/rkisp1-isp.c b/drivers/staging/media/rkisp1/rkisp1-isp.c
index dee8e96f3900..6fdf5ed0b6b1 100644
--- a/drivers/staging/media/rkisp1/rkisp1-isp.c
+++ b/drivers/staging/media/rkisp1/rkisp1-isp.c
@@ -613,6 +613,10 @@ static int rkisp1_isp_enum_mbus_code(struct v4l2_subdev *sd,
 
 		if (code->index == pos - 1) {
 			code->code = fmt->mbus_code;
+			if (fmt->pixel_enc == V4L2_PIXEL_ENC_YUV &&
+			    dir == RKISP1_DIR_SRC)
+				code->flags =
+					V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION;
 			return 0;
 		}
 	}
@@ -639,12 +643,21 @@ static int rkisp1_isp_init_config(struct v4l2_subdev *sd,
 	sink_crop->height = RKISP1_DEFAULT_HEIGHT;
 	sink_crop->left = 0;
 	sink_crop->top = 0;
+	sink_fmt->colorspace = V4L2_COLORSPACE_SRGB;
+	sink_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(sink_fmt->colorspace);
+	sink_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(sink_fmt->colorspace);
+	sink_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
+
 
 	src_fmt = v4l2_subdev_get_try_format(sd, cfg,
 					     RKISP1_ISP_PAD_SOURCE_VIDEO);
 	*src_fmt = *sink_fmt;
 	src_fmt->code = RKISP1_DEF_SRC_PAD_FMT;
-	src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
+	src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
+	src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
+	src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
+	src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
+
 
 	src_crop = v4l2_subdev_get_try_crop(sd, cfg,
 					    RKISP1_ISP_PAD_SOURCE_VIDEO);
@@ -687,10 +700,17 @@ static void rkisp1_isp_set_src_fmt(struct rkisp1_isp *isp,
 		isp->src_fmt = mbus_info;
 	src_fmt->width  = src_crop->width;
 	src_fmt->height = src_crop->height;
-	src_fmt->quantization = format->quantization;
-	/* full range by default */
-	if (!src_fmt->quantization)
+
+	src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
+	src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
+	src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
+
+	if (mbus_info->pixel_enc == V4L2_PIXEL_ENC_BAYER)
 		src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
+	else if (format->quantization == V4L2_QUANTIZATION_DEFAULT)
+		src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
+	else
+		src_fmt->quantization = format->quantization;
 
 	*format = *src_fmt;
 }
@@ -1068,6 +1088,17 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
 	return ret;
 }
 
+int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
+			   struct v4l2_subdev_format *sd_fmt)
+{
+	struct rkisp1_isp *isp = &rkisp1->isp;
+
+	sd_fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
+	sd_fmt->pad = RKISP1_ISP_PAD_SOURCE_VIDEO;
+
+	return v4l2_subdev_call(&isp->sd, pad, get_fmt, NULL, sd_fmt);
+}
+
 void rkisp1_isp_unregister(struct rkisp1_device *rkisp1)
 {
 	struct v4l2_subdev *sd = &rkisp1->isp.sd;
diff --git a/drivers/staging/media/rkisp1/rkisp1-resizer.c b/drivers/staging/media/rkisp1/rkisp1-resizer.c
index 7b6b7ddd4169..8705b133de68 100644
--- a/drivers/staging/media/rkisp1/rkisp1-resizer.c
+++ b/drivers/staging/media/rkisp1/rkisp1-resizer.c
@@ -525,6 +525,7 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
 	const struct rkisp1_isp_mbus_info *mbus_info;
 	struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
 	struct v4l2_rect *sink_crop;
+	struct v4l2_subdev_format isp_sd_fmt;
 
 	sink_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SINK, which);
 	src_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SRC, which);
@@ -539,8 +540,20 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
 	if (which == V4L2_SUBDEV_FORMAT_ACTIVE)
 		rsz->pixel_enc = mbus_info->pixel_enc;
 
+	rkisp1_get_isp_src_fmt(rsz->rkisp1, &isp_sd_fmt);
+
+	sink_fmt->field = isp_sd_fmt.format.field;
+	sink_fmt->colorspace = isp_sd_fmt.format.colorspace;
+	sink_fmt->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
+	sink_fmt->xfer_func = isp_sd_fmt.format.xfer_func;
+
 	/* Propagete to source pad */
 	src_fmt->code = sink_fmt->code;
+	src_fmt->field = sink_fmt->field;
+	src_fmt->colorspace = sink_fmt->colorspace;
+	src_fmt->ycbcr_enc = sink_fmt->ycbcr_enc;
+	src_fmt->xfer_func = sink_fmt->xfer_func;
+	src_fmt->quantization = sink_fmt->quantization;
 
 	sink_fmt->width = clamp_t(u32, format->width,
 				  rsz->config->min_rsz_width,
-- 
2.17.1

Re: [RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad

[Tomasz Figa]

On Thu, Apr 16, 2020 at 04:56:05PM +0200, Dafna Hirschfeld wrote:
> The isp entity has a hardware support to force full range quantization
> for YUV formats. Use the new API to indicate userspace that
> quantization conversion is supported by adding the flag
> V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION during media code enumeration.
> Then uppon s_fmt on the video source pad, we assign the
> quantization from userspace for YUV formats.
> Also in the capture and resizer entities we retrieve the colorspace
> from the isp entity.
> 
> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
> ---
>  drivers/staging/media/rkisp1/rkisp1-capture.c | 28 ++++++-------
>  drivers/staging/media/rkisp1/rkisp1-common.h  |  2 +
>  drivers/staging/media/rkisp1/rkisp1-isp.c     | 39 +++++++++++++++++--
>  drivers/staging/media/rkisp1/rkisp1-resizer.c | 13 +++++++
>  4 files changed, 65 insertions(+), 17 deletions(-)
> 
> diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c
> index fbf62399fe3d..aca0f93bc772 100644
> --- a/drivers/staging/media/rkisp1/rkisp1-capture.c
> +++ b/drivers/staging/media/rkisp1/rkisp1-capture.c
> @@ -1066,14 +1066,13 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
>  			   const struct v4l2_format_info **fmt_info)
>  {
>  	const struct rkisp1_capture_config *config = cap->config;
> -	struct rkisp1_capture *other_cap =
> -			&cap->rkisp1->capture_devs[cap->id ^ 1];
>  	const struct rkisp1_capture_fmt_cfg *fmt;
>  	const struct v4l2_format_info *info;
>  	const unsigned int max_widths[] = { RKISP1_RSZ_MP_SRC_MAX_WIDTH,
>  					    RKISP1_RSZ_SP_SRC_MAX_WIDTH };
>  	const unsigned int max_heights[] = { RKISP1_RSZ_MP_SRC_MAX_HEIGHT,
>  					     RKISP1_RSZ_SP_SRC_MAX_HEIGHT};
> +	struct v4l2_subdev_format isp_sd_fmt;
>  
>  	fmt = rkisp1_find_fmt_cfg(cap, pixm->pixelformat);
>  	if (!fmt) {
> @@ -1081,24 +1080,27 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
>  		pixm->pixelformat = fmt->fourcc;
>  	}
>  
> +	rkisp1_get_isp_src_fmt(cap->rkisp1, &isp_sd_fmt);
> +	pixm->field = isp_sd_fmt.format.field;
> +	pixm->colorspace = isp_sd_fmt.format.colorspace;
> +	pixm->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
> +	pixm->xfer_func = isp_sd_fmt.format.xfer_func;
> +
> +	/*
> +	 * isp has a feature to set full range quantization for yuv formats.

How about "select between limited and full range for YUV formats"?

> +	 * so we need to get the format from the isp.
> +	 */
> +	pixm->quantization = isp_sd_fmt.format.quantization;
> +	if (!v4l2_is_format_yuv(cap->pix.info))
> +		pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> +
>  	pixm->width = clamp_t(u32, pixm->width,
>  			      RKISP1_RSZ_SRC_MIN_WIDTH, max_widths[cap->id]);
>  	pixm->height = clamp_t(u32, pixm->height,
>  			       RKISP1_RSZ_SRC_MIN_HEIGHT, max_heights[cap->id]);
>  
> -	pixm->field = V4L2_FIELD_NONE;
> -	pixm->colorspace = V4L2_COLORSPACE_DEFAULT;
> -	pixm->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> -
>  	info = rkisp1_fill_pixfmt(pixm, cap->id);
>  
> -	/* can not change quantization when stream-on */
> -	if (other_cap->is_streaming)
> -		pixm->quantization = other_cap->pix.fmt.quantization;
> -	/* output full range by default, take effect in params */
> -	else if (!pixm->quantization ||
> -		 pixm->quantization > V4L2_QUANTIZATION_LIM_RANGE)
> -		pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>  
>  	if (fmt_cfg)
>  		*fmt_cfg = fmt;
> diff --git a/drivers/staging/media/rkisp1/rkisp1-common.h b/drivers/staging/media/rkisp1/rkisp1-common.h
> index 2d7b7e078636..7a5576fa14c9 100644
> --- a/drivers/staging/media/rkisp1/rkisp1-common.h
> +++ b/drivers/staging/media/rkisp1/rkisp1-common.h
> @@ -300,6 +300,8 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
>  			struct v4l2_device *v4l2_dev);
>  void rkisp1_isp_unregister(struct rkisp1_device *rkisp1);
>  
> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
> +			   struct v4l2_subdev_format *sd_fmt);
>  const struct rkisp1_isp_mbus_info *rkisp1_isp_mbus_info_get(u32 mbus_code);
>  
>  irqreturn_t rkisp1_isp_isr(struct rkisp1_device *rkisp1);
> diff --git a/drivers/staging/media/rkisp1/rkisp1-isp.c b/drivers/staging/media/rkisp1/rkisp1-isp.c
> index dee8e96f3900..6fdf5ed0b6b1 100644
> --- a/drivers/staging/media/rkisp1/rkisp1-isp.c
> +++ b/drivers/staging/media/rkisp1/rkisp1-isp.c
> @@ -613,6 +613,10 @@ static int rkisp1_isp_enum_mbus_code(struct v4l2_subdev *sd,
>  
>  		if (code->index == pos - 1) {
>  			code->code = fmt->mbus_code;
> +			if (fmt->pixel_enc == V4L2_PIXEL_ENC_YUV &&
> +			    dir == RKISP1_DIR_SRC)
> +				code->flags =
> +					V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION;
>  			return 0;
>  		}
>  	}
> @@ -639,12 +643,21 @@ static int rkisp1_isp_init_config(struct v4l2_subdev *sd,
>  	sink_crop->height = RKISP1_DEFAULT_HEIGHT;
>  	sink_crop->left = 0;
>  	sink_crop->top = 0;
> +	sink_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> +	sink_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(sink_fmt->colorspace);
> +	sink_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(sink_fmt->colorspace);
> +	sink_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> +
>  
>  	src_fmt = v4l2_subdev_get_try_format(sd, cfg,
>  					     RKISP1_ISP_PAD_SOURCE_VIDEO);
>  	*src_fmt = *sink_fmt;
>  	src_fmt->code = RKISP1_DEF_SRC_PAD_FMT;
> -	src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> +	src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> +	src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
> +	src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
> +	src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
> +
>  
>  	src_crop = v4l2_subdev_get_try_crop(sd, cfg,
>  					    RKISP1_ISP_PAD_SOURCE_VIDEO);
> @@ -687,10 +700,17 @@ static void rkisp1_isp_set_src_fmt(struct rkisp1_isp *isp,
>  		isp->src_fmt = mbus_info;
>  	src_fmt->width  = src_crop->width;
>  	src_fmt->height = src_crop->height;
> -	src_fmt->quantization = format->quantization;
> -	/* full range by default */
> -	if (!src_fmt->quantization)
> +
> +	src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> +	src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
> +	src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
> +
> +	if (mbus_info->pixel_enc == V4L2_PIXEL_ENC_BAYER)
>  		src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> +	else if (format->quantization == V4L2_QUANTIZATION_DEFAULT)
> +		src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
> +	else
> +		src_fmt->quantization = format->quantization;
>  
>  	*format = *src_fmt;
>  }
> @@ -1068,6 +1088,17 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
>  	return ret;
>  }
>  
> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
> +			   struct v4l2_subdev_format *sd_fmt)
> +{
> +	struct rkisp1_isp *isp = &rkisp1->isp;
> +
> +	sd_fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
> +	sd_fmt->pad = RKISP1_ISP_PAD_SOURCE_VIDEO;
> +
> +	return v4l2_subdev_call(&isp->sd, pad, get_fmt, NULL, sd_fmt);

Do we need to get through the external API to access data that is
driver-internal anyway?

> +}
> +
>  void rkisp1_isp_unregister(struct rkisp1_device *rkisp1)
>  {
>  	struct v4l2_subdev *sd = &rkisp1->isp.sd;
> diff --git a/drivers/staging/media/rkisp1/rkisp1-resizer.c b/drivers/staging/media/rkisp1/rkisp1-resizer.c
> index 7b6b7ddd4169..8705b133de68 100644
> --- a/drivers/staging/media/rkisp1/rkisp1-resizer.c
> +++ b/drivers/staging/media/rkisp1/rkisp1-resizer.c
> @@ -525,6 +525,7 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
>  	const struct rkisp1_isp_mbus_info *mbus_info;
>  	struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
>  	struct v4l2_rect *sink_crop;
> +	struct v4l2_subdev_format isp_sd_fmt;
>  
>  	sink_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SINK, which);
>  	src_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SRC, which);
> @@ -539,8 +540,20 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
>  	if (which == V4L2_SUBDEV_FORMAT_ACTIVE)
>  		rsz->pixel_enc = mbus_info->pixel_enc;
>  
> +	rkisp1_get_isp_src_fmt(rsz->rkisp1, &isp_sd_fmt);
> +

Is this necessary? My understanding was that in the subdev model, it was
the userspace responsibility to propagate any configuration changes through
the graph.

Also, doing this only here wouldn't fully maintain the
consistency of the state. For example, if one sets the ISP subdev format
first, then the resizer subdev and then the ISP subdev again, wouldn't the
resizer subdev end up with a wrong format?

Best regards,
Tomasz

> +	sink_fmt->field = isp_sd_fmt.format.field;
> +	sink_fmt->colorspace = isp_sd_fmt.format.colorspace;
> +	sink_fmt->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
> +	sink_fmt->xfer_func = isp_sd_fmt.format.xfer_func;
> +
>  	/* Propagete to source pad */
>  	src_fmt->code = sink_fmt->code;
> +	src_fmt->field = sink_fmt->field;
> +	src_fmt->colorspace = sink_fmt->colorspace;
> +	src_fmt->ycbcr_enc = sink_fmt->ycbcr_enc;
> +	src_fmt->xfer_func = sink_fmt->xfer_func;
> +	src_fmt->quantization = sink_fmt->quantization;
>  
>  	sink_fmt->width = clamp_t(u32, format->width,
>  				  rsz->config->min_rsz_width,
> -- 
> 2.17.1
> 

Re: [RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad

[Dafna Hirschfeld]

Hi

On 04.06.20 19:54, Tomasz Figa wrote:
> On Thu, Apr 16, 2020 at 04:56:05PM +0200, Dafna Hirschfeld wrote:
>> The isp entity has a hardware support to force full range quantization
>> for YUV formats. Use the new API to indicate userspace that
>> quantization conversion is supported by adding the flag
>> V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION during media code enumeration.
>> Then uppon s_fmt on the video source pad, we assign the
>> quantization from userspace for YUV formats.
>> Also in the capture and resizer entities we retrieve the colorspace
>> from the isp entity.
>>
>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
>> ---
>>   drivers/staging/media/rkisp1/rkisp1-capture.c | 28 ++++++-------
>>   drivers/staging/media/rkisp1/rkisp1-common.h  |  2 +
>>   drivers/staging/media/rkisp1/rkisp1-isp.c     | 39 +++++++++++++++++--
>>   drivers/staging/media/rkisp1/rkisp1-resizer.c | 13 +++++++
>>   4 files changed, 65 insertions(+), 17 deletions(-)
>>
>> diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c
>> index fbf62399fe3d..aca0f93bc772 100644
>> --- a/drivers/staging/media/rkisp1/rkisp1-capture.c
>> +++ b/drivers/staging/media/rkisp1/rkisp1-capture.c
>> @@ -1066,14 +1066,13 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
>>   			   const struct v4l2_format_info **fmt_info)
>>   {
>>   	const struct rkisp1_capture_config *config = cap->config;
>> -	struct rkisp1_capture *other_cap =
>> -			&cap->rkisp1->capture_devs[cap->id ^ 1];
>>   	const struct rkisp1_capture_fmt_cfg *fmt;
>>   	const struct v4l2_format_info *info;
>>   	const unsigned int max_widths[] = { RKISP1_RSZ_MP_SRC_MAX_WIDTH,
>>   					    RKISP1_RSZ_SP_SRC_MAX_WIDTH };
>>   	const unsigned int max_heights[] = { RKISP1_RSZ_MP_SRC_MAX_HEIGHT,
>>   					     RKISP1_RSZ_SP_SRC_MAX_HEIGHT};
>> +	struct v4l2_subdev_format isp_sd_fmt;
>>   
>>   	fmt = rkisp1_find_fmt_cfg(cap, pixm->pixelformat);
>>   	if (!fmt) {
>> @@ -1081,24 +1080,27 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
>>   		pixm->pixelformat = fmt->fourcc;
>>   	}
>>   
>> +	rkisp1_get_isp_src_fmt(cap->rkisp1, &isp_sd_fmt);
>> +	pixm->field = isp_sd_fmt.format.field;
>> +	pixm->colorspace = isp_sd_fmt.format.colorspace;
>> +	pixm->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
>> +	pixm->xfer_func = isp_sd_fmt.format.xfer_func;
>> +
>> +	/*
>> +	 * isp has a feature to set full range quantization for yuv formats.
> 
> How about "select between limited and full range for YUV formats"?
> 
>> +	 * so we need to get the format from the isp.
>> +	 */
>> +	pixm->quantization = isp_sd_fmt.format.quantization;
>> +	if (!v4l2_is_format_yuv(cap->pix.info))
>> +		pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>> +
>>   	pixm->width = clamp_t(u32, pixm->width,
>>   			      RKISP1_RSZ_SRC_MIN_WIDTH, max_widths[cap->id]);
>>   	pixm->height = clamp_t(u32, pixm->height,
>>   			       RKISP1_RSZ_SRC_MIN_HEIGHT, max_heights[cap->id]);
>>   
>> -	pixm->field = V4L2_FIELD_NONE;
>> -	pixm->colorspace = V4L2_COLORSPACE_DEFAULT;
>> -	pixm->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>> -
>>   	info = rkisp1_fill_pixfmt(pixm, cap->id);
>>   
>> -	/* can not change quantization when stream-on */
>> -	if (other_cap->is_streaming)
>> -		pixm->quantization = other_cap->pix.fmt.quantization;
>> -	/* output full range by default, take effect in params */
>> -	else if (!pixm->quantization ||
>> -		 pixm->quantization > V4L2_QUANTIZATION_LIM_RANGE)
>> -		pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>   
>>   	if (fmt_cfg)
>>   		*fmt_cfg = fmt;
>> diff --git a/drivers/staging/media/rkisp1/rkisp1-common.h b/drivers/staging/media/rkisp1/rkisp1-common.h
>> index 2d7b7e078636..7a5576fa14c9 100644
>> --- a/drivers/staging/media/rkisp1/rkisp1-common.h
>> +++ b/drivers/staging/media/rkisp1/rkisp1-common.h
>> @@ -300,6 +300,8 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
>>   			struct v4l2_device *v4l2_dev);
>>   void rkisp1_isp_unregister(struct rkisp1_device *rkisp1);
>>   
>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
>> +			   struct v4l2_subdev_format *sd_fmt);
>>   const struct rkisp1_isp_mbus_info *rkisp1_isp_mbus_info_get(u32 mbus_code);
>>   
>>   irqreturn_t rkisp1_isp_isr(struct rkisp1_device *rkisp1);
>> diff --git a/drivers/staging/media/rkisp1/rkisp1-isp.c b/drivers/staging/media/rkisp1/rkisp1-isp.c
>> index dee8e96f3900..6fdf5ed0b6b1 100644
>> --- a/drivers/staging/media/rkisp1/rkisp1-isp.c
>> +++ b/drivers/staging/media/rkisp1/rkisp1-isp.c
>> @@ -613,6 +613,10 @@ static int rkisp1_isp_enum_mbus_code(struct v4l2_subdev *sd,
>>   
>>   		if (code->index == pos - 1) {
>>   			code->code = fmt->mbus_code;
>> +			if (fmt->pixel_enc == V4L2_PIXEL_ENC_YUV &&
>> +			    dir == RKISP1_DIR_SRC)
>> +				code->flags =
>> +					V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION;
>>   			return 0;
>>   		}
>>   	}
>> @@ -639,12 +643,21 @@ static int rkisp1_isp_init_config(struct v4l2_subdev *sd,
>>   	sink_crop->height = RKISP1_DEFAULT_HEIGHT;
>>   	sink_crop->left = 0;
>>   	sink_crop->top = 0;
>> +	sink_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>> +	sink_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(sink_fmt->colorspace);
>> +	sink_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(sink_fmt->colorspace);
>> +	sink_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>> +
>>   
>>   	src_fmt = v4l2_subdev_get_try_format(sd, cfg,
>>   					     RKISP1_ISP_PAD_SOURCE_VIDEO);
>>   	*src_fmt = *sink_fmt;
>>   	src_fmt->code = RKISP1_DEF_SRC_PAD_FMT;
>> -	src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>> +	src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>> +	src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
>> +	src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
>> +	src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
>> +
>>   
>>   	src_crop = v4l2_subdev_get_try_crop(sd, cfg,
>>   					    RKISP1_ISP_PAD_SOURCE_VIDEO);
>> @@ -687,10 +700,17 @@ static void rkisp1_isp_set_src_fmt(struct rkisp1_isp *isp,
>>   		isp->src_fmt = mbus_info;
>>   	src_fmt->width  = src_crop->width;
>>   	src_fmt->height = src_crop->height;
>> -	src_fmt->quantization = format->quantization;
>> -	/* full range by default */
>> -	if (!src_fmt->quantization)
>> +
>> +	src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>> +	src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
>> +	src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
>> +
>> +	if (mbus_info->pixel_enc == V4L2_PIXEL_ENC_BAYER)
>>   		src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>> +	else if (format->quantization == V4L2_QUANTIZATION_DEFAULT)
>> +		src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
>> +	else
>> +		src_fmt->quantization = format->quantization;
>>   
>>   	*format = *src_fmt;
>>   }
>> @@ -1068,6 +1088,17 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
>>   	return ret;
>>   }
>>   
>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
>> +			   struct v4l2_subdev_format *sd_fmt)
>> +{
>> +	struct rkisp1_isp *isp = &rkisp1->isp;
>> +
>> +	sd_fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
>> +	sd_fmt->pad = RKISP1_ISP_PAD_SOURCE_VIDEO;
>> +
>> +	return v4l2_subdev_call(&isp->sd, pad, get_fmt, NULL, sd_fmt);
> 
> Do we need to get through the external API to access data that is
> driver-internal anyway?
> 
>> +}
>> +
>>   void rkisp1_isp_unregister(struct rkisp1_device *rkisp1)
>>   {
>>   	struct v4l2_subdev *sd = &rkisp1->isp.sd;
>> diff --git a/drivers/staging/media/rkisp1/rkisp1-resizer.c b/drivers/staging/media/rkisp1/rkisp1-resizer.c
>> index 7b6b7ddd4169..8705b133de68 100644
>> --- a/drivers/staging/media/rkisp1/rkisp1-resizer.c
>> +++ b/drivers/staging/media/rkisp1/rkisp1-resizer.c
>> @@ -525,6 +525,7 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
>>   	const struct rkisp1_isp_mbus_info *mbus_info;
>>   	struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
>>   	struct v4l2_rect *sink_crop;
>> +	struct v4l2_subdev_format isp_sd_fmt;
>>   
>>   	sink_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SINK, which);
>>   	src_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SRC, which);
>> @@ -539,8 +540,20 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
>>   	if (which == V4L2_SUBDEV_FORMAT_ACTIVE)
>>   		rsz->pixel_enc = mbus_info->pixel_enc;
>>   
>> +	rkisp1_get_isp_src_fmt(rsz->rkisp1, &isp_sd_fmt);
>> +
> 
> Is this necessary? My understanding was that in the subdev model, it was
> the userspace responsibility to propagate any configuration changes through
> the graph.
> 
> Also, doing this only here wouldn't fully maintain the
> consistency of the state. For example, if one sets the ISP subdev format
> first, then the resizer subdev and then the ISP subdev again, wouldn't the
> resizer subdev end up with a wrong format?

yes, this is indeed a bug, I am preparing v4 now.
What I thought to do is adding quantization conversion
support also on ther resizer and capture entities.
Then in the 'link_validation' callbacks, there
is a validation that the quantization fields matches.

Thanks,
Dafna

> 
> Best regards,
> Tomasz
> 
>> +	sink_fmt->field = isp_sd_fmt.format.field;
>> +	sink_fmt->colorspace = isp_sd_fmt.format.colorspace;
>> +	sink_fmt->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
>> +	sink_fmt->xfer_func = isp_sd_fmt.format.xfer_func;
>> +
>>   	/* Propagete to source pad */
>>   	src_fmt->code = sink_fmt->code;
>> +	src_fmt->field = sink_fmt->field;
>> +	src_fmt->colorspace = sink_fmt->colorspace;
>> +	src_fmt->ycbcr_enc = sink_fmt->ycbcr_enc;
>> +	src_fmt->xfer_func = sink_fmt->xfer_func;
>> +	src_fmt->quantization = sink_fmt->quantization;
>>   
>>   	sink_fmt->width = clamp_t(u32, format->width,
>>   				  rsz->config->min_rsz_width,
>> -- 
>> 2.17.1
>>

Re: [RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad

[Tomasz Figa]

On Thu, Jun 4, 2020 at 9:11 PM Dafna Hirschfeld
<dafna.hirschfeld@collabora.com> wrote:
>
> Hi
>
> On 04.06.20 19:54, Tomasz Figa wrote:
> > On Thu, Apr 16, 2020 at 04:56:05PM +0200, Dafna Hirschfeld wrote:
> >> The isp entity has a hardware support to force full range quantization
> >> for YUV formats. Use the new API to indicate userspace that
> >> quantization conversion is supported by adding the flag
> >> V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION during media code enumeration.
> >> Then uppon s_fmt on the video source pad, we assign the
> >> quantization from userspace for YUV formats.
> >> Also in the capture and resizer entities we retrieve the colorspace
> >> from the isp entity.
> >>
> >> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
> >> ---
> >>   drivers/staging/media/rkisp1/rkisp1-capture.c | 28 ++++++-------
> >>   drivers/staging/media/rkisp1/rkisp1-common.h  |  2 +
> >>   drivers/staging/media/rkisp1/rkisp1-isp.c     | 39 +++++++++++++++++--
> >>   drivers/staging/media/rkisp1/rkisp1-resizer.c | 13 +++++++
> >>   4 files changed, 65 insertions(+), 17 deletions(-)
> >>
> >> diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c
> >> index fbf62399fe3d..aca0f93bc772 100644
> >> --- a/drivers/staging/media/rkisp1/rkisp1-capture.c
> >> +++ b/drivers/staging/media/rkisp1/rkisp1-capture.c
> >> @@ -1066,14 +1066,13 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
> >>                         const struct v4l2_format_info **fmt_info)
> >>   {
> >>      const struct rkisp1_capture_config *config = cap->config;
> >> -    struct rkisp1_capture *other_cap =
> >> -                    &cap->rkisp1->capture_devs[cap->id ^ 1];
> >>      const struct rkisp1_capture_fmt_cfg *fmt;
> >>      const struct v4l2_format_info *info;
> >>      const unsigned int max_widths[] = { RKISP1_RSZ_MP_SRC_MAX_WIDTH,
> >>                                          RKISP1_RSZ_SP_SRC_MAX_WIDTH };
> >>      const unsigned int max_heights[] = { RKISP1_RSZ_MP_SRC_MAX_HEIGHT,
> >>                                           RKISP1_RSZ_SP_SRC_MAX_HEIGHT};
> >> +    struct v4l2_subdev_format isp_sd_fmt;
> >>
> >>      fmt = rkisp1_find_fmt_cfg(cap, pixm->pixelformat);
> >>      if (!fmt) {
> >> @@ -1081,24 +1080,27 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
> >>              pixm->pixelformat = fmt->fourcc;
> >>      }
> >>
> >> +    rkisp1_get_isp_src_fmt(cap->rkisp1, &isp_sd_fmt);
> >> +    pixm->field = isp_sd_fmt.format.field;
> >> +    pixm->colorspace = isp_sd_fmt.format.colorspace;
> >> +    pixm->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
> >> +    pixm->xfer_func = isp_sd_fmt.format.xfer_func;
> >> +
> >> +    /*
> >> +     * isp has a feature to set full range quantization for yuv formats.
> >
> > How about "select between limited and full range for YUV formats"?
> >
> >> +     * so we need to get the format from the isp.
> >> +     */
> >> +    pixm->quantization = isp_sd_fmt.format.quantization;
> >> +    if (!v4l2_is_format_yuv(cap->pix.info))
> >> +            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >> +
> >>      pixm->width = clamp_t(u32, pixm->width,
> >>                            RKISP1_RSZ_SRC_MIN_WIDTH, max_widths[cap->id]);
> >>      pixm->height = clamp_t(u32, pixm->height,
> >>                             RKISP1_RSZ_SRC_MIN_HEIGHT, max_heights[cap->id]);
> >>
> >> -    pixm->field = V4L2_FIELD_NONE;
> >> -    pixm->colorspace = V4L2_COLORSPACE_DEFAULT;
> >> -    pixm->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> >> -
> >>      info = rkisp1_fill_pixfmt(pixm, cap->id);
> >>
> >> -    /* can not change quantization when stream-on */
> >> -    if (other_cap->is_streaming)
> >> -            pixm->quantization = other_cap->pix.fmt.quantization;
> >> -    /* output full range by default, take effect in params */
> >> -    else if (!pixm->quantization ||
> >> -             pixm->quantization > V4L2_QUANTIZATION_LIM_RANGE)
> >> -            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>
> >>      if (fmt_cfg)
> >>              *fmt_cfg = fmt;
> >> diff --git a/drivers/staging/media/rkisp1/rkisp1-common.h b/drivers/staging/media/rkisp1/rkisp1-common.h
> >> index 2d7b7e078636..7a5576fa14c9 100644
> >> --- a/drivers/staging/media/rkisp1/rkisp1-common.h
> >> +++ b/drivers/staging/media/rkisp1/rkisp1-common.h
> >> @@ -300,6 +300,8 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
> >>                      struct v4l2_device *v4l2_dev);
> >>   void rkisp1_isp_unregister(struct rkisp1_device *rkisp1);
> >>
> >> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
> >> +                       struct v4l2_subdev_format *sd_fmt);
> >>   const struct rkisp1_isp_mbus_info *rkisp1_isp_mbus_info_get(u32 mbus_code);
> >>
> >>   irqreturn_t rkisp1_isp_isr(struct rkisp1_device *rkisp1);
> >> diff --git a/drivers/staging/media/rkisp1/rkisp1-isp.c b/drivers/staging/media/rkisp1/rkisp1-isp.c
> >> index dee8e96f3900..6fdf5ed0b6b1 100644
> >> --- a/drivers/staging/media/rkisp1/rkisp1-isp.c
> >> +++ b/drivers/staging/media/rkisp1/rkisp1-isp.c
> >> @@ -613,6 +613,10 @@ static int rkisp1_isp_enum_mbus_code(struct v4l2_subdev *sd,
> >>
> >>              if (code->index == pos - 1) {
> >>                      code->code = fmt->mbus_code;
> >> +                    if (fmt->pixel_enc == V4L2_PIXEL_ENC_YUV &&
> >> +                        dir == RKISP1_DIR_SRC)
> >> +                            code->flags =
> >> +                                    V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION;
> >>                      return 0;
> >>              }
> >>      }
> >> @@ -639,12 +643,21 @@ static int rkisp1_isp_init_config(struct v4l2_subdev *sd,
> >>      sink_crop->height = RKISP1_DEFAULT_HEIGHT;
> >>      sink_crop->left = 0;
> >>      sink_crop->top = 0;
> >> +    sink_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >> +    sink_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(sink_fmt->colorspace);
> >> +    sink_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(sink_fmt->colorspace);
> >> +    sink_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >> +
> >>
> >>      src_fmt = v4l2_subdev_get_try_format(sd, cfg,
> >>                                           RKISP1_ISP_PAD_SOURCE_VIDEO);
> >>      *src_fmt = *sink_fmt;
> >>      src_fmt->code = RKISP1_DEF_SRC_PAD_FMT;
> >> -    src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
> >> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
> >> +    src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
> >> +
> >>
> >>      src_crop = v4l2_subdev_get_try_crop(sd, cfg,
> >>                                          RKISP1_ISP_PAD_SOURCE_VIDEO);
> >> @@ -687,10 +700,17 @@ static void rkisp1_isp_set_src_fmt(struct rkisp1_isp *isp,
> >>              isp->src_fmt = mbus_info;
> >>      src_fmt->width  = src_crop->width;
> >>      src_fmt->height = src_crop->height;
> >> -    src_fmt->quantization = format->quantization;
> >> -    /* full range by default */
> >> -    if (!src_fmt->quantization)
> >> +
> >> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
> >> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
> >> +
> >> +    if (mbus_info->pixel_enc == V4L2_PIXEL_ENC_BAYER)
> >>              src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >> +    else if (format->quantization == V4L2_QUANTIZATION_DEFAULT)
> >> +            src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
> >> +    else
> >> +            src_fmt->quantization = format->quantization;
> >>
> >>      *format = *src_fmt;
> >>   }
> >> @@ -1068,6 +1088,17 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
> >>      return ret;
> >>   }
> >>
> >> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
> >> +                       struct v4l2_subdev_format *sd_fmt)
> >> +{
> >> +    struct rkisp1_isp *isp = &rkisp1->isp;
> >> +
> >> +    sd_fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
> >> +    sd_fmt->pad = RKISP1_ISP_PAD_SOURCE_VIDEO;
> >> +
> >> +    return v4l2_subdev_call(&isp->sd, pad, get_fmt, NULL, sd_fmt);
> >
> > Do we need to get through the external API to access data that is
> > driver-internal anyway?
> >
> >> +}
> >> +
> >>   void rkisp1_isp_unregister(struct rkisp1_device *rkisp1)
> >>   {
> >>      struct v4l2_subdev *sd = &rkisp1->isp.sd;
> >> diff --git a/drivers/staging/media/rkisp1/rkisp1-resizer.c b/drivers/staging/media/rkisp1/rkisp1-resizer.c
> >> index 7b6b7ddd4169..8705b133de68 100644
> >> --- a/drivers/staging/media/rkisp1/rkisp1-resizer.c
> >> +++ b/drivers/staging/media/rkisp1/rkisp1-resizer.c
> >> @@ -525,6 +525,7 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
> >>      const struct rkisp1_isp_mbus_info *mbus_info;
> >>      struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
> >>      struct v4l2_rect *sink_crop;
> >> +    struct v4l2_subdev_format isp_sd_fmt;
> >>
> >>      sink_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SINK, which);
> >>      src_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SRC, which);
> >> @@ -539,8 +540,20 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
> >>      if (which == V4L2_SUBDEV_FORMAT_ACTIVE)
> >>              rsz->pixel_enc = mbus_info->pixel_enc;
> >>
> >> +    rkisp1_get_isp_src_fmt(rsz->rkisp1, &isp_sd_fmt);
> >> +
> >
> > Is this necessary? My understanding was that in the subdev model, it was
> > the userspace responsibility to propagate any configuration changes through
> > the graph.
> >
> > Also, doing this only here wouldn't fully maintain the
> > consistency of the state. For example, if one sets the ISP subdev format
> > first, then the resizer subdev and then the ISP subdev again, wouldn't the
> > resizer subdev end up with a wrong format?
>
> yes, this is indeed a bug, I am preparing v4 now.
> What I thought to do is adding quantization conversion
> support also on ther resizer and capture entities.
> Then in the 'link_validation' callbacks, there
> is a validation that the quantization fields matches.

My understanding is that, if we have the following topology

[ ISP ] -> [ Resizer ] -> [ Video node ]

then the ISP source pad would have the csc capability, while resizer
and video node would just accept whatever is configured on their sink
pads (no need for csc capability for that) and propagate that to their
outputs, i.e. resizer source pad and video node CAPTURE format.

Is this what you were going to implement?

>
> Thanks,
> Dafna
>
> >
> > Best regards,
> > Tomasz
> >
> >> +    sink_fmt->field = isp_sd_fmt.format.field;
> >> +    sink_fmt->colorspace = isp_sd_fmt.format.colorspace;
> >> +    sink_fmt->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
> >> +    sink_fmt->xfer_func = isp_sd_fmt.format.xfer_func;
> >> +
> >>      /* Propagete to source pad */
> >>      src_fmt->code = sink_fmt->code;
> >> +    src_fmt->field = sink_fmt->field;
> >> +    src_fmt->colorspace = sink_fmt->colorspace;
> >> +    src_fmt->ycbcr_enc = sink_fmt->ycbcr_enc;
> >> +    src_fmt->xfer_func = sink_fmt->xfer_func;
> >> +    src_fmt->quantization = sink_fmt->quantization;
> >>
> >>      sink_fmt->width = clamp_t(u32, format->width,
> >>                                rsz->config->min_rsz_width,
> >> --
> >> 2.17.1
> >>

Re: [RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad

[Tomasz Figa]

On Wed, Jun 10, 2020 at 2:08 PM Tomasz Figa <tfiga@chromium.org> wrote:
>
> On Thu, Jun 4, 2020 at 9:11 PM Dafna Hirschfeld
> <dafna.hirschfeld@collabora.com> wrote:
> >
> > Hi
> >
> > On 04.06.20 19:54, Tomasz Figa wrote:
> > > On Thu, Apr 16, 2020 at 04:56:05PM +0200, Dafna Hirschfeld wrote:
> > >> The isp entity has a hardware support to force full range quantization
> > >> for YUV formats. Use the new API to indicate userspace that
> > >> quantization conversion is supported by adding the flag
> > >> V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION during media code enumeration.
> > >> Then uppon s_fmt on the video source pad, we assign the
> > >> quantization from userspace for YUV formats.
> > >> Also in the capture and resizer entities we retrieve the colorspace
> > >> from the isp entity.
> > >>
> > >> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
> > >> ---
> > >>   drivers/staging/media/rkisp1/rkisp1-capture.c | 28 ++++++-------
> > >>   drivers/staging/media/rkisp1/rkisp1-common.h  |  2 +
> > >>   drivers/staging/media/rkisp1/rkisp1-isp.c     | 39 +++++++++++++++++--
> > >>   drivers/staging/media/rkisp1/rkisp1-resizer.c | 13 +++++++
> > >>   4 files changed, 65 insertions(+), 17 deletions(-)
> > >>
> > >> diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c
> > >> index fbf62399fe3d..aca0f93bc772 100644
> > >> --- a/drivers/staging/media/rkisp1/rkisp1-capture.c
> > >> +++ b/drivers/staging/media/rkisp1/rkisp1-capture.c
> > >> @@ -1066,14 +1066,13 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
> > >>                         const struct v4l2_format_info **fmt_info)
> > >>   {
> > >>      const struct rkisp1_capture_config *config = cap->config;
> > >> -    struct rkisp1_capture *other_cap =
> > >> -                    &cap->rkisp1->capture_devs[cap->id ^ 1];
> > >>      const struct rkisp1_capture_fmt_cfg *fmt;
> > >>      const struct v4l2_format_info *info;
> > >>      const unsigned int max_widths[] = { RKISP1_RSZ_MP_SRC_MAX_WIDTH,
> > >>                                          RKISP1_RSZ_SP_SRC_MAX_WIDTH };
> > >>      const unsigned int max_heights[] = { RKISP1_RSZ_MP_SRC_MAX_HEIGHT,
> > >>                                           RKISP1_RSZ_SP_SRC_MAX_HEIGHT};
> > >> +    struct v4l2_subdev_format isp_sd_fmt;
> > >>
> > >>      fmt = rkisp1_find_fmt_cfg(cap, pixm->pixelformat);
> > >>      if (!fmt) {
> > >> @@ -1081,24 +1080,27 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
> > >>              pixm->pixelformat = fmt->fourcc;
> > >>      }
> > >>
> > >> +    rkisp1_get_isp_src_fmt(cap->rkisp1, &isp_sd_fmt);
> > >> +    pixm->field = isp_sd_fmt.format.field;
> > >> +    pixm->colorspace = isp_sd_fmt.format.colorspace;
> > >> +    pixm->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
> > >> +    pixm->xfer_func = isp_sd_fmt.format.xfer_func;
> > >> +
> > >> +    /*
> > >> +     * isp has a feature to set full range quantization for yuv formats.
> > >
> > > How about "select between limited and full range for YUV formats"?
> > >
> > >> +     * so we need to get the format from the isp.
> > >> +     */
> > >> +    pixm->quantization = isp_sd_fmt.format.quantization;
> > >> +    if (!v4l2_is_format_yuv(cap->pix.info))
> > >> +            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> > >> +
> > >>      pixm->width = clamp_t(u32, pixm->width,
> > >>                            RKISP1_RSZ_SRC_MIN_WIDTH, max_widths[cap->id]);
> > >>      pixm->height = clamp_t(u32, pixm->height,
> > >>                             RKISP1_RSZ_SRC_MIN_HEIGHT, max_heights[cap->id]);
> > >>
> > >> -    pixm->field = V4L2_FIELD_NONE;
> > >> -    pixm->colorspace = V4L2_COLORSPACE_DEFAULT;
> > >> -    pixm->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> > >> -
> > >>      info = rkisp1_fill_pixfmt(pixm, cap->id);
> > >>
> > >> -    /* can not change quantization when stream-on */
> > >> -    if (other_cap->is_streaming)
> > >> -            pixm->quantization = other_cap->pix.fmt.quantization;
> > >> -    /* output full range by default, take effect in params */
> > >> -    else if (!pixm->quantization ||
> > >> -             pixm->quantization > V4L2_QUANTIZATION_LIM_RANGE)
> > >> -            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> > >>
> > >>      if (fmt_cfg)
> > >>              *fmt_cfg = fmt;
> > >> diff --git a/drivers/staging/media/rkisp1/rkisp1-common.h b/drivers/staging/media/rkisp1/rkisp1-common.h
> > >> index 2d7b7e078636..7a5576fa14c9 100644
> > >> --- a/drivers/staging/media/rkisp1/rkisp1-common.h
> > >> +++ b/drivers/staging/media/rkisp1/rkisp1-common.h
> > >> @@ -300,6 +300,8 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
> > >>                      struct v4l2_device *v4l2_dev);
> > >>   void rkisp1_isp_unregister(struct rkisp1_device *rkisp1);
> > >>
> > >> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
> > >> +                       struct v4l2_subdev_format *sd_fmt);
> > >>   const struct rkisp1_isp_mbus_info *rkisp1_isp_mbus_info_get(u32 mbus_code);
> > >>
> > >>   irqreturn_t rkisp1_isp_isr(struct rkisp1_device *rkisp1);
> > >> diff --git a/drivers/staging/media/rkisp1/rkisp1-isp.c b/drivers/staging/media/rkisp1/rkisp1-isp.c
> > >> index dee8e96f3900..6fdf5ed0b6b1 100644
> > >> --- a/drivers/staging/media/rkisp1/rkisp1-isp.c
> > >> +++ b/drivers/staging/media/rkisp1/rkisp1-isp.c
> > >> @@ -613,6 +613,10 @@ static int rkisp1_isp_enum_mbus_code(struct v4l2_subdev *sd,
> > >>
> > >>              if (code->index == pos - 1) {
> > >>                      code->code = fmt->mbus_code;
> > >> +                    if (fmt->pixel_enc == V4L2_PIXEL_ENC_YUV &&
> > >> +                        dir == RKISP1_DIR_SRC)
> > >> +                            code->flags =
> > >> +                                    V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION;
> > >>                      return 0;
> > >>              }
> > >>      }
> > >> @@ -639,12 +643,21 @@ static int rkisp1_isp_init_config(struct v4l2_subdev *sd,
> > >>      sink_crop->height = RKISP1_DEFAULT_HEIGHT;
> > >>      sink_crop->left = 0;
> > >>      sink_crop->top = 0;
> > >> +    sink_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> > >> +    sink_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(sink_fmt->colorspace);
> > >> +    sink_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(sink_fmt->colorspace);
> > >> +    sink_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> > >> +
> > >>
> > >>      src_fmt = v4l2_subdev_get_try_format(sd, cfg,
> > >>                                           RKISP1_ISP_PAD_SOURCE_VIDEO);
> > >>      *src_fmt = *sink_fmt;
> > >>      src_fmt->code = RKISP1_DEF_SRC_PAD_FMT;
> > >> -    src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> > >> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> > >> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
> > >> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
> > >> +    src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
> > >> +
> > >>
> > >>      src_crop = v4l2_subdev_get_try_crop(sd, cfg,
> > >>                                          RKISP1_ISP_PAD_SOURCE_VIDEO);
> > >> @@ -687,10 +700,17 @@ static void rkisp1_isp_set_src_fmt(struct rkisp1_isp *isp,
> > >>              isp->src_fmt = mbus_info;
> > >>      src_fmt->width  = src_crop->width;
> > >>      src_fmt->height = src_crop->height;
> > >> -    src_fmt->quantization = format->quantization;
> > >> -    /* full range by default */
> > >> -    if (!src_fmt->quantization)
> > >> +
> > >> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> > >> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
> > >> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
> > >> +
> > >> +    if (mbus_info->pixel_enc == V4L2_PIXEL_ENC_BAYER)
> > >>              src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> > >> +    else if (format->quantization == V4L2_QUANTIZATION_DEFAULT)
> > >> +            src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
> > >> +    else
> > >> +            src_fmt->quantization = format->quantization;
> > >>
> > >>      *format = *src_fmt;
> > >>   }
> > >> @@ -1068,6 +1088,17 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
> > >>      return ret;
> > >>   }
> > >>
> > >> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
> > >> +                       struct v4l2_subdev_format *sd_fmt)
> > >> +{
> > >> +    struct rkisp1_isp *isp = &rkisp1->isp;
> > >> +
> > >> +    sd_fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
> > >> +    sd_fmt->pad = RKISP1_ISP_PAD_SOURCE_VIDEO;
> > >> +
> > >> +    return v4l2_subdev_call(&isp->sd, pad, get_fmt, NULL, sd_fmt);
> > >
> > > Do we need to get through the external API to access data that is
> > > driver-internal anyway?
> > >
> > >> +}
> > >> +
> > >>   void rkisp1_isp_unregister(struct rkisp1_device *rkisp1)
> > >>   {
> > >>      struct v4l2_subdev *sd = &rkisp1->isp.sd;
> > >> diff --git a/drivers/staging/media/rkisp1/rkisp1-resizer.c b/drivers/staging/media/rkisp1/rkisp1-resizer.c
> > >> index 7b6b7ddd4169..8705b133de68 100644
> > >> --- a/drivers/staging/media/rkisp1/rkisp1-resizer.c
> > >> +++ b/drivers/staging/media/rkisp1/rkisp1-resizer.c
> > >> @@ -525,6 +525,7 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
> > >>      const struct rkisp1_isp_mbus_info *mbus_info;
> > >>      struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
> > >>      struct v4l2_rect *sink_crop;
> > >> +    struct v4l2_subdev_format isp_sd_fmt;
> > >>
> > >>      sink_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SINK, which);
> > >>      src_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SRC, which);
> > >> @@ -539,8 +540,20 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
> > >>      if (which == V4L2_SUBDEV_FORMAT_ACTIVE)
> > >>              rsz->pixel_enc = mbus_info->pixel_enc;
> > >>
> > >> +    rkisp1_get_isp_src_fmt(rsz->rkisp1, &isp_sd_fmt);
> > >> +
> > >
> > > Is this necessary? My understanding was that in the subdev model, it was
> > > the userspace responsibility to propagate any configuration changes through
> > > the graph.
> > >
> > > Also, doing this only here wouldn't fully maintain the
> > > consistency of the state. For example, if one sets the ISP subdev format
> > > first, then the resizer subdev and then the ISP subdev again, wouldn't the
> > > resizer subdev end up with a wrong format?
> >
> > yes, this is indeed a bug, I am preparing v4 now.
> > What I thought to do is adding quantization conversion
> > support also on ther resizer and capture entities.
> > Then in the 'link_validation' callbacks, there
> > is a validation that the quantization fields matches.
>
> My understanding is that, if we have the following topology
>
> [ ISP ] -> [ Resizer ] -> [ Video node ]
>
> then the ISP source pad would have the csc capability, while resizer
> and video node would just accept whatever is configured on their sink
> pads (no need for csc capability for that) and propagate that to their
> outputs, i.e. resizer source pad and video node CAPTURE format.
>
> Is this what you were going to implement?

Gentle ping.

Best regards,
Tomasz

Re: [RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad

[Dafna Hirschfeld]

On 18.06.20 19:27, Tomasz Figa wrote:
> On Wed, Jun 10, 2020 at 2:08 PM Tomasz Figa <tfiga@chromium.org> wrote:
>>
>> On Thu, Jun 4, 2020 at 9:11 PM Dafna Hirschfeld
>> <dafna.hirschfeld@collabora.com> wrote:
>>>
>>> Hi
>>>
>>> On 04.06.20 19:54, Tomasz Figa wrote:
>>>> On Thu, Apr 16, 2020 at 04:56:05PM +0200, Dafna Hirschfeld wrote:
>>>>> The isp entity has a hardware support to force full range quantization
>>>>> for YUV formats. Use the new API to indicate userspace that
>>>>> quantization conversion is supported by adding the flag
>>>>> V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION during media code enumeration.
>>>>> Then uppon s_fmt on the video source pad, we assign the
>>>>> quantization from userspace for YUV formats.
>>>>> Also in the capture and resizer entities we retrieve the colorspace
>>>>> from the isp entity.
>>>>>
>>>>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
>>>>> ---
>>>>>    drivers/staging/media/rkisp1/rkisp1-capture.c | 28 ++++++-------
>>>>>    drivers/staging/media/rkisp1/rkisp1-common.h  |  2 +
>>>>>    drivers/staging/media/rkisp1/rkisp1-isp.c     | 39 +++++++++++++++++--
>>>>>    drivers/staging/media/rkisp1/rkisp1-resizer.c | 13 +++++++
>>>>>    4 files changed, 65 insertions(+), 17 deletions(-)
>>>>>
>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>> index fbf62399fe3d..aca0f93bc772 100644
>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>> @@ -1066,14 +1066,13 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
>>>>>                          const struct v4l2_format_info **fmt_info)
>>>>>    {
>>>>>       const struct rkisp1_capture_config *config = cap->config;
>>>>> -    struct rkisp1_capture *other_cap =
>>>>> -                    &cap->rkisp1->capture_devs[cap->id ^ 1];
>>>>>       const struct rkisp1_capture_fmt_cfg *fmt;
>>>>>       const struct v4l2_format_info *info;
>>>>>       const unsigned int max_widths[] = { RKISP1_RSZ_MP_SRC_MAX_WIDTH,
>>>>>                                           RKISP1_RSZ_SP_SRC_MAX_WIDTH };
>>>>>       const unsigned int max_heights[] = { RKISP1_RSZ_MP_SRC_MAX_HEIGHT,
>>>>>                                            RKISP1_RSZ_SP_SRC_MAX_HEIGHT};
>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
>>>>>
>>>>>       fmt = rkisp1_find_fmt_cfg(cap, pixm->pixelformat);
>>>>>       if (!fmt) {
>>>>> @@ -1081,24 +1080,27 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
>>>>>               pixm->pixelformat = fmt->fourcc;
>>>>>       }
>>>>>
>>>>> +    rkisp1_get_isp_src_fmt(cap->rkisp1, &isp_sd_fmt);
>>>>> +    pixm->field = isp_sd_fmt.format.field;
>>>>> +    pixm->colorspace = isp_sd_fmt.format.colorspace;
>>>>> +    pixm->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
>>>>> +    pixm->xfer_func = isp_sd_fmt.format.xfer_func;
>>>>> +
>>>>> +    /*
>>>>> +     * isp has a feature to set full range quantization for yuv formats.
>>>>
>>>> How about "select between limited and full range for YUV formats"?
>>>>
>>>>> +     * so we need to get the format from the isp.
>>>>> +     */
>>>>> +    pixm->quantization = isp_sd_fmt.format.quantization;
>>>>> +    if (!v4l2_is_format_yuv(cap->pix.info))
>>>>> +            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>> +
>>>>>       pixm->width = clamp_t(u32, pixm->width,
>>>>>                             RKISP1_RSZ_SRC_MIN_WIDTH, max_widths[cap->id]);
>>>>>       pixm->height = clamp_t(u32, pixm->height,
>>>>>                              RKISP1_RSZ_SRC_MIN_HEIGHT, max_heights[cap->id]);
>>>>>
>>>>> -    pixm->field = V4L2_FIELD_NONE;
>>>>> -    pixm->colorspace = V4L2_COLORSPACE_DEFAULT;
>>>>> -    pixm->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>>>>> -
>>>>>       info = rkisp1_fill_pixfmt(pixm, cap->id);
>>>>>
>>>>> -    /* can not change quantization when stream-on */
>>>>> -    if (other_cap->is_streaming)
>>>>> -            pixm->quantization = other_cap->pix.fmt.quantization;
>>>>> -    /* output full range by default, take effect in params */
>>>>> -    else if (!pixm->quantization ||
>>>>> -             pixm->quantization > V4L2_QUANTIZATION_LIM_RANGE)
>>>>> -            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>
>>>>>       if (fmt_cfg)
>>>>>               *fmt_cfg = fmt;
>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-common.h b/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>> index 2d7b7e078636..7a5576fa14c9 100644
>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>> @@ -300,6 +300,8 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
>>>>>                       struct v4l2_device *v4l2_dev);
>>>>>    void rkisp1_isp_unregister(struct rkisp1_device *rkisp1);
>>>>>
>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
>>>>> +                       struct v4l2_subdev_format *sd_fmt);
>>>>>    const struct rkisp1_isp_mbus_info *rkisp1_isp_mbus_info_get(u32 mbus_code);
>>>>>
>>>>>    irqreturn_t rkisp1_isp_isr(struct rkisp1_device *rkisp1);
>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-isp.c b/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>> index dee8e96f3900..6fdf5ed0b6b1 100644
>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>> @@ -613,6 +613,10 @@ static int rkisp1_isp_enum_mbus_code(struct v4l2_subdev *sd,
>>>>>
>>>>>               if (code->index == pos - 1) {
>>>>>                       code->code = fmt->mbus_code;
>>>>> +                    if (fmt->pixel_enc == V4L2_PIXEL_ENC_YUV &&
>>>>> +                        dir == RKISP1_DIR_SRC)
>>>>> +                            code->flags =
>>>>> +                                    V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION;
>>>>>                       return 0;
>>>>>               }
>>>>>       }
>>>>> @@ -639,12 +643,21 @@ static int rkisp1_isp_init_config(struct v4l2_subdev *sd,
>>>>>       sink_crop->height = RKISP1_DEFAULT_HEIGHT;
>>>>>       sink_crop->left = 0;
>>>>>       sink_crop->top = 0;
>>>>> +    sink_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>> +    sink_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(sink_fmt->colorspace);
>>>>> +    sink_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(sink_fmt->colorspace);
>>>>> +    sink_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>> +
>>>>>
>>>>>       src_fmt = v4l2_subdev_get_try_format(sd, cfg,
>>>>>                                            RKISP1_ISP_PAD_SOURCE_VIDEO);
>>>>>       *src_fmt = *sink_fmt;
>>>>>       src_fmt->code = RKISP1_DEF_SRC_PAD_FMT;
>>>>> -    src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
>>>>> +    src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
>>>>> +
>>>>>
>>>>>       src_crop = v4l2_subdev_get_try_crop(sd, cfg,
>>>>>                                           RKISP1_ISP_PAD_SOURCE_VIDEO);
>>>>> @@ -687,10 +700,17 @@ static void rkisp1_isp_set_src_fmt(struct rkisp1_isp *isp,
>>>>>               isp->src_fmt = mbus_info;
>>>>>       src_fmt->width  = src_crop->width;
>>>>>       src_fmt->height = src_crop->height;
>>>>> -    src_fmt->quantization = format->quantization;
>>>>> -    /* full range by default */
>>>>> -    if (!src_fmt->quantization)
>>>>> +
>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
>>>>> +
>>>>> +    if (mbus_info->pixel_enc == V4L2_PIXEL_ENC_BAYER)
>>>>>               src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>> +    else if (format->quantization == V4L2_QUANTIZATION_DEFAULT)
>>>>> +            src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
>>>>> +    else
>>>>> +            src_fmt->quantization = format->quantization;
>>>>>
>>>>>       *format = *src_fmt;
>>>>>    }
>>>>> @@ -1068,6 +1088,17 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
>>>>>       return ret;
>>>>>    }
>>>>>
>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
>>>>> +                       struct v4l2_subdev_format *sd_fmt)
>>>>> +{
>>>>> +    struct rkisp1_isp *isp = &rkisp1->isp;
>>>>> +
>>>>> +    sd_fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
>>>>> +    sd_fmt->pad = RKISP1_ISP_PAD_SOURCE_VIDEO;
>>>>> +
>>>>> +    return v4l2_subdev_call(&isp->sd, pad, get_fmt, NULL, sd_fmt);
>>>>
>>>> Do we need to get through the external API to access data that is
>>>> driver-internal anyway?
>>>>
>>>>> +}
>>>>> +
>>>>>    void rkisp1_isp_unregister(struct rkisp1_device *rkisp1)
>>>>>    {
>>>>>       struct v4l2_subdev *sd = &rkisp1->isp.sd;
>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-resizer.c b/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>> index 7b6b7ddd4169..8705b133de68 100644
>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>> @@ -525,6 +525,7 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
>>>>>       const struct rkisp1_isp_mbus_info *mbus_info;
>>>>>       struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
>>>>>       struct v4l2_rect *sink_crop;
>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
>>>>>
>>>>>       sink_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SINK, which);
>>>>>       src_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SRC, which);
>>>>> @@ -539,8 +540,20 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
>>>>>       if (which == V4L2_SUBDEV_FORMAT_ACTIVE)
>>>>>               rsz->pixel_enc = mbus_info->pixel_enc;
>>>>>
>>>>> +    rkisp1_get_isp_src_fmt(rsz->rkisp1, &isp_sd_fmt);
>>>>> +
>>>>
>>>> Is this necessary? My understanding was that in the subdev model, it was
>>>> the userspace responsibility to propagate any configuration changes through
>>>> the graph.
>>>>
>>>> Also, doing this only here wouldn't fully maintain the
>>>> consistency of the state. For example, if one sets the ISP subdev format
>>>> first, then the resizer subdev and then the ISP subdev again, wouldn't the
>>>> resizer subdev end up with a wrong format?
>>>
>>> yes, this is indeed a bug, I am preparing v4 now.
>>> What I thought to do is adding quantization conversion
>>> support also on ther resizer and capture entities.
>>> Then in the 'link_validation' callbacks, there
>>> is a validation that the quantization fields matches.
>>
>> My understanding is that, if we have the following topology
>>
>> [ ISP ] -> [ Resizer ] -> [ Video node ]
>>
>> then the ISP source pad would have the csc capability, while resizer
>> and video node would just accept whatever is configured on their sink
>> pads (no need for csc capability for that) and propagate that to their
>> outputs, i.e. resizer source pad and video node CAPTURE format.
>>
>> Is this what you were going to implement?
Hi, I sent a v4 where the CSC capability is set on the reiszer and capture as well.
I can send a v5 implementing it the way you suggest. Currently the doc says that the colorspace fields
must be set by the driver for capture streams. This implies that userspace can set it
only if the CSC is supported.

Another option would be to remove the CSC API for subdevices and to decide that the quantiztion is a feature
of the capture device.

Thanks,
Dafna
> 
> Gentle ping.
> 
> Best regards,
> Tomasz
> 

Re: [RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad

[Tomasz Figa]

On Thu, Jun 18, 2020 at 08:26:33PM +0200, Dafna Hirschfeld wrote:
> 
> 
> On 18.06.20 19:27, Tomasz Figa wrote:
> > On Wed, Jun 10, 2020 at 2:08 PM Tomasz Figa <tfiga@chromium.org> wrote:
> > > 
> > > On Thu, Jun 4, 2020 at 9:11 PM Dafna Hirschfeld
> > > <dafna.hirschfeld@collabora.com> wrote:
> > > > 
> > > > Hi
> > > > 
> > > > On 04.06.20 19:54, Tomasz Figa wrote:
> > > > > On Thu, Apr 16, 2020 at 04:56:05PM +0200, Dafna Hirschfeld wrote:
> > > > > > The isp entity has a hardware support to force full range quantization
> > > > > > for YUV formats. Use the new API to indicate userspace that
> > > > > > quantization conversion is supported by adding the flag
> > > > > > V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION during media code enumeration.
> > > > > > Then uppon s_fmt on the video source pad, we assign the
> > > > > > quantization from userspace for YUV formats.
> > > > > > Also in the capture and resizer entities we retrieve the colorspace
> > > > > > from the isp entity.
> > > > > > 
> > > > > > Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
> > > > > > ---
> > > > > >    drivers/staging/media/rkisp1/rkisp1-capture.c | 28 ++++++-------
> > > > > >    drivers/staging/media/rkisp1/rkisp1-common.h  |  2 +
> > > > > >    drivers/staging/media/rkisp1/rkisp1-isp.c     | 39 +++++++++++++++++--
> > > > > >    drivers/staging/media/rkisp1/rkisp1-resizer.c | 13 +++++++
> > > > > >    4 files changed, 65 insertions(+), 17 deletions(-)
> > > > > > 
> > > > > > diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c
> > > > > > index fbf62399fe3d..aca0f93bc772 100644
> > > > > > --- a/drivers/staging/media/rkisp1/rkisp1-capture.c
> > > > > > +++ b/drivers/staging/media/rkisp1/rkisp1-capture.c
> > > > > > @@ -1066,14 +1066,13 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
> > > > > >                          const struct v4l2_format_info **fmt_info)
> > > > > >    {
> > > > > >       const struct rkisp1_capture_config *config = cap->config;
> > > > > > -    struct rkisp1_capture *other_cap =
> > > > > > -                    &cap->rkisp1->capture_devs[cap->id ^ 1];
> > > > > >       const struct rkisp1_capture_fmt_cfg *fmt;
> > > > > >       const struct v4l2_format_info *info;
> > > > > >       const unsigned int max_widths[] = { RKISP1_RSZ_MP_SRC_MAX_WIDTH,
> > > > > >                                           RKISP1_RSZ_SP_SRC_MAX_WIDTH };
> > > > > >       const unsigned int max_heights[] = { RKISP1_RSZ_MP_SRC_MAX_HEIGHT,
> > > > > >                                            RKISP1_RSZ_SP_SRC_MAX_HEIGHT};
> > > > > > +    struct v4l2_subdev_format isp_sd_fmt;
> > > > > > 
> > > > > >       fmt = rkisp1_find_fmt_cfg(cap, pixm->pixelformat);
> > > > > >       if (!fmt) {
> > > > > > @@ -1081,24 +1080,27 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
> > > > > >               pixm->pixelformat = fmt->fourcc;
> > > > > >       }
> > > > > > 
> > > > > > +    rkisp1_get_isp_src_fmt(cap->rkisp1, &isp_sd_fmt);
> > > > > > +    pixm->field = isp_sd_fmt.format.field;
> > > > > > +    pixm->colorspace = isp_sd_fmt.format.colorspace;
> > > > > > +    pixm->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
> > > > > > +    pixm->xfer_func = isp_sd_fmt.format.xfer_func;
> > > > > > +
> > > > > > +    /*
> > > > > > +     * isp has a feature to set full range quantization for yuv formats.
> > > > > 
> > > > > How about "select between limited and full range for YUV formats"?
> > > > > 
> > > > > > +     * so we need to get the format from the isp.
> > > > > > +     */
> > > > > > +    pixm->quantization = isp_sd_fmt.format.quantization;
> > > > > > +    if (!v4l2_is_format_yuv(cap->pix.info))
> > > > > > +            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> > > > > > +
> > > > > >       pixm->width = clamp_t(u32, pixm->width,
> > > > > >                             RKISP1_RSZ_SRC_MIN_WIDTH, max_widths[cap->id]);
> > > > > >       pixm->height = clamp_t(u32, pixm->height,
> > > > > >                              RKISP1_RSZ_SRC_MIN_HEIGHT, max_heights[cap->id]);
> > > > > > 
> > > > > > -    pixm->field = V4L2_FIELD_NONE;
> > > > > > -    pixm->colorspace = V4L2_COLORSPACE_DEFAULT;
> > > > > > -    pixm->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> > > > > > -
> > > > > >       info = rkisp1_fill_pixfmt(pixm, cap->id);
> > > > > > 
> > > > > > -    /* can not change quantization when stream-on */
> > > > > > -    if (other_cap->is_streaming)
> > > > > > -            pixm->quantization = other_cap->pix.fmt.quantization;
> > > > > > -    /* output full range by default, take effect in params */
> > > > > > -    else if (!pixm->quantization ||
> > > > > > -             pixm->quantization > V4L2_QUANTIZATION_LIM_RANGE)
> > > > > > -            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> > > > > > 
> > > > > >       if (fmt_cfg)
> > > > > >               *fmt_cfg = fmt;
> > > > > > diff --git a/drivers/staging/media/rkisp1/rkisp1-common.h b/drivers/staging/media/rkisp1/rkisp1-common.h
> > > > > > index 2d7b7e078636..7a5576fa14c9 100644
> > > > > > --- a/drivers/staging/media/rkisp1/rkisp1-common.h
> > > > > > +++ b/drivers/staging/media/rkisp1/rkisp1-common.h
> > > > > > @@ -300,6 +300,8 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
> > > > > >                       struct v4l2_device *v4l2_dev);
> > > > > >    void rkisp1_isp_unregister(struct rkisp1_device *rkisp1);
> > > > > > 
> > > > > > +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
> > > > > > +                       struct v4l2_subdev_format *sd_fmt);
> > > > > >    const struct rkisp1_isp_mbus_info *rkisp1_isp_mbus_info_get(u32 mbus_code);
> > > > > > 
> > > > > >    irqreturn_t rkisp1_isp_isr(struct rkisp1_device *rkisp1);
> > > > > > diff --git a/drivers/staging/media/rkisp1/rkisp1-isp.c b/drivers/staging/media/rkisp1/rkisp1-isp.c
> > > > > > index dee8e96f3900..6fdf5ed0b6b1 100644
> > > > > > --- a/drivers/staging/media/rkisp1/rkisp1-isp.c
> > > > > > +++ b/drivers/staging/media/rkisp1/rkisp1-isp.c
> > > > > > @@ -613,6 +613,10 @@ static int rkisp1_isp_enum_mbus_code(struct v4l2_subdev *sd,
> > > > > > 
> > > > > >               if (code->index == pos - 1) {
> > > > > >                       code->code = fmt->mbus_code;
> > > > > > +                    if (fmt->pixel_enc == V4L2_PIXEL_ENC_YUV &&
> > > > > > +                        dir == RKISP1_DIR_SRC)
> > > > > > +                            code->flags =
> > > > > > +                                    V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION;
> > > > > >                       return 0;
> > > > > >               }
> > > > > >       }
> > > > > > @@ -639,12 +643,21 @@ static int rkisp1_isp_init_config(struct v4l2_subdev *sd,
> > > > > >       sink_crop->height = RKISP1_DEFAULT_HEIGHT;
> > > > > >       sink_crop->left = 0;
> > > > > >       sink_crop->top = 0;
> > > > > > +    sink_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> > > > > > +    sink_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(sink_fmt->colorspace);
> > > > > > +    sink_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(sink_fmt->colorspace);
> > > > > > +    sink_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> > > > > > +
> > > > > > 
> > > > > >       src_fmt = v4l2_subdev_get_try_format(sd, cfg,
> > > > > >                                            RKISP1_ISP_PAD_SOURCE_VIDEO);
> > > > > >       *src_fmt = *sink_fmt;
> > > > > >       src_fmt->code = RKISP1_DEF_SRC_PAD_FMT;
> > > > > > -    src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> > > > > > +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> > > > > > +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
> > > > > > +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
> > > > > > +    src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
> > > > > > +
> > > > > > 
> > > > > >       src_crop = v4l2_subdev_get_try_crop(sd, cfg,
> > > > > >                                           RKISP1_ISP_PAD_SOURCE_VIDEO);
> > > > > > @@ -687,10 +700,17 @@ static void rkisp1_isp_set_src_fmt(struct rkisp1_isp *isp,
> > > > > >               isp->src_fmt = mbus_info;
> > > > > >       src_fmt->width  = src_crop->width;
> > > > > >       src_fmt->height = src_crop->height;
> > > > > > -    src_fmt->quantization = format->quantization;
> > > > > > -    /* full range by default */
> > > > > > -    if (!src_fmt->quantization)
> > > > > > +
> > > > > > +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> > > > > > +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
> > > > > > +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
> > > > > > +
> > > > > > +    if (mbus_info->pixel_enc == V4L2_PIXEL_ENC_BAYER)
> > > > > >               src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> > > > > > +    else if (format->quantization == V4L2_QUANTIZATION_DEFAULT)
> > > > > > +            src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
> > > > > > +    else
> > > > > > +            src_fmt->quantization = format->quantization;
> > > > > > 
> > > > > >       *format = *src_fmt;
> > > > > >    }
> > > > > > @@ -1068,6 +1088,17 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
> > > > > >       return ret;
> > > > > >    }
> > > > > > 
> > > > > > +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
> > > > > > +                       struct v4l2_subdev_format *sd_fmt)
> > > > > > +{
> > > > > > +    struct rkisp1_isp *isp = &rkisp1->isp;
> > > > > > +
> > > > > > +    sd_fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
> > > > > > +    sd_fmt->pad = RKISP1_ISP_PAD_SOURCE_VIDEO;
> > > > > > +
> > > > > > +    return v4l2_subdev_call(&isp->sd, pad, get_fmt, NULL, sd_fmt);
> > > > > 
> > > > > Do we need to get through the external API to access data that is
> > > > > driver-internal anyway?
> > > > > 
> > > > > > +}
> > > > > > +
> > > > > >    void rkisp1_isp_unregister(struct rkisp1_device *rkisp1)
> > > > > >    {
> > > > > >       struct v4l2_subdev *sd = &rkisp1->isp.sd;
> > > > > > diff --git a/drivers/staging/media/rkisp1/rkisp1-resizer.c b/drivers/staging/media/rkisp1/rkisp1-resizer.c
> > > > > > index 7b6b7ddd4169..8705b133de68 100644
> > > > > > --- a/drivers/staging/media/rkisp1/rkisp1-resizer.c
> > > > > > +++ b/drivers/staging/media/rkisp1/rkisp1-resizer.c
> > > > > > @@ -525,6 +525,7 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
> > > > > >       const struct rkisp1_isp_mbus_info *mbus_info;
> > > > > >       struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
> > > > > >       struct v4l2_rect *sink_crop;
> > > > > > +    struct v4l2_subdev_format isp_sd_fmt;
> > > > > > 
> > > > > >       sink_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SINK, which);
> > > > > >       src_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SRC, which);
> > > > > > @@ -539,8 +540,20 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
> > > > > >       if (which == V4L2_SUBDEV_FORMAT_ACTIVE)
> > > > > >               rsz->pixel_enc = mbus_info->pixel_enc;
> > > > > > 
> > > > > > +    rkisp1_get_isp_src_fmt(rsz->rkisp1, &isp_sd_fmt);
> > > > > > +
> > > > > 
> > > > > Is this necessary? My understanding was that in the subdev model, it was
> > > > > the userspace responsibility to propagate any configuration changes through
> > > > > the graph.
> > > > > 
> > > > > Also, doing this only here wouldn't fully maintain the
> > > > > consistency of the state. For example, if one sets the ISP subdev format
> > > > > first, then the resizer subdev and then the ISP subdev again, wouldn't the
> > > > > resizer subdev end up with a wrong format?
> > > > 
> > > > yes, this is indeed a bug, I am preparing v4 now.
> > > > What I thought to do is adding quantization conversion
> > > > support also on ther resizer and capture entities.
> > > > Then in the 'link_validation' callbacks, there
> > > > is a validation that the quantization fields matches.
> > > 
> > > My understanding is that, if we have the following topology
> > > 
> > > [ ISP ] -> [ Resizer ] -> [ Video node ]
> > > 
> > > then the ISP source pad would have the csc capability, while resizer
> > > and video node would just accept whatever is configured on their sink
> > > pads (no need for csc capability for that) and propagate that to their
> > > outputs, i.e. resizer source pad and video node CAPTURE format.
> > > 
> > > Is this what you were going to implement?
> Hi, I sent a v4 where the CSC capability is set on the reiszer and capture as well.
> I can send a v5 implementing it the way you suggest. Currently the doc says that the colorspace fields
> must be set by the driver for capture streams. This implies that userspace can set it
> only if the CSC is supported.

Why would the userspace have to set it for the capture stream on the
resizer and video nodes? Couldn't the userspace set it to DEFAULT and
then the driver override to whatever it received on the corresponding
sink?

Best regards,
Tomasz

Re: [RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad

[Dafna Hirschfeld]

On 18.06.20 20:49, Tomasz Figa wrote:
> On Thu, Jun 18, 2020 at 08:26:33PM +0200, Dafna Hirschfeld wrote:
>>
>>
>> On 18.06.20 19:27, Tomasz Figa wrote:
>>> On Wed, Jun 10, 2020 at 2:08 PM Tomasz Figa <tfiga@chromium.org> wrote:
>>>>
>>>> On Thu, Jun 4, 2020 at 9:11 PM Dafna Hirschfeld
>>>> <dafna.hirschfeld@collabora.com> wrote:
>>>>>
>>>>> Hi
>>>>>
>>>>> On 04.06.20 19:54, Tomasz Figa wrote:
>>>>>> On Thu, Apr 16, 2020 at 04:56:05PM +0200, Dafna Hirschfeld wrote:
>>>>>>> The isp entity has a hardware support to force full range quantization
>>>>>>> for YUV formats. Use the new API to indicate userspace that
>>>>>>> quantization conversion is supported by adding the flag
>>>>>>> V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION during media code enumeration.
>>>>>>> Then uppon s_fmt on the video source pad, we assign the
>>>>>>> quantization from userspace for YUV formats.
>>>>>>> Also in the capture and resizer entities we retrieve the colorspace
>>>>>>> from the isp entity.
>>>>>>>
>>>>>>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
>>>>>>> ---
>>>>>>>     drivers/staging/media/rkisp1/rkisp1-capture.c | 28 ++++++-------
>>>>>>>     drivers/staging/media/rkisp1/rkisp1-common.h  |  2 +
>>>>>>>     drivers/staging/media/rkisp1/rkisp1-isp.c     | 39 +++++++++++++++++--
>>>>>>>     drivers/staging/media/rkisp1/rkisp1-resizer.c | 13 +++++++
>>>>>>>     4 files changed, 65 insertions(+), 17 deletions(-)
>>>>>>>
>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>>> index fbf62399fe3d..aca0f93bc772 100644
>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>>> @@ -1066,14 +1066,13 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
>>>>>>>                           const struct v4l2_format_info **fmt_info)
>>>>>>>     {
>>>>>>>        const struct rkisp1_capture_config *config = cap->config;
>>>>>>> -    struct rkisp1_capture *other_cap =
>>>>>>> -                    &cap->rkisp1->capture_devs[cap->id ^ 1];
>>>>>>>        const struct rkisp1_capture_fmt_cfg *fmt;
>>>>>>>        const struct v4l2_format_info *info;
>>>>>>>        const unsigned int max_widths[] = { RKISP1_RSZ_MP_SRC_MAX_WIDTH,
>>>>>>>                                            RKISP1_RSZ_SP_SRC_MAX_WIDTH };
>>>>>>>        const unsigned int max_heights[] = { RKISP1_RSZ_MP_SRC_MAX_HEIGHT,
>>>>>>>                                             RKISP1_RSZ_SP_SRC_MAX_HEIGHT};
>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
>>>>>>>
>>>>>>>        fmt = rkisp1_find_fmt_cfg(cap, pixm->pixelformat);
>>>>>>>        if (!fmt) {
>>>>>>> @@ -1081,24 +1080,27 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
>>>>>>>                pixm->pixelformat = fmt->fourcc;
>>>>>>>        }
>>>>>>>
>>>>>>> +    rkisp1_get_isp_src_fmt(cap->rkisp1, &isp_sd_fmt);
>>>>>>> +    pixm->field = isp_sd_fmt.format.field;
>>>>>>> +    pixm->colorspace = isp_sd_fmt.format.colorspace;
>>>>>>> +    pixm->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
>>>>>>> +    pixm->xfer_func = isp_sd_fmt.format.xfer_func;
>>>>>>> +
>>>>>>> +    /*
>>>>>>> +     * isp has a feature to set full range quantization for yuv formats.
>>>>>>
>>>>>> How about "select between limited and full range for YUV formats"?
>>>>>>
>>>>>>> +     * so we need to get the format from the isp.
>>>>>>> +     */
>>>>>>> +    pixm->quantization = isp_sd_fmt.format.quantization;
>>>>>>> +    if (!v4l2_is_format_yuv(cap->pix.info))
>>>>>>> +            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>> +
>>>>>>>        pixm->width = clamp_t(u32, pixm->width,
>>>>>>>                              RKISP1_RSZ_SRC_MIN_WIDTH, max_widths[cap->id]);
>>>>>>>        pixm->height = clamp_t(u32, pixm->height,
>>>>>>>                               RKISP1_RSZ_SRC_MIN_HEIGHT, max_heights[cap->id]);
>>>>>>>
>>>>>>> -    pixm->field = V4L2_FIELD_NONE;
>>>>>>> -    pixm->colorspace = V4L2_COLORSPACE_DEFAULT;
>>>>>>> -    pixm->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>>>>>>> -
>>>>>>>        info = rkisp1_fill_pixfmt(pixm, cap->id);
>>>>>>>
>>>>>>> -    /* can not change quantization when stream-on */
>>>>>>> -    if (other_cap->is_streaming)
>>>>>>> -            pixm->quantization = other_cap->pix.fmt.quantization;
>>>>>>> -    /* output full range by default, take effect in params */
>>>>>>> -    else if (!pixm->quantization ||
>>>>>>> -             pixm->quantization > V4L2_QUANTIZATION_LIM_RANGE)
>>>>>>> -            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>
>>>>>>>        if (fmt_cfg)
>>>>>>>                *fmt_cfg = fmt;
>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-common.h b/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>>> index 2d7b7e078636..7a5576fa14c9 100644
>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>>> @@ -300,6 +300,8 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
>>>>>>>                        struct v4l2_device *v4l2_dev);
>>>>>>>     void rkisp1_isp_unregister(struct rkisp1_device *rkisp1);
>>>>>>>
>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
>>>>>>> +                       struct v4l2_subdev_format *sd_fmt);
>>>>>>>     const struct rkisp1_isp_mbus_info *rkisp1_isp_mbus_info_get(u32 mbus_code);
>>>>>>>
>>>>>>>     irqreturn_t rkisp1_isp_isr(struct rkisp1_device *rkisp1);
>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-isp.c b/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>>> index dee8e96f3900..6fdf5ed0b6b1 100644
>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>>> @@ -613,6 +613,10 @@ static int rkisp1_isp_enum_mbus_code(struct v4l2_subdev *sd,
>>>>>>>
>>>>>>>                if (code->index == pos - 1) {
>>>>>>>                        code->code = fmt->mbus_code;
>>>>>>> +                    if (fmt->pixel_enc == V4L2_PIXEL_ENC_YUV &&
>>>>>>> +                        dir == RKISP1_DIR_SRC)
>>>>>>> +                            code->flags =
>>>>>>> +                                    V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION;
>>>>>>>                        return 0;
>>>>>>>                }
>>>>>>>        }
>>>>>>> @@ -639,12 +643,21 @@ static int rkisp1_isp_init_config(struct v4l2_subdev *sd,
>>>>>>>        sink_crop->height = RKISP1_DEFAULT_HEIGHT;
>>>>>>>        sink_crop->left = 0;
>>>>>>>        sink_crop->top = 0;
>>>>>>> +    sink_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>>> +    sink_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(sink_fmt->colorspace);
>>>>>>> +    sink_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(sink_fmt->colorspace);
>>>>>>> +    sink_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>> +
>>>>>>>
>>>>>>>        src_fmt = v4l2_subdev_get_try_format(sd, cfg,
>>>>>>>                                             RKISP1_ISP_PAD_SOURCE_VIDEO);
>>>>>>>        *src_fmt = *sink_fmt;
>>>>>>>        src_fmt->code = RKISP1_DEF_SRC_PAD_FMT;
>>>>>>> -    src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
>>>>>>> +    src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
>>>>>>> +
>>>>>>>
>>>>>>>        src_crop = v4l2_subdev_get_try_crop(sd, cfg,
>>>>>>>                                            RKISP1_ISP_PAD_SOURCE_VIDEO);
>>>>>>> @@ -687,10 +700,17 @@ static void rkisp1_isp_set_src_fmt(struct rkisp1_isp *isp,
>>>>>>>                isp->src_fmt = mbus_info;
>>>>>>>        src_fmt->width  = src_crop->width;
>>>>>>>        src_fmt->height = src_crop->height;
>>>>>>> -    src_fmt->quantization = format->quantization;
>>>>>>> -    /* full range by default */
>>>>>>> -    if (!src_fmt->quantization)
>>>>>>> +
>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
>>>>>>> +
>>>>>>> +    if (mbus_info->pixel_enc == V4L2_PIXEL_ENC_BAYER)
>>>>>>>                src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>> +    else if (format->quantization == V4L2_QUANTIZATION_DEFAULT)
>>>>>>> +            src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
>>>>>>> +    else
>>>>>>> +            src_fmt->quantization = format->quantization;
>>>>>>>
>>>>>>>        *format = *src_fmt;
>>>>>>>     }
>>>>>>> @@ -1068,6 +1088,17 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
>>>>>>>        return ret;
>>>>>>>     }
>>>>>>>
>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
>>>>>>> +                       struct v4l2_subdev_format *sd_fmt)
>>>>>>> +{
>>>>>>> +    struct rkisp1_isp *isp = &rkisp1->isp;
>>>>>>> +
>>>>>>> +    sd_fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
>>>>>>> +    sd_fmt->pad = RKISP1_ISP_PAD_SOURCE_VIDEO;
>>>>>>> +
>>>>>>> +    return v4l2_subdev_call(&isp->sd, pad, get_fmt, NULL, sd_fmt);
>>>>>>
>>>>>> Do we need to get through the external API to access data that is
>>>>>> driver-internal anyway?
>>>>>>
>>>>>>> +}
>>>>>>> +
>>>>>>>     void rkisp1_isp_unregister(struct rkisp1_device *rkisp1)
>>>>>>>     {
>>>>>>>        struct v4l2_subdev *sd = &rkisp1->isp.sd;
>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-resizer.c b/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>>> index 7b6b7ddd4169..8705b133de68 100644
>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>>> @@ -525,6 +525,7 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
>>>>>>>        const struct rkisp1_isp_mbus_info *mbus_info;
>>>>>>>        struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
>>>>>>>        struct v4l2_rect *sink_crop;
>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
>>>>>>>
>>>>>>>        sink_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SINK, which);
>>>>>>>        src_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SRC, which);
>>>>>>> @@ -539,8 +540,20 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
>>>>>>>        if (which == V4L2_SUBDEV_FORMAT_ACTIVE)
>>>>>>>                rsz->pixel_enc = mbus_info->pixel_enc;
>>>>>>>
>>>>>>> +    rkisp1_get_isp_src_fmt(rsz->rkisp1, &isp_sd_fmt);
>>>>>>> +
>>>>>>
>>>>>> Is this necessary? My understanding was that in the subdev model, it was
>>>>>> the userspace responsibility to propagate any configuration changes through
>>>>>> the graph.
>>>>>>
>>>>>> Also, doing this only here wouldn't fully maintain the
>>>>>> consistency of the state. For example, if one sets the ISP subdev format
>>>>>> first, then the resizer subdev and then the ISP subdev again, wouldn't the
>>>>>> resizer subdev end up with a wrong format?
>>>>>
>>>>> yes, this is indeed a bug, I am preparing v4 now.
>>>>> What I thought to do is adding quantization conversion
>>>>> support also on ther resizer and capture entities.
>>>>> Then in the 'link_validation' callbacks, there
>>>>> is a validation that the quantization fields matches.
>>>>
>>>> My understanding is that, if we have the following topology
>>>>
>>>> [ ISP ] -> [ Resizer ] -> [ Video node ]
>>>>
>>>> then the ISP source pad would have the csc capability, while resizer
>>>> and video node would just accept whatever is configured on their sink
>>>> pads (no need for csc capability for that) and propagate that to their
>>>> outputs, i.e. resizer source pad and video node CAPTURE format.
>>>>
>>>> Is this what you were going to implement?
>> Hi, I sent a v4 where the CSC capability is set on the reiszer and capture as well.
>> I can send a v5 implementing it the way you suggest. Currently the doc says that the colorspace fields
>> must be set by the driver for capture streams. This implies that userspace can set it
>> only if the CSC is supported.
> 
> Why would the userspace have to set it for the capture stream on the
> resizer and video nodes? Couldn't the userspace set it to DEFAULT and
> then the driver override to whatever it received on the corresponding
> sink?
What do you mean "received on the corresponding sink"? Received from whom?


Thanks,
Dafna
> 
> Best regards,
> Tomasz
> 

Re: [RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad

[Tomasz Figa]

On Thu, Jun 18, 2020 at 08:54:34PM +0200, Dafna Hirschfeld wrote:
> 
> 
> On 18.06.20 20:49, Tomasz Figa wrote:
> > On Thu, Jun 18, 2020 at 08:26:33PM +0200, Dafna Hirschfeld wrote:
> > > 
> > > 
> > > On 18.06.20 19:27, Tomasz Figa wrote:
> > > > On Wed, Jun 10, 2020 at 2:08 PM Tomasz Figa <tfiga@chromium.org> wrote:
> > > > > 
> > > > > On Thu, Jun 4, 2020 at 9:11 PM Dafna Hirschfeld
> > > > > <dafna.hirschfeld@collabora.com> wrote:
> > > > > > 
> > > > > > Hi
> > > > > > 
> > > > > > On 04.06.20 19:54, Tomasz Figa wrote:
> > > > > > > On Thu, Apr 16, 2020 at 04:56:05PM +0200, Dafna Hirschfeld wrote:
> > > > > > > > The isp entity has a hardware support to force full range quantization
> > > > > > > > for YUV formats. Use the new API to indicate userspace that
> > > > > > > > quantization conversion is supported by adding the flag
> > > > > > > > V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION during media code enumeration.
> > > > > > > > Then uppon s_fmt on the video source pad, we assign the
> > > > > > > > quantization from userspace for YUV formats.
> > > > > > > > Also in the capture and resizer entities we retrieve the colorspace
> > > > > > > > from the isp entity.
> > > > > > > > 
> > > > > > > > Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
> > > > > > > > ---
> > > > > > > >     drivers/staging/media/rkisp1/rkisp1-capture.c | 28 ++++++-------
> > > > > > > >     drivers/staging/media/rkisp1/rkisp1-common.h  |  2 +
> > > > > > > >     drivers/staging/media/rkisp1/rkisp1-isp.c     | 39 +++++++++++++++++--
> > > > > > > >     drivers/staging/media/rkisp1/rkisp1-resizer.c | 13 +++++++
> > > > > > > >     4 files changed, 65 insertions(+), 17 deletions(-)
> > > > > > > > 
> > > > > > > > diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c
> > > > > > > > index fbf62399fe3d..aca0f93bc772 100644
> > > > > > > > --- a/drivers/staging/media/rkisp1/rkisp1-capture.c
> > > > > > > > +++ b/drivers/staging/media/rkisp1/rkisp1-capture.c
> > > > > > > > @@ -1066,14 +1066,13 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
> > > > > > > >                           const struct v4l2_format_info **fmt_info)
> > > > > > > >     {
> > > > > > > >        const struct rkisp1_capture_config *config = cap->config;
> > > > > > > > -    struct rkisp1_capture *other_cap =
> > > > > > > > -                    &cap->rkisp1->capture_devs[cap->id ^ 1];
> > > > > > > >        const struct rkisp1_capture_fmt_cfg *fmt;
> > > > > > > >        const struct v4l2_format_info *info;
> > > > > > > >        const unsigned int max_widths[] = { RKISP1_RSZ_MP_SRC_MAX_WIDTH,
> > > > > > > >                                            RKISP1_RSZ_SP_SRC_MAX_WIDTH };
> > > > > > > >        const unsigned int max_heights[] = { RKISP1_RSZ_MP_SRC_MAX_HEIGHT,
> > > > > > > >                                             RKISP1_RSZ_SP_SRC_MAX_HEIGHT};
> > > > > > > > +    struct v4l2_subdev_format isp_sd_fmt;
> > > > > > > > 
> > > > > > > >        fmt = rkisp1_find_fmt_cfg(cap, pixm->pixelformat);
> > > > > > > >        if (!fmt) {
> > > > > > > > @@ -1081,24 +1080,27 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
> > > > > > > >                pixm->pixelformat = fmt->fourcc;
> > > > > > > >        }
> > > > > > > > 
> > > > > > > > +    rkisp1_get_isp_src_fmt(cap->rkisp1, &isp_sd_fmt);
> > > > > > > > +    pixm->field = isp_sd_fmt.format.field;
> > > > > > > > +    pixm->colorspace = isp_sd_fmt.format.colorspace;
> > > > > > > > +    pixm->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
> > > > > > > > +    pixm->xfer_func = isp_sd_fmt.format.xfer_func;
> > > > > > > > +
> > > > > > > > +    /*
> > > > > > > > +     * isp has a feature to set full range quantization for yuv formats.
> > > > > > > 
> > > > > > > How about "select between limited and full range for YUV formats"?
> > > > > > > 
> > > > > > > > +     * so we need to get the format from the isp.
> > > > > > > > +     */
> > > > > > > > +    pixm->quantization = isp_sd_fmt.format.quantization;
> > > > > > > > +    if (!v4l2_is_format_yuv(cap->pix.info))
> > > > > > > > +            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> > > > > > > > +
> > > > > > > >        pixm->width = clamp_t(u32, pixm->width,
> > > > > > > >                              RKISP1_RSZ_SRC_MIN_WIDTH, max_widths[cap->id]);
> > > > > > > >        pixm->height = clamp_t(u32, pixm->height,
> > > > > > > >                               RKISP1_RSZ_SRC_MIN_HEIGHT, max_heights[cap->id]);
> > > > > > > > 
> > > > > > > > -    pixm->field = V4L2_FIELD_NONE;
> > > > > > > > -    pixm->colorspace = V4L2_COLORSPACE_DEFAULT;
> > > > > > > > -    pixm->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> > > > > > > > -
> > > > > > > >        info = rkisp1_fill_pixfmt(pixm, cap->id);
> > > > > > > > 
> > > > > > > > -    /* can not change quantization when stream-on */
> > > > > > > > -    if (other_cap->is_streaming)
> > > > > > > > -            pixm->quantization = other_cap->pix.fmt.quantization;
> > > > > > > > -    /* output full range by default, take effect in params */
> > > > > > > > -    else if (!pixm->quantization ||
> > > > > > > > -             pixm->quantization > V4L2_QUANTIZATION_LIM_RANGE)
> > > > > > > > -            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> > > > > > > > 
> > > > > > > >        if (fmt_cfg)
> > > > > > > >                *fmt_cfg = fmt;
> > > > > > > > diff --git a/drivers/staging/media/rkisp1/rkisp1-common.h b/drivers/staging/media/rkisp1/rkisp1-common.h
> > > > > > > > index 2d7b7e078636..7a5576fa14c9 100644
> > > > > > > > --- a/drivers/staging/media/rkisp1/rkisp1-common.h
> > > > > > > > +++ b/drivers/staging/media/rkisp1/rkisp1-common.h
> > > > > > > > @@ -300,6 +300,8 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
> > > > > > > >                        struct v4l2_device *v4l2_dev);
> > > > > > > >     void rkisp1_isp_unregister(struct rkisp1_device *rkisp1);
> > > > > > > > 
> > > > > > > > +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
> > > > > > > > +                       struct v4l2_subdev_format *sd_fmt);
> > > > > > > >     const struct rkisp1_isp_mbus_info *rkisp1_isp_mbus_info_get(u32 mbus_code);
> > > > > > > > 
> > > > > > > >     irqreturn_t rkisp1_isp_isr(struct rkisp1_device *rkisp1);
> > > > > > > > diff --git a/drivers/staging/media/rkisp1/rkisp1-isp.c b/drivers/staging/media/rkisp1/rkisp1-isp.c
> > > > > > > > index dee8e96f3900..6fdf5ed0b6b1 100644
> > > > > > > > --- a/drivers/staging/media/rkisp1/rkisp1-isp.c
> > > > > > > > +++ b/drivers/staging/media/rkisp1/rkisp1-isp.c
> > > > > > > > @@ -613,6 +613,10 @@ static int rkisp1_isp_enum_mbus_code(struct v4l2_subdev *sd,
> > > > > > > > 
> > > > > > > >                if (code->index == pos - 1) {
> > > > > > > >                        code->code = fmt->mbus_code;
> > > > > > > > +                    if (fmt->pixel_enc == V4L2_PIXEL_ENC_YUV &&
> > > > > > > > +                        dir == RKISP1_DIR_SRC)
> > > > > > > > +                            code->flags =
> > > > > > > > +                                    V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION;
> > > > > > > >                        return 0;
> > > > > > > >                }
> > > > > > > >        }
> > > > > > > > @@ -639,12 +643,21 @@ static int rkisp1_isp_init_config(struct v4l2_subdev *sd,
> > > > > > > >        sink_crop->height = RKISP1_DEFAULT_HEIGHT;
> > > > > > > >        sink_crop->left = 0;
> > > > > > > >        sink_crop->top = 0;
> > > > > > > > +    sink_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> > > > > > > > +    sink_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(sink_fmt->colorspace);
> > > > > > > > +    sink_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(sink_fmt->colorspace);
> > > > > > > > +    sink_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> > > > > > > > +
> > > > > > > > 
> > > > > > > >        src_fmt = v4l2_subdev_get_try_format(sd, cfg,
> > > > > > > >                                             RKISP1_ISP_PAD_SOURCE_VIDEO);
> > > > > > > >        *src_fmt = *sink_fmt;
> > > > > > > >        src_fmt->code = RKISP1_DEF_SRC_PAD_FMT;
> > > > > > > > -    src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> > > > > > > > +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> > > > > > > > +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
> > > > > > > > +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
> > > > > > > > +    src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
> > > > > > > > +
> > > > > > > > 
> > > > > > > >        src_crop = v4l2_subdev_get_try_crop(sd, cfg,
> > > > > > > >                                            RKISP1_ISP_PAD_SOURCE_VIDEO);
> > > > > > > > @@ -687,10 +700,17 @@ static void rkisp1_isp_set_src_fmt(struct rkisp1_isp *isp,
> > > > > > > >                isp->src_fmt = mbus_info;
> > > > > > > >        src_fmt->width  = src_crop->width;
> > > > > > > >        src_fmt->height = src_crop->height;
> > > > > > > > -    src_fmt->quantization = format->quantization;
> > > > > > > > -    /* full range by default */
> > > > > > > > -    if (!src_fmt->quantization)
> > > > > > > > +
> > > > > > > > +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> > > > > > > > +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
> > > > > > > > +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
> > > > > > > > +
> > > > > > > > +    if (mbus_info->pixel_enc == V4L2_PIXEL_ENC_BAYER)
> > > > > > > >                src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> > > > > > > > +    else if (format->quantization == V4L2_QUANTIZATION_DEFAULT)
> > > > > > > > +            src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
> > > > > > > > +    else
> > > > > > > > +            src_fmt->quantization = format->quantization;
> > > > > > > > 
> > > > > > > >        *format = *src_fmt;
> > > > > > > >     }
> > > > > > > > @@ -1068,6 +1088,17 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
> > > > > > > >        return ret;
> > > > > > > >     }
> > > > > > > > 
> > > > > > > > +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
> > > > > > > > +                       struct v4l2_subdev_format *sd_fmt)
> > > > > > > > +{
> > > > > > > > +    struct rkisp1_isp *isp = &rkisp1->isp;
> > > > > > > > +
> > > > > > > > +    sd_fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
> > > > > > > > +    sd_fmt->pad = RKISP1_ISP_PAD_SOURCE_VIDEO;
> > > > > > > > +
> > > > > > > > +    return v4l2_subdev_call(&isp->sd, pad, get_fmt, NULL, sd_fmt);
> > > > > > > 
> > > > > > > Do we need to get through the external API to access data that is
> > > > > > > driver-internal anyway?
> > > > > > > 
> > > > > > > > +}
> > > > > > > > +
> > > > > > > >     void rkisp1_isp_unregister(struct rkisp1_device *rkisp1)
> > > > > > > >     {
> > > > > > > >        struct v4l2_subdev *sd = &rkisp1->isp.sd;
> > > > > > > > diff --git a/drivers/staging/media/rkisp1/rkisp1-resizer.c b/drivers/staging/media/rkisp1/rkisp1-resizer.c
> > > > > > > > index 7b6b7ddd4169..8705b133de68 100644
> > > > > > > > --- a/drivers/staging/media/rkisp1/rkisp1-resizer.c
> > > > > > > > +++ b/drivers/staging/media/rkisp1/rkisp1-resizer.c
> > > > > > > > @@ -525,6 +525,7 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
> > > > > > > >        const struct rkisp1_isp_mbus_info *mbus_info;
> > > > > > > >        struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
> > > > > > > >        struct v4l2_rect *sink_crop;
> > > > > > > > +    struct v4l2_subdev_format isp_sd_fmt;
> > > > > > > > 
> > > > > > > >        sink_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SINK, which);
> > > > > > > >        src_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SRC, which);
> > > > > > > > @@ -539,8 +540,20 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
> > > > > > > >        if (which == V4L2_SUBDEV_FORMAT_ACTIVE)
> > > > > > > >                rsz->pixel_enc = mbus_info->pixel_enc;
> > > > > > > > 
> > > > > > > > +    rkisp1_get_isp_src_fmt(rsz->rkisp1, &isp_sd_fmt);
> > > > > > > > +
> > > > > > > 
> > > > > > > Is this necessary? My understanding was that in the subdev model, it was
> > > > > > > the userspace responsibility to propagate any configuration changes through
> > > > > > > the graph.
> > > > > > > 
> > > > > > > Also, doing this only here wouldn't fully maintain the
> > > > > > > consistency of the state. For example, if one sets the ISP subdev format
> > > > > > > first, then the resizer subdev and then the ISP subdev again, wouldn't the
> > > > > > > resizer subdev end up with a wrong format?
> > > > > > 
> > > > > > yes, this is indeed a bug, I am preparing v4 now.
> > > > > > What I thought to do is adding quantization conversion
> > > > > > support also on ther resizer and capture entities.
> > > > > > Then in the 'link_validation' callbacks, there
> > > > > > is a validation that the quantization fields matches.
> > > > > 
> > > > > My understanding is that, if we have the following topology
> > > > > 
> > > > > [ ISP ] -> [ Resizer ] -> [ Video node ]
> > > > > 
> > > > > then the ISP source pad would have the csc capability, while resizer
> > > > > and video node would just accept whatever is configured on their sink
> > > > > pads (no need for csc capability for that) and propagate that to their
> > > > > outputs, i.e. resizer source pad and video node CAPTURE format.
> > > > > 
> > > > > Is this what you were going to implement?
> > > Hi, I sent a v4 where the CSC capability is set on the reiszer and capture as well.
> > > I can send a v5 implementing it the way you suggest. Currently the doc says that the colorspace fields
> > > must be set by the driver for capture streams. This implies that userspace can set it
> > > only if the CSC is supported.
> > 
> > Why would the userspace have to set it for the capture stream on the
> > resizer and video nodes? Couldn't the userspace set it to DEFAULT and
> > then the driver override to whatever it received on the corresponding
> > sink?
> What do you mean "received on the corresponding sink"? Received from whom?
> 

The resizer entity has both a sink and a source, right?

The userspace is responsible for setting the right colorspace on the
resizer sink.

The driver is responsible for propagating that colorspace information to
the resizer source. Similarly, it is also responsible for updating the
state of the corresponding video node.

Best regards,
Tomasz

Re: [RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad

[Helen Koike]

On 6/18/20 4:25 PM, Tomasz Figa wrote:
> On Thu, Jun 18, 2020 at 08:54:34PM +0200, Dafna Hirschfeld wrote:
>>
>>
>> On 18.06.20 20:49, Tomasz Figa wrote:
>>> On Thu, Jun 18, 2020 at 08:26:33PM +0200, Dafna Hirschfeld wrote:
>>>>
>>>>
>>>> On 18.06.20 19:27, Tomasz Figa wrote:
>>>>> On Wed, Jun 10, 2020 at 2:08 PM Tomasz Figa <tfiga@chromium.org> wrote:
>>>>>>
>>>>>> On Thu, Jun 4, 2020 at 9:11 PM Dafna Hirschfeld
>>>>>> <dafna.hirschfeld@collabora.com> wrote:
>>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> On 04.06.20 19:54, Tomasz Figa wrote:
>>>>>>>> On Thu, Apr 16, 2020 at 04:56:05PM +0200, Dafna Hirschfeld wrote:
>>>>>>>>> The isp entity has a hardware support to force full range quantization
>>>>>>>>> for YUV formats. Use the new API to indicate userspace that
>>>>>>>>> quantization conversion is supported by adding the flag
>>>>>>>>> V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION during media code enumeration.
>>>>>>>>> Then uppon s_fmt on the video source pad, we assign the
>>>>>>>>> quantization from userspace for YUV formats.
>>>>>>>>> Also in the capture and resizer entities we retrieve the colorspace
>>>>>>>>> from the isp entity.
>>>>>>>>>
>>>>>>>>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
>>>>>>>>> ---
>>>>>>>>>     drivers/staging/media/rkisp1/rkisp1-capture.c | 28 ++++++-------
>>>>>>>>>     drivers/staging/media/rkisp1/rkisp1-common.h  |  2 +
>>>>>>>>>     drivers/staging/media/rkisp1/rkisp1-isp.c     | 39 +++++++++++++++++--
>>>>>>>>>     drivers/staging/media/rkisp1/rkisp1-resizer.c | 13 +++++++
>>>>>>>>>     4 files changed, 65 insertions(+), 17 deletions(-)
>>>>>>>>>
>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>>>>> index fbf62399fe3d..aca0f93bc772 100644
>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>>>>> @@ -1066,14 +1066,13 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
>>>>>>>>>                           const struct v4l2_format_info **fmt_info)
>>>>>>>>>     {
>>>>>>>>>        const struct rkisp1_capture_config *config = cap->config;
>>>>>>>>> -    struct rkisp1_capture *other_cap =
>>>>>>>>> -                    &cap->rkisp1->capture_devs[cap->id ^ 1];
>>>>>>>>>        const struct rkisp1_capture_fmt_cfg *fmt;
>>>>>>>>>        const struct v4l2_format_info *info;
>>>>>>>>>        const unsigned int max_widths[] = { RKISP1_RSZ_MP_SRC_MAX_WIDTH,
>>>>>>>>>                                            RKISP1_RSZ_SP_SRC_MAX_WIDTH };
>>>>>>>>>        const unsigned int max_heights[] = { RKISP1_RSZ_MP_SRC_MAX_HEIGHT,
>>>>>>>>>                                             RKISP1_RSZ_SP_SRC_MAX_HEIGHT};
>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
>>>>>>>>>
>>>>>>>>>        fmt = rkisp1_find_fmt_cfg(cap, pixm->pixelformat);
>>>>>>>>>        if (!fmt) {
>>>>>>>>> @@ -1081,24 +1080,27 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
>>>>>>>>>                pixm->pixelformat = fmt->fourcc;
>>>>>>>>>        }
>>>>>>>>>
>>>>>>>>> +    rkisp1_get_isp_src_fmt(cap->rkisp1, &isp_sd_fmt);
>>>>>>>>> +    pixm->field = isp_sd_fmt.format.field;
>>>>>>>>> +    pixm->colorspace = isp_sd_fmt.format.colorspace;
>>>>>>>>> +    pixm->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
>>>>>>>>> +    pixm->xfer_func = isp_sd_fmt.format.xfer_func;
>>>>>>>>> +
>>>>>>>>> +    /*
>>>>>>>>> +     * isp has a feature to set full range quantization for yuv formats.
>>>>>>>>
>>>>>>>> How about "select between limited and full range for YUV formats"?
>>>>>>>>
>>>>>>>>> +     * so we need to get the format from the isp.
>>>>>>>>> +     */
>>>>>>>>> +    pixm->quantization = isp_sd_fmt.format.quantization;
>>>>>>>>> +    if (!v4l2_is_format_yuv(cap->pix.info))
>>>>>>>>> +            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>> +
>>>>>>>>>        pixm->width = clamp_t(u32, pixm->width,
>>>>>>>>>                              RKISP1_RSZ_SRC_MIN_WIDTH, max_widths[cap->id]);
>>>>>>>>>        pixm->height = clamp_t(u32, pixm->height,
>>>>>>>>>                               RKISP1_RSZ_SRC_MIN_HEIGHT, max_heights[cap->id]);
>>>>>>>>>
>>>>>>>>> -    pixm->field = V4L2_FIELD_NONE;
>>>>>>>>> -    pixm->colorspace = V4L2_COLORSPACE_DEFAULT;
>>>>>>>>> -    pixm->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>>>>>>>>> -
>>>>>>>>>        info = rkisp1_fill_pixfmt(pixm, cap->id);
>>>>>>>>>
>>>>>>>>> -    /* can not change quantization when stream-on */
>>>>>>>>> -    if (other_cap->is_streaming)
>>>>>>>>> -            pixm->quantization = other_cap->pix.fmt.quantization;
>>>>>>>>> -    /* output full range by default, take effect in params */
>>>>>>>>> -    else if (!pixm->quantization ||
>>>>>>>>> -             pixm->quantization > V4L2_QUANTIZATION_LIM_RANGE)
>>>>>>>>> -            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>
>>>>>>>>>        if (fmt_cfg)
>>>>>>>>>                *fmt_cfg = fmt;
>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-common.h b/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>>>>> index 2d7b7e078636..7a5576fa14c9 100644
>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>>>>> @@ -300,6 +300,8 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
>>>>>>>>>                        struct v4l2_device *v4l2_dev);
>>>>>>>>>     void rkisp1_isp_unregister(struct rkisp1_device *rkisp1);
>>>>>>>>>
>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt);
>>>>>>>>>     const struct rkisp1_isp_mbus_info *rkisp1_isp_mbus_info_get(u32 mbus_code);
>>>>>>>>>
>>>>>>>>>     irqreturn_t rkisp1_isp_isr(struct rkisp1_device *rkisp1);
>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-isp.c b/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>>>>> index dee8e96f3900..6fdf5ed0b6b1 100644
>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>>>>> @@ -613,6 +613,10 @@ static int rkisp1_isp_enum_mbus_code(struct v4l2_subdev *sd,
>>>>>>>>>
>>>>>>>>>                if (code->index == pos - 1) {
>>>>>>>>>                        code->code = fmt->mbus_code;
>>>>>>>>> +                    if (fmt->pixel_enc == V4L2_PIXEL_ENC_YUV &&
>>>>>>>>> +                        dir == RKISP1_DIR_SRC)
>>>>>>>>> +                            code->flags =
>>>>>>>>> +                                    V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION;
>>>>>>>>>                        return 0;
>>>>>>>>>                }
>>>>>>>>>        }
>>>>>>>>> @@ -639,12 +643,21 @@ static int rkisp1_isp_init_config(struct v4l2_subdev *sd,
>>>>>>>>>        sink_crop->height = RKISP1_DEFAULT_HEIGHT;
>>>>>>>>>        sink_crop->left = 0;
>>>>>>>>>        sink_crop->top = 0;
>>>>>>>>> +    sink_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>>>>> +    sink_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(sink_fmt->colorspace);
>>>>>>>>> +    sink_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(sink_fmt->colorspace);
>>>>>>>>> +    sink_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>> +
>>>>>>>>>
>>>>>>>>>        src_fmt = v4l2_subdev_get_try_format(sd, cfg,
>>>>>>>>>                                             RKISP1_ISP_PAD_SOURCE_VIDEO);
>>>>>>>>>        *src_fmt = *sink_fmt;
>>>>>>>>>        src_fmt->code = RKISP1_DEF_SRC_PAD_FMT;
>>>>>>>>> -    src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
>>>>>>>>> +    src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
>>>>>>>>> +
>>>>>>>>>
>>>>>>>>>        src_crop = v4l2_subdev_get_try_crop(sd, cfg,
>>>>>>>>>                                            RKISP1_ISP_PAD_SOURCE_VIDEO);
>>>>>>>>> @@ -687,10 +700,17 @@ static void rkisp1_isp_set_src_fmt(struct rkisp1_isp *isp,
>>>>>>>>>                isp->src_fmt = mbus_info;
>>>>>>>>>        src_fmt->width  = src_crop->width;
>>>>>>>>>        src_fmt->height = src_crop->height;
>>>>>>>>> -    src_fmt->quantization = format->quantization;
>>>>>>>>> -    /* full range by default */
>>>>>>>>> -    if (!src_fmt->quantization)
>>>>>>>>> +
>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
>>>>>>>>> +
>>>>>>>>> +    if (mbus_info->pixel_enc == V4L2_PIXEL_ENC_BAYER)
>>>>>>>>>                src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>> +    else if (format->quantization == V4L2_QUANTIZATION_DEFAULT)
>>>>>>>>> +            src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
>>>>>>>>> +    else
>>>>>>>>> +            src_fmt->quantization = format->quantization;
>>>>>>>>>
>>>>>>>>>        *format = *src_fmt;
>>>>>>>>>     }
>>>>>>>>> @@ -1068,6 +1088,17 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
>>>>>>>>>        return ret;
>>>>>>>>>     }
>>>>>>>>>
>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt)
>>>>>>>>> +{
>>>>>>>>> +    struct rkisp1_isp *isp = &rkisp1->isp;
>>>>>>>>> +
>>>>>>>>> +    sd_fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
>>>>>>>>> +    sd_fmt->pad = RKISP1_ISP_PAD_SOURCE_VIDEO;
>>>>>>>>> +
>>>>>>>>> +    return v4l2_subdev_call(&isp->sd, pad, get_fmt, NULL, sd_fmt);
>>>>>>>>
>>>>>>>> Do we need to get through the external API to access data that is
>>>>>>>> driver-internal anyway?
>>>>>>>>
>>>>>>>>> +}
>>>>>>>>> +
>>>>>>>>>     void rkisp1_isp_unregister(struct rkisp1_device *rkisp1)
>>>>>>>>>     {
>>>>>>>>>        struct v4l2_subdev *sd = &rkisp1->isp.sd;
>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-resizer.c b/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>>>>> index 7b6b7ddd4169..8705b133de68 100644
>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>>>>> @@ -525,6 +525,7 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
>>>>>>>>>        const struct rkisp1_isp_mbus_info *mbus_info;
>>>>>>>>>        struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
>>>>>>>>>        struct v4l2_rect *sink_crop;
>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
>>>>>>>>>
>>>>>>>>>        sink_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SINK, which);
>>>>>>>>>        src_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SRC, which);
>>>>>>>>> @@ -539,8 +540,20 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
>>>>>>>>>        if (which == V4L2_SUBDEV_FORMAT_ACTIVE)
>>>>>>>>>                rsz->pixel_enc = mbus_info->pixel_enc;
>>>>>>>>>
>>>>>>>>> +    rkisp1_get_isp_src_fmt(rsz->rkisp1, &isp_sd_fmt);
>>>>>>>>> +
>>>>>>>>
>>>>>>>> Is this necessary? My understanding was that in the subdev model, it was
>>>>>>>> the userspace responsibility to propagate any configuration changes through
>>>>>>>> the graph.
>>>>>>>>
>>>>>>>> Also, doing this only here wouldn't fully maintain the
>>>>>>>> consistency of the state. For example, if one sets the ISP subdev format
>>>>>>>> first, then the resizer subdev and then the ISP subdev again, wouldn't the
>>>>>>>> resizer subdev end up with a wrong format?
>>>>>>>
>>>>>>> yes, this is indeed a bug, I am preparing v4 now.
>>>>>>> What I thought to do is adding quantization conversion
>>>>>>> support also on ther resizer and capture entities.
>>>>>>> Then in the 'link_validation' callbacks, there
>>>>>>> is a validation that the quantization fields matches.
>>>>>>
>>>>>> My understanding is that, if we have the following topology
>>>>>>
>>>>>> [ ISP ] -> [ Resizer ] -> [ Video node ]
>>>>>>
>>>>>> then the ISP source pad would have the csc capability, while resizer
>>>>>> and video node would just accept whatever is configured on their sink
>>>>>> pads (no need for csc capability for that) and propagate that to their
>>>>>> outputs, i.e. resizer source pad and video node CAPTURE format.
>>>>>>
>>>>>> Is this what you were going to implement?
>>>> Hi, I sent a v4 where the CSC capability is set on the reiszer and capture as well.
>>>> I can send a v5 implementing it the way you suggest. Currently the doc says that the colorspace fields
>>>> must be set by the driver for capture streams. This implies that userspace can set it
>>>> only if the CSC is supported.
>>>
>>> Why would the userspace have to set it for the capture stream on the
>>> resizer and video nodes? Couldn't the userspace set it to DEFAULT and
>>> then the driver override to whatever it received on the corresponding
>>> sink?

According to the documentation [1]:

"the default quantization encoding as defined by the colorspace"

Then, for instance, the default for V4L2_COLORSPACE_SRGB is limited range [2].

Unless we change the definition of DEFAULT to let it accept whatever it receives on the corresponding sink.

Then the way I understand is:

[sensor]->[isp]->[resizer]->[capture]

Userspace should set quantization on:
* sensor source pad
* isp sink pad
* resizer sink pad
* capture node

The remaining source pads would get values propagated from their sink pads.

[1] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-defs.html#c.v4l2_quantization
[2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-details.html#col-srgb

Regards,
Helen

>> What do you mean "received on the corresponding sink"? Received from whom?
>>
> 
> The resizer entity has both a sink and a source, right?
> 
> The userspace is responsible for setting the right colorspace on the
> resizer sink.
> 
> The driver is responsible for propagating that colorspace information to
> the resizer source. Similarly, it is also responsible for updating the
> state of the corresponding video node> 
> Best regards,
> Tomasz
> 

Re: [RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad

[Tomasz Figa]

On Fri, Jun 26, 2020 at 2:15 AM Helen Koike <helen.koike@collabora.com> wrote:
>
>
>
> On 6/18/20 4:25 PM, Tomasz Figa wrote:
> > On Thu, Jun 18, 2020 at 08:54:34PM +0200, Dafna Hirschfeld wrote:
> >>
> >>
> >> On 18.06.20 20:49, Tomasz Figa wrote:
> >>> On Thu, Jun 18, 2020 at 08:26:33PM +0200, Dafna Hirschfeld wrote:
> >>>>
> >>>>
> >>>> On 18.06.20 19:27, Tomasz Figa wrote:
> >>>>> On Wed, Jun 10, 2020 at 2:08 PM Tomasz Figa <tfiga@chromium.org> wrote:
> >>>>>>
> >>>>>> On Thu, Jun 4, 2020 at 9:11 PM Dafna Hirschfeld
> >>>>>> <dafna.hirschfeld@collabora.com> wrote:
> >>>>>>>
> >>>>>>> Hi
> >>>>>>>
> >>>>>>> On 04.06.20 19:54, Tomasz Figa wrote:
> >>>>>>>> On Thu, Apr 16, 2020 at 04:56:05PM +0200, Dafna Hirschfeld wrote:
> >>>>>>>>> The isp entity has a hardware support to force full range quantization
> >>>>>>>>> for YUV formats. Use the new API to indicate userspace that
> >>>>>>>>> quantization conversion is supported by adding the flag
> >>>>>>>>> V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION during media code enumeration.
> >>>>>>>>> Then uppon s_fmt on the video source pad, we assign the
> >>>>>>>>> quantization from userspace for YUV formats.
> >>>>>>>>> Also in the capture and resizer entities we retrieve the colorspace
> >>>>>>>>> from the isp entity.
> >>>>>>>>>
> >>>>>>>>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
> >>>>>>>>> ---
> >>>>>>>>>     drivers/staging/media/rkisp1/rkisp1-capture.c | 28 ++++++-------
> >>>>>>>>>     drivers/staging/media/rkisp1/rkisp1-common.h  |  2 +
> >>>>>>>>>     drivers/staging/media/rkisp1/rkisp1-isp.c     | 39 +++++++++++++++++--
> >>>>>>>>>     drivers/staging/media/rkisp1/rkisp1-resizer.c | 13 +++++++
> >>>>>>>>>     4 files changed, 65 insertions(+), 17 deletions(-)
> >>>>>>>>>
> >>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c
> >>>>>>>>> index fbf62399fe3d..aca0f93bc772 100644
> >>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-capture.c
> >>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-capture.c
> >>>>>>>>> @@ -1066,14 +1066,13 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
> >>>>>>>>>                           const struct v4l2_format_info **fmt_info)
> >>>>>>>>>     {
> >>>>>>>>>        const struct rkisp1_capture_config *config = cap->config;
> >>>>>>>>> -    struct rkisp1_capture *other_cap =
> >>>>>>>>> -                    &cap->rkisp1->capture_devs[cap->id ^ 1];
> >>>>>>>>>        const struct rkisp1_capture_fmt_cfg *fmt;
> >>>>>>>>>        const struct v4l2_format_info *info;
> >>>>>>>>>        const unsigned int max_widths[] = { RKISP1_RSZ_MP_SRC_MAX_WIDTH,
> >>>>>>>>>                                            RKISP1_RSZ_SP_SRC_MAX_WIDTH };
> >>>>>>>>>        const unsigned int max_heights[] = { RKISP1_RSZ_MP_SRC_MAX_HEIGHT,
> >>>>>>>>>                                             RKISP1_RSZ_SP_SRC_MAX_HEIGHT};
> >>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
> >>>>>>>>>
> >>>>>>>>>        fmt = rkisp1_find_fmt_cfg(cap, pixm->pixelformat);
> >>>>>>>>>        if (!fmt) {
> >>>>>>>>> @@ -1081,24 +1080,27 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
> >>>>>>>>>                pixm->pixelformat = fmt->fourcc;
> >>>>>>>>>        }
> >>>>>>>>>
> >>>>>>>>> +    rkisp1_get_isp_src_fmt(cap->rkisp1, &isp_sd_fmt);
> >>>>>>>>> +    pixm->field = isp_sd_fmt.format.field;
> >>>>>>>>> +    pixm->colorspace = isp_sd_fmt.format.colorspace;
> >>>>>>>>> +    pixm->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
> >>>>>>>>> +    pixm->xfer_func = isp_sd_fmt.format.xfer_func;
> >>>>>>>>> +
> >>>>>>>>> +    /*
> >>>>>>>>> +     * isp has a feature to set full range quantization for yuv formats.
> >>>>>>>>
> >>>>>>>> How about "select between limited and full range for YUV formats"?
> >>>>>>>>
> >>>>>>>>> +     * so we need to get the format from the isp.
> >>>>>>>>> +     */
> >>>>>>>>> +    pixm->quantization = isp_sd_fmt.format.quantization;
> >>>>>>>>> +    if (!v4l2_is_format_yuv(cap->pix.info))
> >>>>>>>>> +            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>> +
> >>>>>>>>>        pixm->width = clamp_t(u32, pixm->width,
> >>>>>>>>>                              RKISP1_RSZ_SRC_MIN_WIDTH, max_widths[cap->id]);
> >>>>>>>>>        pixm->height = clamp_t(u32, pixm->height,
> >>>>>>>>>                               RKISP1_RSZ_SRC_MIN_HEIGHT, max_heights[cap->id]);
> >>>>>>>>>
> >>>>>>>>> -    pixm->field = V4L2_FIELD_NONE;
> >>>>>>>>> -    pixm->colorspace = V4L2_COLORSPACE_DEFAULT;
> >>>>>>>>> -    pixm->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> >>>>>>>>> -
> >>>>>>>>>        info = rkisp1_fill_pixfmt(pixm, cap->id);
> >>>>>>>>>
> >>>>>>>>> -    /* can not change quantization when stream-on */
> >>>>>>>>> -    if (other_cap->is_streaming)
> >>>>>>>>> -            pixm->quantization = other_cap->pix.fmt.quantization;
> >>>>>>>>> -    /* output full range by default, take effect in params */
> >>>>>>>>> -    else if (!pixm->quantization ||
> >>>>>>>>> -             pixm->quantization > V4L2_QUANTIZATION_LIM_RANGE)
> >>>>>>>>> -            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>
> >>>>>>>>>        if (fmt_cfg)
> >>>>>>>>>                *fmt_cfg = fmt;
> >>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-common.h b/drivers/staging/media/rkisp1/rkisp1-common.h
> >>>>>>>>> index 2d7b7e078636..7a5576fa14c9 100644
> >>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-common.h
> >>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-common.h
> >>>>>>>>> @@ -300,6 +300,8 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
> >>>>>>>>>                        struct v4l2_device *v4l2_dev);
> >>>>>>>>>     void rkisp1_isp_unregister(struct rkisp1_device *rkisp1);
> >>>>>>>>>
> >>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
> >>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt);
> >>>>>>>>>     const struct rkisp1_isp_mbus_info *rkisp1_isp_mbus_info_get(u32 mbus_code);
> >>>>>>>>>
> >>>>>>>>>     irqreturn_t rkisp1_isp_isr(struct rkisp1_device *rkisp1);
> >>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-isp.c b/drivers/staging/media/rkisp1/rkisp1-isp.c
> >>>>>>>>> index dee8e96f3900..6fdf5ed0b6b1 100644
> >>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-isp.c
> >>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-isp.c
> >>>>>>>>> @@ -613,6 +613,10 @@ static int rkisp1_isp_enum_mbus_code(struct v4l2_subdev *sd,
> >>>>>>>>>
> >>>>>>>>>                if (code->index == pos - 1) {
> >>>>>>>>>                        code->code = fmt->mbus_code;
> >>>>>>>>> +                    if (fmt->pixel_enc == V4L2_PIXEL_ENC_YUV &&
> >>>>>>>>> +                        dir == RKISP1_DIR_SRC)
> >>>>>>>>> +                            code->flags =
> >>>>>>>>> +                                    V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION;
> >>>>>>>>>                        return 0;
> >>>>>>>>>                }
> >>>>>>>>>        }
> >>>>>>>>> @@ -639,12 +643,21 @@ static int rkisp1_isp_init_config(struct v4l2_subdev *sd,
> >>>>>>>>>        sink_crop->height = RKISP1_DEFAULT_HEIGHT;
> >>>>>>>>>        sink_crop->left = 0;
> >>>>>>>>>        sink_crop->top = 0;
> >>>>>>>>> +    sink_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >>>>>>>>> +    sink_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(sink_fmt->colorspace);
> >>>>>>>>> +    sink_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(sink_fmt->colorspace);
> >>>>>>>>> +    sink_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>> +
> >>>>>>>>>
> >>>>>>>>>        src_fmt = v4l2_subdev_get_try_format(sd, cfg,
> >>>>>>>>>                                             RKISP1_ISP_PAD_SOURCE_VIDEO);
> >>>>>>>>>        *src_fmt = *sink_fmt;
> >>>>>>>>>        src_fmt->code = RKISP1_DEF_SRC_PAD_FMT;
> >>>>>>>>> -    src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>> +    src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
> >>>>>>>>> +
> >>>>>>>>>
> >>>>>>>>>        src_crop = v4l2_subdev_get_try_crop(sd, cfg,
> >>>>>>>>>                                            RKISP1_ISP_PAD_SOURCE_VIDEO);
> >>>>>>>>> @@ -687,10 +700,17 @@ static void rkisp1_isp_set_src_fmt(struct rkisp1_isp *isp,
> >>>>>>>>>                isp->src_fmt = mbus_info;
> >>>>>>>>>        src_fmt->width  = src_crop->width;
> >>>>>>>>>        src_fmt->height = src_crop->height;
> >>>>>>>>> -    src_fmt->quantization = format->quantization;
> >>>>>>>>> -    /* full range by default */
> >>>>>>>>> -    if (!src_fmt->quantization)
> >>>>>>>>> +
> >>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>> +
> >>>>>>>>> +    if (mbus_info->pixel_enc == V4L2_PIXEL_ENC_BAYER)
> >>>>>>>>>                src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>> +    else if (format->quantization == V4L2_QUANTIZATION_DEFAULT)
> >>>>>>>>> +            src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
> >>>>>>>>> +    else
> >>>>>>>>> +            src_fmt->quantization = format->quantization;
> >>>>>>>>>
> >>>>>>>>>        *format = *src_fmt;
> >>>>>>>>>     }
> >>>>>>>>> @@ -1068,6 +1088,17 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
> >>>>>>>>>        return ret;
> >>>>>>>>>     }
> >>>>>>>>>
> >>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
> >>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt)
> >>>>>>>>> +{
> >>>>>>>>> +    struct rkisp1_isp *isp = &rkisp1->isp;
> >>>>>>>>> +
> >>>>>>>>> +    sd_fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
> >>>>>>>>> +    sd_fmt->pad = RKISP1_ISP_PAD_SOURCE_VIDEO;
> >>>>>>>>> +
> >>>>>>>>> +    return v4l2_subdev_call(&isp->sd, pad, get_fmt, NULL, sd_fmt);
> >>>>>>>>
> >>>>>>>> Do we need to get through the external API to access data that is
> >>>>>>>> driver-internal anyway?
> >>>>>>>>
> >>>>>>>>> +}
> >>>>>>>>> +
> >>>>>>>>>     void rkisp1_isp_unregister(struct rkisp1_device *rkisp1)
> >>>>>>>>>     {
> >>>>>>>>>        struct v4l2_subdev *sd = &rkisp1->isp.sd;
> >>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-resizer.c b/drivers/staging/media/rkisp1/rkisp1-resizer.c
> >>>>>>>>> index 7b6b7ddd4169..8705b133de68 100644
> >>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-resizer.c
> >>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-resizer.c
> >>>>>>>>> @@ -525,6 +525,7 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
> >>>>>>>>>        const struct rkisp1_isp_mbus_info *mbus_info;
> >>>>>>>>>        struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
> >>>>>>>>>        struct v4l2_rect *sink_crop;
> >>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
> >>>>>>>>>
> >>>>>>>>>        sink_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SINK, which);
> >>>>>>>>>        src_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SRC, which);
> >>>>>>>>> @@ -539,8 +540,20 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
> >>>>>>>>>        if (which == V4L2_SUBDEV_FORMAT_ACTIVE)
> >>>>>>>>>                rsz->pixel_enc = mbus_info->pixel_enc;
> >>>>>>>>>
> >>>>>>>>> +    rkisp1_get_isp_src_fmt(rsz->rkisp1, &isp_sd_fmt);
> >>>>>>>>> +
> >>>>>>>>
> >>>>>>>> Is this necessary? My understanding was that in the subdev model, it was
> >>>>>>>> the userspace responsibility to propagate any configuration changes through
> >>>>>>>> the graph.
> >>>>>>>>
> >>>>>>>> Also, doing this only here wouldn't fully maintain the
> >>>>>>>> consistency of the state. For example, if one sets the ISP subdev format
> >>>>>>>> first, then the resizer subdev and then the ISP subdev again, wouldn't the
> >>>>>>>> resizer subdev end up with a wrong format?
> >>>>>>>
> >>>>>>> yes, this is indeed a bug, I am preparing v4 now.
> >>>>>>> What I thought to do is adding quantization conversion
> >>>>>>> support also on ther resizer and capture entities.
> >>>>>>> Then in the 'link_validation' callbacks, there
> >>>>>>> is a validation that the quantization fields matches.
> >>>>>>
> >>>>>> My understanding is that, if we have the following topology
> >>>>>>
> >>>>>> [ ISP ] -> [ Resizer ] -> [ Video node ]
> >>>>>>
> >>>>>> then the ISP source pad would have the csc capability, while resizer
> >>>>>> and video node would just accept whatever is configured on their sink
> >>>>>> pads (no need for csc capability for that) and propagate that to their
> >>>>>> outputs, i.e. resizer source pad and video node CAPTURE format.
> >>>>>>
> >>>>>> Is this what you were going to implement?
> >>>> Hi, I sent a v4 where the CSC capability is set on the reiszer and capture as well.
> >>>> I can send a v5 implementing it the way you suggest. Currently the doc says that the colorspace fields
> >>>> must be set by the driver for capture streams. This implies that userspace can set it
> >>>> only if the CSC is supported.
> >>>
> >>> Why would the userspace have to set it for the capture stream on the
> >>> resizer and video nodes? Couldn't the userspace set it to DEFAULT and
> >>> then the driver override to whatever it received on the corresponding
> >>> sink?
>
> According to the documentation [1]:
>
> "the default quantization encoding as defined by the colorspace"
>
> Then, for instance, the default for V4L2_COLORSPACE_SRGB is limited range [2].
>
> Unless we change the definition of DEFAULT to let it accept whatever it receives on the corresponding sink.
>
> Then the way I understand is:
>
> [sensor]->[isp]->[resizer]->[capture]
>
> Userspace should set quantization on:
> * sensor source pad
> * isp sink pad
> * resizer sink pad
> * capture node
>
> The remaining source pads would get values propagated from their sink pads.
>
> [1] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-defs.html#c.v4l2_quantization
> [2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-details.html#col-srgb

After discussing this offline, we concluded that:
- only the source pad of the ISP entity should expose the CSC capability,
- the CAPTURE video node should expose only the formats/color spaces
compatible with the current setting of the resizer source,
- consistency between the resizer source format and CAPTURE format
should be implicitly maintained by the driver.

Below is the rationale for that:
- In the MC model, the video node represents the DMA alone (and thus
the reason to have the resizer entities at all) and since the
colorspace is not a property of the DMA, it doesn't belong there.
- Even if we put that aside, both MP and SP can only output the same
colorspace, which would create a dependency between the configuration
of the two video nodes, which is not something well defined by V4L2.
- A video interface in the MC API doesn't expose control over its
input format, which means that the relation between the other side of
the link (resizer source) and the video node is implicit and needs to
be handled internally in the driver.
- V4L2, as opposed to MC, requires that the current state is always
consistent and valid. In this case it means that an ioctl on the video
node must not be able to alter the state in a way that would make it
inconsistent with the state of the matching resizer source.

Best regards,
Tomasz

Re: [RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad

[Laurent Pinchart]

Hi Tomasz,

On Sun, Jun 28, 2020 at 03:10:41PM +0200, Tomasz Figa wrote:
> On Fri, Jun 26, 2020 at 2:15 AM Helen Koike <helen.koike@collabora.com> wrote:
> > On 6/18/20 4:25 PM, Tomasz Figa wrote:
> > > On Thu, Jun 18, 2020 at 08:54:34PM +0200, Dafna Hirschfeld wrote:
> > >> On 18.06.20 20:49, Tomasz Figa wrote:
> > >>> On Thu, Jun 18, 2020 at 08:26:33PM +0200, Dafna Hirschfeld wrote:
> > >>>> On 18.06.20 19:27, Tomasz Figa wrote:
> > >>>>> On Wed, Jun 10, 2020 at 2:08 PM Tomasz Figa wrote:
> > >>>>>> On Thu, Jun 4, 2020 at 9:11 PM Dafna Hirschfeld wrote:
> > >>>>>>> On 04.06.20 19:54, Tomasz Figa wrote:
> > >>>>>>>> On Thu, Apr 16, 2020 at 04:56:05PM +0200, Dafna Hirschfeld wrote:
> > >>>>>>>>> The isp entity has a hardware support to force full range quantization
> > >>>>>>>>> for YUV formats. Use the new API to indicate userspace that
> > >>>>>>>>> quantization conversion is supported by adding the flag
> > >>>>>>>>> V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION during media code enumeration.
> > >>>>>>>>> Then uppon s_fmt on the video source pad, we assign the
> > >>>>>>>>> quantization from userspace for YUV formats.
> > >>>>>>>>> Also in the capture and resizer entities we retrieve the colorspace
> > >>>>>>>>> from the isp entity.
> > >>>>>>>>>
> > >>>>>>>>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
> > >>>>>>>>> ---
> > >>>>>>>>>     drivers/staging/media/rkisp1/rkisp1-capture.c | 28 ++++++-------
> > >>>>>>>>>     drivers/staging/media/rkisp1/rkisp1-common.h  |  2 +
> > >>>>>>>>>     drivers/staging/media/rkisp1/rkisp1-isp.c     | 39 +++++++++++++++++--
> > >>>>>>>>>     drivers/staging/media/rkisp1/rkisp1-resizer.c | 13 +++++++
> > >>>>>>>>>     4 files changed, 65 insertions(+), 17 deletions(-)
> > >>>>>>>>>
> > >>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c
> > >>>>>>>>> index fbf62399fe3d..aca0f93bc772 100644
> > >>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-capture.c
> > >>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-capture.c
> > >>>>>>>>> @@ -1066,14 +1066,13 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
> > >>>>>>>>>                           const struct v4l2_format_info **fmt_info)
> > >>>>>>>>>     {
> > >>>>>>>>>        const struct rkisp1_capture_config *config = cap->config;
> > >>>>>>>>> -    struct rkisp1_capture *other_cap =
> > >>>>>>>>> -                    &cap->rkisp1->capture_devs[cap->id ^ 1];
> > >>>>>>>>>        const struct rkisp1_capture_fmt_cfg *fmt;
> > >>>>>>>>>        const struct v4l2_format_info *info;
> > >>>>>>>>>        const unsigned int max_widths[] = { RKISP1_RSZ_MP_SRC_MAX_WIDTH,
> > >>>>>>>>>                                            RKISP1_RSZ_SP_SRC_MAX_WIDTH };
> > >>>>>>>>>        const unsigned int max_heights[] = { RKISP1_RSZ_MP_SRC_MAX_HEIGHT,
> > >>>>>>>>>                                             RKISP1_RSZ_SP_SRC_MAX_HEIGHT};
> > >>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
> > >>>>>>>>>
> > >>>>>>>>>        fmt = rkisp1_find_fmt_cfg(cap, pixm->pixelformat);
> > >>>>>>>>>        if (!fmt) {
> > >>>>>>>>> @@ -1081,24 +1080,27 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
> > >>>>>>>>>                pixm->pixelformat = fmt->fourcc;
> > >>>>>>>>>        }
> > >>>>>>>>>
> > >>>>>>>>> +    rkisp1_get_isp_src_fmt(cap->rkisp1, &isp_sd_fmt);
> > >>>>>>>>> +    pixm->field = isp_sd_fmt.format.field;
> > >>>>>>>>> +    pixm->colorspace = isp_sd_fmt.format.colorspace;
> > >>>>>>>>> +    pixm->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
> > >>>>>>>>> +    pixm->xfer_func = isp_sd_fmt.format.xfer_func;
> > >>>>>>>>> +
> > >>>>>>>>> +    /*
> > >>>>>>>>> +     * isp has a feature to set full range quantization for yuv formats.
> > >>>>>>>>
> > >>>>>>>> How about "select between limited and full range for YUV formats"?
> > >>>>>>>>
> > >>>>>>>>> +     * so we need to get the format from the isp.
> > >>>>>>>>> +     */
> > >>>>>>>>> +    pixm->quantization = isp_sd_fmt.format.quantization;
> > >>>>>>>>> +    if (!v4l2_is_format_yuv(cap->pix.info))
> > >>>>>>>>> +            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> > >>>>>>>>> +
> > >>>>>>>>>        pixm->width = clamp_t(u32, pixm->width,
> > >>>>>>>>>                              RKISP1_RSZ_SRC_MIN_WIDTH, max_widths[cap->id]);
> > >>>>>>>>>        pixm->height = clamp_t(u32, pixm->height,
> > >>>>>>>>>                               RKISP1_RSZ_SRC_MIN_HEIGHT, max_heights[cap->id]);
> > >>>>>>>>>
> > >>>>>>>>> -    pixm->field = V4L2_FIELD_NONE;
> > >>>>>>>>> -    pixm->colorspace = V4L2_COLORSPACE_DEFAULT;
> > >>>>>>>>> -    pixm->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> > >>>>>>>>> -
> > >>>>>>>>>        info = rkisp1_fill_pixfmt(pixm, cap->id);
> > >>>>>>>>>
> > >>>>>>>>> -    /* can not change quantization when stream-on */
> > >>>>>>>>> -    if (other_cap->is_streaming)
> > >>>>>>>>> -            pixm->quantization = other_cap->pix.fmt.quantization;
> > >>>>>>>>> -    /* output full range by default, take effect in params */
> > >>>>>>>>> -    else if (!pixm->quantization ||
> > >>>>>>>>> -             pixm->quantization > V4L2_QUANTIZATION_LIM_RANGE)
> > >>>>>>>>> -            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> > >>>>>>>>>
> > >>>>>>>>>        if (fmt_cfg)
> > >>>>>>>>>                *fmt_cfg = fmt;
> > >>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-common.h b/drivers/staging/media/rkisp1/rkisp1-common.h
> > >>>>>>>>> index 2d7b7e078636..7a5576fa14c9 100644
> > >>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-common.h
> > >>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-common.h
> > >>>>>>>>> @@ -300,6 +300,8 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
> > >>>>>>>>>                        struct v4l2_device *v4l2_dev);
> > >>>>>>>>>     void rkisp1_isp_unregister(struct rkisp1_device *rkisp1);
> > >>>>>>>>>
> > >>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
> > >>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt);
> > >>>>>>>>>     const struct rkisp1_isp_mbus_info *rkisp1_isp_mbus_info_get(u32 mbus_code);
> > >>>>>>>>>
> > >>>>>>>>>     irqreturn_t rkisp1_isp_isr(struct rkisp1_device *rkisp1);
> > >>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-isp.c b/drivers/staging/media/rkisp1/rkisp1-isp.c
> > >>>>>>>>> index dee8e96f3900..6fdf5ed0b6b1 100644
> > >>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-isp.c
> > >>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-isp.c
> > >>>>>>>>> @@ -613,6 +613,10 @@ static int rkisp1_isp_enum_mbus_code(struct v4l2_subdev *sd,
> > >>>>>>>>>
> > >>>>>>>>>                if (code->index == pos - 1) {
> > >>>>>>>>>                        code->code = fmt->mbus_code;
> > >>>>>>>>> +                    if (fmt->pixel_enc == V4L2_PIXEL_ENC_YUV &&
> > >>>>>>>>> +                        dir == RKISP1_DIR_SRC)
> > >>>>>>>>> +                            code->flags =
> > >>>>>>>>> +                                    V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION;
> > >>>>>>>>>                        return 0;
> > >>>>>>>>>                }
> > >>>>>>>>>        }
> > >>>>>>>>> @@ -639,12 +643,21 @@ static int rkisp1_isp_init_config(struct v4l2_subdev *sd,
> > >>>>>>>>>        sink_crop->height = RKISP1_DEFAULT_HEIGHT;
> > >>>>>>>>>        sink_crop->left = 0;
> > >>>>>>>>>        sink_crop->top = 0;
> > >>>>>>>>> +    sink_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> > >>>>>>>>> +    sink_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(sink_fmt->colorspace);
> > >>>>>>>>> +    sink_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(sink_fmt->colorspace);
> > >>>>>>>>> +    sink_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> > >>>>>>>>> +
> > >>>>>>>>>
> > >>>>>>>>>        src_fmt = v4l2_subdev_get_try_format(sd, cfg,
> > >>>>>>>>>                                             RKISP1_ISP_PAD_SOURCE_VIDEO);
> > >>>>>>>>>        *src_fmt = *sink_fmt;
> > >>>>>>>>>        src_fmt->code = RKISP1_DEF_SRC_PAD_FMT;
> > >>>>>>>>> -    src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> > >>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> > >>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
> > >>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
> > >>>>>>>>> +    src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
> > >>>>>>>>> +
> > >>>>>>>>>
> > >>>>>>>>>        src_crop = v4l2_subdev_get_try_crop(sd, cfg,
> > >>>>>>>>>                                            RKISP1_ISP_PAD_SOURCE_VIDEO);
> > >>>>>>>>> @@ -687,10 +700,17 @@ static void rkisp1_isp_set_src_fmt(struct rkisp1_isp *isp,
> > >>>>>>>>>                isp->src_fmt = mbus_info;
> > >>>>>>>>>        src_fmt->width  = src_crop->width;
> > >>>>>>>>>        src_fmt->height = src_crop->height;
> > >>>>>>>>> -    src_fmt->quantization = format->quantization;
> > >>>>>>>>> -    /* full range by default */
> > >>>>>>>>> -    if (!src_fmt->quantization)
> > >>>>>>>>> +
> > >>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> > >>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
> > >>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
> > >>>>>>>>> +
> > >>>>>>>>> +    if (mbus_info->pixel_enc == V4L2_PIXEL_ENC_BAYER)
> > >>>>>>>>>                src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> > >>>>>>>>> +    else if (format->quantization == V4L2_QUANTIZATION_DEFAULT)
> > >>>>>>>>> +            src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
> > >>>>>>>>> +    else
> > >>>>>>>>> +            src_fmt->quantization = format->quantization;
> > >>>>>>>>>
> > >>>>>>>>>        *format = *src_fmt;
> > >>>>>>>>>     }
> > >>>>>>>>> @@ -1068,6 +1088,17 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
> > >>>>>>>>>        return ret;
> > >>>>>>>>>     }
> > >>>>>>>>>
> > >>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
> > >>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt)
> > >>>>>>>>> +{
> > >>>>>>>>> +    struct rkisp1_isp *isp = &rkisp1->isp;
> > >>>>>>>>> +
> > >>>>>>>>> +    sd_fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
> > >>>>>>>>> +    sd_fmt->pad = RKISP1_ISP_PAD_SOURCE_VIDEO;
> > >>>>>>>>> +
> > >>>>>>>>> +    return v4l2_subdev_call(&isp->sd, pad, get_fmt, NULL, sd_fmt);
> > >>>>>>>>
> > >>>>>>>> Do we need to get through the external API to access data that is
> > >>>>>>>> driver-internal anyway?
> > >>>>>>>>
> > >>>>>>>>> +}
> > >>>>>>>>> +
> > >>>>>>>>>     void rkisp1_isp_unregister(struct rkisp1_device *rkisp1)
> > >>>>>>>>>     {
> > >>>>>>>>>        struct v4l2_subdev *sd = &rkisp1->isp.sd;
> > >>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-resizer.c b/drivers/staging/media/rkisp1/rkisp1-resizer.c
> > >>>>>>>>> index 7b6b7ddd4169..8705b133de68 100644
> > >>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-resizer.c
> > >>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-resizer.c
> > >>>>>>>>> @@ -525,6 +525,7 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
> > >>>>>>>>>        const struct rkisp1_isp_mbus_info *mbus_info;
> > >>>>>>>>>        struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
> > >>>>>>>>>        struct v4l2_rect *sink_crop;
> > >>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
> > >>>>>>>>>
> > >>>>>>>>>        sink_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SINK, which);
> > >>>>>>>>>        src_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SRC, which);
> > >>>>>>>>> @@ -539,8 +540,20 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
> > >>>>>>>>>        if (which == V4L2_SUBDEV_FORMAT_ACTIVE)
> > >>>>>>>>>                rsz->pixel_enc = mbus_info->pixel_enc;
> > >>>>>>>>>
> > >>>>>>>>> +    rkisp1_get_isp_src_fmt(rsz->rkisp1, &isp_sd_fmt);
> > >>>>>>>>> +
> > >>>>>>>>
> > >>>>>>>> Is this necessary? My understanding was that in the subdev model, it was
> > >>>>>>>> the userspace responsibility to propagate any configuration changes through
> > >>>>>>>> the graph.
> > >>>>>>>>
> > >>>>>>>> Also, doing this only here wouldn't fully maintain the
> > >>>>>>>> consistency of the state. For example, if one sets the ISP subdev format
> > >>>>>>>> first, then the resizer subdev and then the ISP subdev again, wouldn't the
> > >>>>>>>> resizer subdev end up with a wrong format?
> > >>>>>>>
> > >>>>>>> yes, this is indeed a bug, I am preparing v4 now.
> > >>>>>>> What I thought to do is adding quantization conversion
> > >>>>>>> support also on ther resizer and capture entities.
> > >>>>>>> Then in the 'link_validation' callbacks, there
> > >>>>>>> is a validation that the quantization fields matches.
> > >>>>>>
> > >>>>>> My understanding is that, if we have the following topology
> > >>>>>>
> > >>>>>> [ ISP ] -> [ Resizer ] -> [ Video node ]
> > >>>>>>
> > >>>>>> then the ISP source pad would have the csc capability, while resizer
> > >>>>>> and video node would just accept whatever is configured on their sink
> > >>>>>> pads (no need for csc capability for that) and propagate that to their
> > >>>>>> outputs, i.e. resizer source pad and video node CAPTURE format.
> > >>>>>>
> > >>>>>> Is this what you were going to implement?
> > >>>> Hi, I sent a v4 where the CSC capability is set on the reiszer and capture as well.
> > >>>> I can send a v5 implementing it the way you suggest. Currently the doc says that the colorspace fields
> > >>>> must be set by the driver for capture streams. This implies that userspace can set it
> > >>>> only if the CSC is supported.
> > >>>
> > >>> Why would the userspace have to set it for the capture stream on the
> > >>> resizer and video nodes? Couldn't the userspace set it to DEFAULT and
> > >>> then the driver override to whatever it received on the corresponding
> > >>> sink?
> >
> > According to the documentation [1]:
> >
> > "the default quantization encoding as defined by the colorspace"
> >
> > Then, for instance, the default for V4L2_COLORSPACE_SRGB is limited range [2].
> >
> > Unless we change the definition of DEFAULT to let it accept whatever it receives on the corresponding sink.
> >
> > Then the way I understand is:
> >
> > [sensor]->[isp]->[resizer]->[capture]
> >
> > Userspace should set quantization on:
> > * sensor source pad
> > * isp sink pad
> > * resizer sink pad
> > * capture node
> >
> > The remaining source pads would get values propagated from their sink pads.
> >
> > [1] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-defs.html#c.v4l2_quantization
> > [2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-details.html#col-srgb
> 
> After discussing this offline, we concluded that:
> - only the source pad of the ISP entity should expose the CSC capability,
> - the CAPTURE video node should expose only the formats/color spaces
> compatible with the current setting of the resizer source,
> - consistency between the resizer source format and CAPTURE format
> should be implicitly maintained by the driver.

Do you mean it should be validated at streamon time ? In MC-based
drivers, entities are isolated from each other from a configuration
point of view, and the validity of the configuration across the whole
pipeline is checked at streamon time. I don't think CSC parameters
should be treated differently than formats in this regard.

> Below is the rationale for that:
> - In the MC model, the video node represents the DMA alone (and thus
> the reason to have the resizer entities at all) and since the
> colorspace is not a property of the DMA, it doesn't belong there.
> - Even if we put that aside, both MP and SP can only output the same
> colorspace, which would create a dependency between the configuration
> of the two video nodes, which is not something well defined by V4L2.
> - A video interface in the MC API doesn't expose control over its
> input format, which means that the relation between the other side of
> the link (resizer source) and the video node is implicit and needs to
> be handled internally in the driver.
> - V4L2, as opposed to MC, requires that the current state is always
> consistent and valid. In this case it means that an ioctl on the video
> node must not be able to alter the state in a way that would make it
> inconsistent with the state of the matching resizer source.

-- 
Regards,

Laurent Pinchart

Re: [RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad

[Tomasz Figa]

On Sun, Jun 28, 2020 at 3:15 PM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> Hi Tomasz,
>
> On Sun, Jun 28, 2020 at 03:10:41PM +0200, Tomasz Figa wrote:
> > On Fri, Jun 26, 2020 at 2:15 AM Helen Koike <helen.koike@collabora.com> wrote:
> > > On 6/18/20 4:25 PM, Tomasz Figa wrote:
> > > > On Thu, Jun 18, 2020 at 08:54:34PM +0200, Dafna Hirschfeld wrote:
> > > >> On 18.06.20 20:49, Tomasz Figa wrote:
> > > >>> On Thu, Jun 18, 2020 at 08:26:33PM +0200, Dafna Hirschfeld wrote:
> > > >>>> On 18.06.20 19:27, Tomasz Figa wrote:
> > > >>>>> On Wed, Jun 10, 2020 at 2:08 PM Tomasz Figa wrote:
> > > >>>>>> On Thu, Jun 4, 2020 at 9:11 PM Dafna Hirschfeld wrote:
> > > >>>>>>> On 04.06.20 19:54, Tomasz Figa wrote:
> > > >>>>>>>> On Thu, Apr 16, 2020 at 04:56:05PM +0200, Dafna Hirschfeld wrote:
> > > >>>>>>>>> The isp entity has a hardware support to force full range quantization
> > > >>>>>>>>> for YUV formats. Use the new API to indicate userspace that
> > > >>>>>>>>> quantization conversion is supported by adding the flag
> > > >>>>>>>>> V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION during media code enumeration.
> > > >>>>>>>>> Then uppon s_fmt on the video source pad, we assign the
> > > >>>>>>>>> quantization from userspace for YUV formats.
> > > >>>>>>>>> Also in the capture and resizer entities we retrieve the colorspace
> > > >>>>>>>>> from the isp entity.
> > > >>>>>>>>>
> > > >>>>>>>>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
> > > >>>>>>>>> ---
> > > >>>>>>>>>     drivers/staging/media/rkisp1/rkisp1-capture.c | 28 ++++++-------
> > > >>>>>>>>>     drivers/staging/media/rkisp1/rkisp1-common.h  |  2 +
> > > >>>>>>>>>     drivers/staging/media/rkisp1/rkisp1-isp.c     | 39 +++++++++++++++++--
> > > >>>>>>>>>     drivers/staging/media/rkisp1/rkisp1-resizer.c | 13 +++++++
> > > >>>>>>>>>     4 files changed, 65 insertions(+), 17 deletions(-)
> > > >>>>>>>>>
> > > >>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c
> > > >>>>>>>>> index fbf62399fe3d..aca0f93bc772 100644
> > > >>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-capture.c
> > > >>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-capture.c
> > > >>>>>>>>> @@ -1066,14 +1066,13 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
> > > >>>>>>>>>                           const struct v4l2_format_info **fmt_info)
> > > >>>>>>>>>     {
> > > >>>>>>>>>        const struct rkisp1_capture_config *config = cap->config;
> > > >>>>>>>>> -    struct rkisp1_capture *other_cap =
> > > >>>>>>>>> -                    &cap->rkisp1->capture_devs[cap->id ^ 1];
> > > >>>>>>>>>        const struct rkisp1_capture_fmt_cfg *fmt;
> > > >>>>>>>>>        const struct v4l2_format_info *info;
> > > >>>>>>>>>        const unsigned int max_widths[] = { RKISP1_RSZ_MP_SRC_MAX_WIDTH,
> > > >>>>>>>>>                                            RKISP1_RSZ_SP_SRC_MAX_WIDTH };
> > > >>>>>>>>>        const unsigned int max_heights[] = { RKISP1_RSZ_MP_SRC_MAX_HEIGHT,
> > > >>>>>>>>>                                             RKISP1_RSZ_SP_SRC_MAX_HEIGHT};
> > > >>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
> > > >>>>>>>>>
> > > >>>>>>>>>        fmt = rkisp1_find_fmt_cfg(cap, pixm->pixelformat);
> > > >>>>>>>>>        if (!fmt) {
> > > >>>>>>>>> @@ -1081,24 +1080,27 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
> > > >>>>>>>>>                pixm->pixelformat = fmt->fourcc;
> > > >>>>>>>>>        }
> > > >>>>>>>>>
> > > >>>>>>>>> +    rkisp1_get_isp_src_fmt(cap->rkisp1, &isp_sd_fmt);
> > > >>>>>>>>> +    pixm->field = isp_sd_fmt.format.field;
> > > >>>>>>>>> +    pixm->colorspace = isp_sd_fmt.format.colorspace;
> > > >>>>>>>>> +    pixm->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
> > > >>>>>>>>> +    pixm->xfer_func = isp_sd_fmt.format.xfer_func;
> > > >>>>>>>>> +
> > > >>>>>>>>> +    /*
> > > >>>>>>>>> +     * isp has a feature to set full range quantization for yuv formats.
> > > >>>>>>>>
> > > >>>>>>>> How about "select between limited and full range for YUV formats"?
> > > >>>>>>>>
> > > >>>>>>>>> +     * so we need to get the format from the isp.
> > > >>>>>>>>> +     */
> > > >>>>>>>>> +    pixm->quantization = isp_sd_fmt.format.quantization;
> > > >>>>>>>>> +    if (!v4l2_is_format_yuv(cap->pix.info))
> > > >>>>>>>>> +            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> > > >>>>>>>>> +
> > > >>>>>>>>>        pixm->width = clamp_t(u32, pixm->width,
> > > >>>>>>>>>                              RKISP1_RSZ_SRC_MIN_WIDTH, max_widths[cap->id]);
> > > >>>>>>>>>        pixm->height = clamp_t(u32, pixm->height,
> > > >>>>>>>>>                               RKISP1_RSZ_SRC_MIN_HEIGHT, max_heights[cap->id]);
> > > >>>>>>>>>
> > > >>>>>>>>> -    pixm->field = V4L2_FIELD_NONE;
> > > >>>>>>>>> -    pixm->colorspace = V4L2_COLORSPACE_DEFAULT;
> > > >>>>>>>>> -    pixm->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> > > >>>>>>>>> -
> > > >>>>>>>>>        info = rkisp1_fill_pixfmt(pixm, cap->id);
> > > >>>>>>>>>
> > > >>>>>>>>> -    /* can not change quantization when stream-on */
> > > >>>>>>>>> -    if (other_cap->is_streaming)
> > > >>>>>>>>> -            pixm->quantization = other_cap->pix.fmt.quantization;
> > > >>>>>>>>> -    /* output full range by default, take effect in params */
> > > >>>>>>>>> -    else if (!pixm->quantization ||
> > > >>>>>>>>> -             pixm->quantization > V4L2_QUANTIZATION_LIM_RANGE)
> > > >>>>>>>>> -            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> > > >>>>>>>>>
> > > >>>>>>>>>        if (fmt_cfg)
> > > >>>>>>>>>                *fmt_cfg = fmt;
> > > >>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-common.h b/drivers/staging/media/rkisp1/rkisp1-common.h
> > > >>>>>>>>> index 2d7b7e078636..7a5576fa14c9 100644
> > > >>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-common.h
> > > >>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-common.h
> > > >>>>>>>>> @@ -300,6 +300,8 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
> > > >>>>>>>>>                        struct v4l2_device *v4l2_dev);
> > > >>>>>>>>>     void rkisp1_isp_unregister(struct rkisp1_device *rkisp1);
> > > >>>>>>>>>
> > > >>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
> > > >>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt);
> > > >>>>>>>>>     const struct rkisp1_isp_mbus_info *rkisp1_isp_mbus_info_get(u32 mbus_code);
> > > >>>>>>>>>
> > > >>>>>>>>>     irqreturn_t rkisp1_isp_isr(struct rkisp1_device *rkisp1);
> > > >>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-isp.c b/drivers/staging/media/rkisp1/rkisp1-isp.c
> > > >>>>>>>>> index dee8e96f3900..6fdf5ed0b6b1 100644
> > > >>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-isp.c
> > > >>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-isp.c
> > > >>>>>>>>> @@ -613,6 +613,10 @@ static int rkisp1_isp_enum_mbus_code(struct v4l2_subdev *sd,
> > > >>>>>>>>>
> > > >>>>>>>>>                if (code->index == pos - 1) {
> > > >>>>>>>>>                        code->code = fmt->mbus_code;
> > > >>>>>>>>> +                    if (fmt->pixel_enc == V4L2_PIXEL_ENC_YUV &&
> > > >>>>>>>>> +                        dir == RKISP1_DIR_SRC)
> > > >>>>>>>>> +                            code->flags =
> > > >>>>>>>>> +                                    V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION;
> > > >>>>>>>>>                        return 0;
> > > >>>>>>>>>                }
> > > >>>>>>>>>        }
> > > >>>>>>>>> @@ -639,12 +643,21 @@ static int rkisp1_isp_init_config(struct v4l2_subdev *sd,
> > > >>>>>>>>>        sink_crop->height = RKISP1_DEFAULT_HEIGHT;
> > > >>>>>>>>>        sink_crop->left = 0;
> > > >>>>>>>>>        sink_crop->top = 0;
> > > >>>>>>>>> +    sink_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> > > >>>>>>>>> +    sink_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(sink_fmt->colorspace);
> > > >>>>>>>>> +    sink_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(sink_fmt->colorspace);
> > > >>>>>>>>> +    sink_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> > > >>>>>>>>> +
> > > >>>>>>>>>
> > > >>>>>>>>>        src_fmt = v4l2_subdev_get_try_format(sd, cfg,
> > > >>>>>>>>>                                             RKISP1_ISP_PAD_SOURCE_VIDEO);
> > > >>>>>>>>>        *src_fmt = *sink_fmt;
> > > >>>>>>>>>        src_fmt->code = RKISP1_DEF_SRC_PAD_FMT;
> > > >>>>>>>>> -    src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> > > >>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> > > >>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
> > > >>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
> > > >>>>>>>>> +    src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
> > > >>>>>>>>> +
> > > >>>>>>>>>
> > > >>>>>>>>>        src_crop = v4l2_subdev_get_try_crop(sd, cfg,
> > > >>>>>>>>>                                            RKISP1_ISP_PAD_SOURCE_VIDEO);
> > > >>>>>>>>> @@ -687,10 +700,17 @@ static void rkisp1_isp_set_src_fmt(struct rkisp1_isp *isp,
> > > >>>>>>>>>                isp->src_fmt = mbus_info;
> > > >>>>>>>>>        src_fmt->width  = src_crop->width;
> > > >>>>>>>>>        src_fmt->height = src_crop->height;
> > > >>>>>>>>> -    src_fmt->quantization = format->quantization;
> > > >>>>>>>>> -    /* full range by default */
> > > >>>>>>>>> -    if (!src_fmt->quantization)
> > > >>>>>>>>> +
> > > >>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> > > >>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
> > > >>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
> > > >>>>>>>>> +
> > > >>>>>>>>> +    if (mbus_info->pixel_enc == V4L2_PIXEL_ENC_BAYER)
> > > >>>>>>>>>                src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> > > >>>>>>>>> +    else if (format->quantization == V4L2_QUANTIZATION_DEFAULT)
> > > >>>>>>>>> +            src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
> > > >>>>>>>>> +    else
> > > >>>>>>>>> +            src_fmt->quantization = format->quantization;
> > > >>>>>>>>>
> > > >>>>>>>>>        *format = *src_fmt;
> > > >>>>>>>>>     }
> > > >>>>>>>>> @@ -1068,6 +1088,17 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
> > > >>>>>>>>>        return ret;
> > > >>>>>>>>>     }
> > > >>>>>>>>>
> > > >>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
> > > >>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt)
> > > >>>>>>>>> +{
> > > >>>>>>>>> +    struct rkisp1_isp *isp = &rkisp1->isp;
> > > >>>>>>>>> +
> > > >>>>>>>>> +    sd_fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
> > > >>>>>>>>> +    sd_fmt->pad = RKISP1_ISP_PAD_SOURCE_VIDEO;
> > > >>>>>>>>> +
> > > >>>>>>>>> +    return v4l2_subdev_call(&isp->sd, pad, get_fmt, NULL, sd_fmt);
> > > >>>>>>>>
> > > >>>>>>>> Do we need to get through the external API to access data that is
> > > >>>>>>>> driver-internal anyway?
> > > >>>>>>>>
> > > >>>>>>>>> +}
> > > >>>>>>>>> +
> > > >>>>>>>>>     void rkisp1_isp_unregister(struct rkisp1_device *rkisp1)
> > > >>>>>>>>>     {
> > > >>>>>>>>>        struct v4l2_subdev *sd = &rkisp1->isp.sd;
> > > >>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-resizer.c b/drivers/staging/media/rkisp1/rkisp1-resizer.c
> > > >>>>>>>>> index 7b6b7ddd4169..8705b133de68 100644
> > > >>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-resizer.c
> > > >>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-resizer.c
> > > >>>>>>>>> @@ -525,6 +525,7 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
> > > >>>>>>>>>        const struct rkisp1_isp_mbus_info *mbus_info;
> > > >>>>>>>>>        struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
> > > >>>>>>>>>        struct v4l2_rect *sink_crop;
> > > >>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
> > > >>>>>>>>>
> > > >>>>>>>>>        sink_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SINK, which);
> > > >>>>>>>>>        src_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SRC, which);
> > > >>>>>>>>> @@ -539,8 +540,20 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
> > > >>>>>>>>>        if (which == V4L2_SUBDEV_FORMAT_ACTIVE)
> > > >>>>>>>>>                rsz->pixel_enc = mbus_info->pixel_enc;
> > > >>>>>>>>>
> > > >>>>>>>>> +    rkisp1_get_isp_src_fmt(rsz->rkisp1, &isp_sd_fmt);
> > > >>>>>>>>> +
> > > >>>>>>>>
> > > >>>>>>>> Is this necessary? My understanding was that in the subdev model, it was
> > > >>>>>>>> the userspace responsibility to propagate any configuration changes through
> > > >>>>>>>> the graph.
> > > >>>>>>>>
> > > >>>>>>>> Also, doing this only here wouldn't fully maintain the
> > > >>>>>>>> consistency of the state. For example, if one sets the ISP subdev format
> > > >>>>>>>> first, then the resizer subdev and then the ISP subdev again, wouldn't the
> > > >>>>>>>> resizer subdev end up with a wrong format?
> > > >>>>>>>
> > > >>>>>>> yes, this is indeed a bug, I am preparing v4 now.
> > > >>>>>>> What I thought to do is adding quantization conversion
> > > >>>>>>> support also on ther resizer and capture entities.
> > > >>>>>>> Then in the 'link_validation' callbacks, there
> > > >>>>>>> is a validation that the quantization fields matches.
> > > >>>>>>
> > > >>>>>> My understanding is that, if we have the following topology
> > > >>>>>>
> > > >>>>>> [ ISP ] -> [ Resizer ] -> [ Video node ]
> > > >>>>>>
> > > >>>>>> then the ISP source pad would have the csc capability, while resizer
> > > >>>>>> and video node would just accept whatever is configured on their sink
> > > >>>>>> pads (no need for csc capability for that) and propagate that to their
> > > >>>>>> outputs, i.e. resizer source pad and video node CAPTURE format.
> > > >>>>>>
> > > >>>>>> Is this what you were going to implement?
> > > >>>> Hi, I sent a v4 where the CSC capability is set on the reiszer and capture as well.
> > > >>>> I can send a v5 implementing it the way you suggest. Currently the doc says that the colorspace fields
> > > >>>> must be set by the driver for capture streams. This implies that userspace can set it
> > > >>>> only if the CSC is supported.
> > > >>>
> > > >>> Why would the userspace have to set it for the capture stream on the
> > > >>> resizer and video nodes? Couldn't the userspace set it to DEFAULT and
> > > >>> then the driver override to whatever it received on the corresponding
> > > >>> sink?
> > >
> > > According to the documentation [1]:
> > >
> > > "the default quantization encoding as defined by the colorspace"
> > >
> > > Then, for instance, the default for V4L2_COLORSPACE_SRGB is limited range [2].
> > >
> > > Unless we change the definition of DEFAULT to let it accept whatever it receives on the corresponding sink.
> > >
> > > Then the way I understand is:
> > >
> > > [sensor]->[isp]->[resizer]->[capture]
> > >
> > > Userspace should set quantization on:
> > > * sensor source pad
> > > * isp sink pad
> > > * resizer sink pad
> > > * capture node
> > >
> > > The remaining source pads would get values propagated from their sink pads.
> > >
> > > [1] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-defs.html#c.v4l2_quantization
> > > [2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-details.html#col-srgb
> >
> > After discussing this offline, we concluded that:
> > - only the source pad of the ISP entity should expose the CSC capability,
> > - the CAPTURE video node should expose only the formats/color spaces
> > compatible with the current setting of the resizer source,
> > - consistency between the resizer source format and CAPTURE format
> > should be implicitly maintained by the driver.
>
> Do you mean it should be validated at streamon time ?

Nope. Any V4L2 state must be kept valid all the time, as mandated by
the general V4L2 principles. Wrong configuration of the video node
must not be a reason for STREAMON failing.

> In MC-based
> drivers, entities are isolated from each other from a configuration
> point of view, and the validity of the configuration across the whole
> pipeline is checked at streamon time. I don't think CSC parameters
> should be treated differently than formats in this regard.

Indeed, the ISP entity and the resizer entity are isolated from each
other and they need to be validated separately. However, the resizer
entity is not isolated from the video node, as there is no
configuration exposed for the video node in the MC API. If MC exposed
an explicit format control for the video interface sink pad, then the
resizer would have to be validated separately indeed, but still that
video interface sink pad would be tied to the rest of the
configuration of the video node (e.g. the CAPTURE format).

Best regards,
Tomasz

>
> > Below is the rationale for that:
> > - In the MC model, the video node represents the DMA alone (and thus
> > the reason to have the resizer entities at all) and since the
> > colorspace is not a property of the DMA, it doesn't belong there.
> > - Even if we put that aside, both MP and SP can only output the same
> > colorspace, which would create a dependency between the configuration
> > of the two video nodes, which is not something well defined by V4L2.
> > - A video interface in the MC API doesn't expose control over its
> > input format, which means that the relation between the other side of
> > the link (resizer source) and the video node is implicit and needs to
> > be handled internally in the driver.
> > - V4L2, as opposed to MC, requires that the current state is always
> > consistent and valid. In this case it means that an ioctl on the video
> > node must not be able to alter the state in a way that would make it
> > inconsistent with the state of the matching resizer source.
>
> --
> Regards,
>
> Laurent Pinchart

Re: [RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad

[Dafna Hirschfeld]

On 28.06.20 16:59, Tomasz Figa wrote:
> On Sun, Jun 28, 2020 at 3:15 PM Laurent Pinchart
> <laurent.pinchart@ideasonboard.com> wrote:
>>
>> Hi Tomasz,
>>
>> On Sun, Jun 28, 2020 at 03:10:41PM +0200, Tomasz Figa wrote:
>>> On Fri, Jun 26, 2020 at 2:15 AM Helen Koike <helen.koike@collabora.com> wrote:
>>>> On 6/18/20 4:25 PM, Tomasz Figa wrote:
>>>>> On Thu, Jun 18, 2020 at 08:54:34PM +0200, Dafna Hirschfeld wrote:
>>>>>> On 18.06.20 20:49, Tomasz Figa wrote:
>>>>>>> On Thu, Jun 18, 2020 at 08:26:33PM +0200, Dafna Hirschfeld wrote:
>>>>>>>> On 18.06.20 19:27, Tomasz Figa wrote:
>>>>>>>>> On Wed, Jun 10, 2020 at 2:08 PM Tomasz Figa wrote:
>>>>>>>>>> On Thu, Jun 4, 2020 at 9:11 PM Dafna Hirschfeld wrote:
>>>>>>>>>>> On 04.06.20 19:54, Tomasz Figa wrote:
>>>>>>>>>>>> On Thu, Apr 16, 2020 at 04:56:05PM +0200, Dafna Hirschfeld wrote:
>>>>>>>>>>>>> The isp entity has a hardware support to force full range quantization
>>>>>>>>>>>>> for YUV formats. Use the new API to indicate userspace that
>>>>>>>>>>>>> quantization conversion is supported by adding the flag
>>>>>>>>>>>>> V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION during media code enumeration.
>>>>>>>>>>>>> Then uppon s_fmt on the video source pad, we assign the
>>>>>>>>>>>>> quantization from userspace for YUV formats.
>>>>>>>>>>>>> Also in the capture and resizer entities we retrieve the colorspace
>>>>>>>>>>>>> from the isp entity.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
>>>>>>>>>>>>> ---
>>>>>>>>>>>>>      drivers/staging/media/rkisp1/rkisp1-capture.c | 28 ++++++-------
>>>>>>>>>>>>>      drivers/staging/media/rkisp1/rkisp1-common.h  |  2 +
>>>>>>>>>>>>>      drivers/staging/media/rkisp1/rkisp1-isp.c     | 39 +++++++++++++++++--
>>>>>>>>>>>>>      drivers/staging/media/rkisp1/rkisp1-resizer.c | 13 +++++++
>>>>>>>>>>>>>      4 files changed, 65 insertions(+), 17 deletions(-)
>>>>>>>>>>>>>
>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>>>>>>>>> index fbf62399fe3d..aca0f93bc772 100644
>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>>>>>>>>> @@ -1066,14 +1066,13 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
>>>>>>>>>>>>>                            const struct v4l2_format_info **fmt_info)
>>>>>>>>>>>>>      {
>>>>>>>>>>>>>         const struct rkisp1_capture_config *config = cap->config;
>>>>>>>>>>>>> -    struct rkisp1_capture *other_cap =
>>>>>>>>>>>>> -                    &cap->rkisp1->capture_devs[cap->id ^ 1];
>>>>>>>>>>>>>         const struct rkisp1_capture_fmt_cfg *fmt;
>>>>>>>>>>>>>         const struct v4l2_format_info *info;
>>>>>>>>>>>>>         const unsigned int max_widths[] = { RKISP1_RSZ_MP_SRC_MAX_WIDTH,
>>>>>>>>>>>>>                                             RKISP1_RSZ_SP_SRC_MAX_WIDTH };
>>>>>>>>>>>>>         const unsigned int max_heights[] = { RKISP1_RSZ_MP_SRC_MAX_HEIGHT,
>>>>>>>>>>>>>                                              RKISP1_RSZ_SP_SRC_MAX_HEIGHT};
>>>>>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
>>>>>>>>>>>>>
>>>>>>>>>>>>>         fmt = rkisp1_find_fmt_cfg(cap, pixm->pixelformat);
>>>>>>>>>>>>>         if (!fmt) {
>>>>>>>>>>>>> @@ -1081,24 +1080,27 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
>>>>>>>>>>>>>                 pixm->pixelformat = fmt->fourcc;
>>>>>>>>>>>>>         }
>>>>>>>>>>>>>
>>>>>>>>>>>>> +    rkisp1_get_isp_src_fmt(cap->rkisp1, &isp_sd_fmt);
>>>>>>>>>>>>> +    pixm->field = isp_sd_fmt.format.field;
>>>>>>>>>>>>> +    pixm->colorspace = isp_sd_fmt.format.colorspace;
>>>>>>>>>>>>> +    pixm->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
>>>>>>>>>>>>> +    pixm->xfer_func = isp_sd_fmt.format.xfer_func;
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +    /*
>>>>>>>>>>>>> +     * isp has a feature to set full range quantization for yuv formats.
>>>>>>>>>>>>
>>>>>>>>>>>> How about "select between limited and full range for YUV formats"?
>>>>>>>>>>>>
>>>>>>>>>>>>> +     * so we need to get the format from the isp.
>>>>>>>>>>>>> +     */
>>>>>>>>>>>>> +    pixm->quantization = isp_sd_fmt.format.quantization;
>>>>>>>>>>>>> +    if (!v4l2_is_format_yuv(cap->pix.info))
>>>>>>>>>>>>> +            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>> +
>>>>>>>>>>>>>         pixm->width = clamp_t(u32, pixm->width,
>>>>>>>>>>>>>                               RKISP1_RSZ_SRC_MIN_WIDTH, max_widths[cap->id]);
>>>>>>>>>>>>>         pixm->height = clamp_t(u32, pixm->height,
>>>>>>>>>>>>>                                RKISP1_RSZ_SRC_MIN_HEIGHT, max_heights[cap->id]);
>>>>>>>>>>>>>
>>>>>>>>>>>>> -    pixm->field = V4L2_FIELD_NONE;
>>>>>>>>>>>>> -    pixm->colorspace = V4L2_COLORSPACE_DEFAULT;
>>>>>>>>>>>>> -    pixm->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>>>>>>>>>>>>> -
>>>>>>>>>>>>>         info = rkisp1_fill_pixfmt(pixm, cap->id);
>>>>>>>>>>>>>
>>>>>>>>>>>>> -    /* can not change quantization when stream-on */
>>>>>>>>>>>>> -    if (other_cap->is_streaming)
>>>>>>>>>>>>> -            pixm->quantization = other_cap->pix.fmt.quantization;
>>>>>>>>>>>>> -    /* output full range by default, take effect in params */
>>>>>>>>>>>>> -    else if (!pixm->quantization ||
>>>>>>>>>>>>> -             pixm->quantization > V4L2_QUANTIZATION_LIM_RANGE)
>>>>>>>>>>>>> -            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>
>>>>>>>>>>>>>         if (fmt_cfg)
>>>>>>>>>>>>>                 *fmt_cfg = fmt;
>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-common.h b/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>>>>>>>>> index 2d7b7e078636..7a5576fa14c9 100644
>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>>>>>>>>> @@ -300,6 +300,8 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
>>>>>>>>>>>>>                         struct v4l2_device *v4l2_dev);
>>>>>>>>>>>>>      void rkisp1_isp_unregister(struct rkisp1_device *rkisp1);
>>>>>>>>>>>>>
>>>>>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
>>>>>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt);
>>>>>>>>>>>>>      const struct rkisp1_isp_mbus_info *rkisp1_isp_mbus_info_get(u32 mbus_code);
>>>>>>>>>>>>>
>>>>>>>>>>>>>      irqreturn_t rkisp1_isp_isr(struct rkisp1_device *rkisp1);
>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-isp.c b/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>>>>>>>>> index dee8e96f3900..6fdf5ed0b6b1 100644
>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>>>>>>>>> @@ -613,6 +613,10 @@ static int rkisp1_isp_enum_mbus_code(struct v4l2_subdev *sd,
>>>>>>>>>>>>>
>>>>>>>>>>>>>                 if (code->index == pos - 1) {
>>>>>>>>>>>>>                         code->code = fmt->mbus_code;
>>>>>>>>>>>>> +                    if (fmt->pixel_enc == V4L2_PIXEL_ENC_YUV &&
>>>>>>>>>>>>> +                        dir == RKISP1_DIR_SRC)
>>>>>>>>>>>>> +                            code->flags =
>>>>>>>>>>>>> +                                    V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION;
>>>>>>>>>>>>>                         return 0;
>>>>>>>>>>>>>                 }
>>>>>>>>>>>>>         }
>>>>>>>>>>>>> @@ -639,12 +643,21 @@ static int rkisp1_isp_init_config(struct v4l2_subdev *sd,
>>>>>>>>>>>>>         sink_crop->height = RKISP1_DEFAULT_HEIGHT;
>>>>>>>>>>>>>         sink_crop->left = 0;
>>>>>>>>>>>>>         sink_crop->top = 0;
>>>>>>>>>>>>> +    sink_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>>>>>>>>> +    sink_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(sink_fmt->colorspace);
>>>>>>>>>>>>> +    sink_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(sink_fmt->colorspace);
>>>>>>>>>>>>> +    sink_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>> +
>>>>>>>>>>>>>
>>>>>>>>>>>>>         src_fmt = v4l2_subdev_get_try_format(sd, cfg,
>>>>>>>>>>>>>                                              RKISP1_ISP_PAD_SOURCE_VIDEO);
>>>>>>>>>>>>>         *src_fmt = *sink_fmt;
>>>>>>>>>>>>>         src_fmt->code = RKISP1_DEF_SRC_PAD_FMT;
>>>>>>>>>>>>> -    src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
>>>>>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
>>>>>>>>>>>>> +    src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
>>>>>>>>>>>>> +
>>>>>>>>>>>>>
>>>>>>>>>>>>>         src_crop = v4l2_subdev_get_try_crop(sd, cfg,
>>>>>>>>>>>>>                                             RKISP1_ISP_PAD_SOURCE_VIDEO);
>>>>>>>>>>>>> @@ -687,10 +700,17 @@ static void rkisp1_isp_set_src_fmt(struct rkisp1_isp *isp,
>>>>>>>>>>>>>                 isp->src_fmt = mbus_info;
>>>>>>>>>>>>>         src_fmt->width  = src_crop->width;
>>>>>>>>>>>>>         src_fmt->height = src_crop->height;
>>>>>>>>>>>>> -    src_fmt->quantization = format->quantization;
>>>>>>>>>>>>> -    /* full range by default */
>>>>>>>>>>>>> -    if (!src_fmt->quantization)
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
>>>>>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +    if (mbus_info->pixel_enc == V4L2_PIXEL_ENC_BAYER)
>>>>>>>>>>>>>                 src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>> +    else if (format->quantization == V4L2_QUANTIZATION_DEFAULT)
>>>>>>>>>>>>> +            src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
>>>>>>>>>>>>> +    else
>>>>>>>>>>>>> +            src_fmt->quantization = format->quantization;
>>>>>>>>>>>>>
>>>>>>>>>>>>>         *format = *src_fmt;
>>>>>>>>>>>>>      }
>>>>>>>>>>>>> @@ -1068,6 +1088,17 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
>>>>>>>>>>>>>         return ret;
>>>>>>>>>>>>>      }
>>>>>>>>>>>>>
>>>>>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
>>>>>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt)
>>>>>>>>>>>>> +{
>>>>>>>>>>>>> +    struct rkisp1_isp *isp = &rkisp1->isp;
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +    sd_fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
>>>>>>>>>>>>> +    sd_fmt->pad = RKISP1_ISP_PAD_SOURCE_VIDEO;
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +    return v4l2_subdev_call(&isp->sd, pad, get_fmt, NULL, sd_fmt);
>>>>>>>>>>>>
>>>>>>>>>>>> Do we need to get through the external API to access data that is
>>>>>>>>>>>> driver-internal anyway?
>>>>>>>>>>>>
>>>>>>>>>>>>> +}
>>>>>>>>>>>>> +
>>>>>>>>>>>>>      void rkisp1_isp_unregister(struct rkisp1_device *rkisp1)
>>>>>>>>>>>>>      {
>>>>>>>>>>>>>         struct v4l2_subdev *sd = &rkisp1->isp.sd;
>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-resizer.c b/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>>>>>>>>> index 7b6b7ddd4169..8705b133de68 100644
>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>>>>>>>>> @@ -525,6 +525,7 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
>>>>>>>>>>>>>         const struct rkisp1_isp_mbus_info *mbus_info;
>>>>>>>>>>>>>         struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
>>>>>>>>>>>>>         struct v4l2_rect *sink_crop;
>>>>>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
>>>>>>>>>>>>>
>>>>>>>>>>>>>         sink_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SINK, which);
>>>>>>>>>>>>>         src_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SRC, which);
>>>>>>>>>>>>> @@ -539,8 +540,20 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
>>>>>>>>>>>>>         if (which == V4L2_SUBDEV_FORMAT_ACTIVE)
>>>>>>>>>>>>>                 rsz->pixel_enc = mbus_info->pixel_enc;
>>>>>>>>>>>>>
>>>>>>>>>>>>> +    rkisp1_get_isp_src_fmt(rsz->rkisp1, &isp_sd_fmt);
>>>>>>>>>>>>> +
>>>>>>>>>>>>
>>>>>>>>>>>> Is this necessary? My understanding was that in the subdev model, it was
>>>>>>>>>>>> the userspace responsibility to propagate any configuration changes through
>>>>>>>>>>>> the graph.
>>>>>>>>>>>>
>>>>>>>>>>>> Also, doing this only here wouldn't fully maintain the
>>>>>>>>>>>> consistency of the state. For example, if one sets the ISP subdev format
>>>>>>>>>>>> first, then the resizer subdev and then the ISP subdev again, wouldn't the
>>>>>>>>>>>> resizer subdev end up with a wrong format?
>>>>>>>>>>>
>>>>>>>>>>> yes, this is indeed a bug, I am preparing v4 now.
>>>>>>>>>>> What I thought to do is adding quantization conversion
>>>>>>>>>>> support also on ther resizer and capture entities.
>>>>>>>>>>> Then in the 'link_validation' callbacks, there
>>>>>>>>>>> is a validation that the quantization fields matches.
>>>>>>>>>>
>>>>>>>>>> My understanding is that, if we have the following topology
>>>>>>>>>>
>>>>>>>>>> [ ISP ] -> [ Resizer ] -> [ Video node ]
>>>>>>>>>>
>>>>>>>>>> then the ISP source pad would have the csc capability, while resizer
>>>>>>>>>> and video node would just accept whatever is configured on their sink
>>>>>>>>>> pads (no need for csc capability for that) and propagate that to their
>>>>>>>>>> outputs, i.e. resizer source pad and video node CAPTURE format.
>>>>>>>>>>
>>>>>>>>>> Is this what you were going to implement?
>>>>>>>> Hi, I sent a v4 where the CSC capability is set on the reiszer and capture as well.
>>>>>>>> I can send a v5 implementing it the way you suggest. Currently the doc says that the colorspace fields
>>>>>>>> must be set by the driver for capture streams. This implies that userspace can set it
>>>>>>>> only if the CSC is supported.
>>>>>>>
>>>>>>> Why would the userspace have to set it for the capture stream on the
>>>>>>> resizer and video nodes? Couldn't the userspace set it to DEFAULT and
>>>>>>> then the driver override to whatever it received on the corresponding
>>>>>>> sink?
>>>>
>>>> According to the documentation [1]:
>>>>
>>>> "the default quantization encoding as defined by the colorspace"
>>>>
>>>> Then, for instance, the default for V4L2_COLORSPACE_SRGB is limited range [2].
>>>>
>>>> Unless we change the definition of DEFAULT to let it accept whatever it receives on the corresponding sink.
>>>>
>>>> Then the way I understand is:
>>>>
>>>> [sensor]->[isp]->[resizer]->[capture]
>>>>
>>>> Userspace should set quantization on:
>>>> * sensor source pad
>>>> * isp sink pad
>>>> * resizer sink pad
>>>> * capture node
>>>>
>>>> The remaining source pads would get values propagated from their sink pads.
>>>>
>>>> [1] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-defs.html#c.v4l2_quantization
>>>> [2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-details.html#col-srgb
>>>
>>> After discussing this offline, we concluded that:
>>> - only the source pad of the ISP entity should expose the CSC capability,
>>> - the CAPTURE video node should expose only the formats/color spaces
>>> compatible with the current setting of the resizer source,
>>> - consistency between the resizer source format and CAPTURE format
>>> should be implicitly maintained by the driver.
>>
>> Do you mean it should be validated at streamon time ?
> 
> Nope. Any V4L2 state must be kept valid all the time, as mandated by
> the general V4L2 principles. Wrong configuration of the video node
> must not be a reason for STREAMON failing.
> 
>> In MC-based
>> drivers, entities are isolated from each other from a configuration
>> point of view, and the validity of the configuration across the whole
>> pipeline is checked at streamon time. I don't think CSC parameters
>> should be treated differently than formats in this regard.
> 
> Indeed, the ISP entity and the resizer entity are isolated from each
> other and they need to be validated separately. However, the resizer
> entity is not isolated from the video node, as there is no
> configuration exposed for the video node in the MC API. If MC exposed
> an explicit format control for the video interface sink pad, then the
> resizer would have to be validated separately indeed, but still that
> video interface sink pad would be tied to the rest of the
> configuration of the video node (e.g. the CAPTURE format).

Two questions:

The validation callback between the resizer and the isp has to include check
that the quantization values matches ?
Userspace have to configure the quantizaion on the resizer sink pad without that CSC
API beeing expose on it?


Thanks,
Dafna
  
> 
> Best regards,
> Tomasz
> 
>>
>>> Below is the rationale for that:
>>> - In the MC model, the video node represents the DMA alone (and thus
>>> the reason to have the resizer entities at all) and since the
>>> colorspace is not a property of the DMA, it doesn't belong there.
>>> - Even if we put that aside, both MP and SP can only output the same
>>> colorspace, which would create a dependency between the configuration
>>> of the two video nodes, which is not something well defined by V4L2.
>>> - A video interface in the MC API doesn't expose control over its
>>> input format, which means that the relation between the other side of
>>> the link (resizer source) and the video node is implicit and needs to
>>> be handled internally in the driver.
>>> - V4L2, as opposed to MC, requires that the current state is always
>>> consistent and valid. In this case it means that an ioctl on the video
>>> node must not be able to alter the state in a way that would make it
>>> inconsistent with the state of the matching resizer source.
>>
>> --
>> Regards,
>>
>> Laurent Pinchart

Re: [RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad

[Helen Koike]

Hello Dafna, Laurent and Tomasz,

On 6/28/20 4:34 PM, Dafna Hirschfeld wrote:
> 
> 
> On 28.06.20 16:59, Tomasz Figa wrote:
>> On Sun, Jun 28, 2020 at 3:15 PM Laurent Pinchart
>> <laurent.pinchart@ideasonboard.com> wrote:
>>>
>>> Hi Tomasz,
>>>
>>> On Sun, Jun 28, 2020 at 03:10:41PM +0200, Tomasz Figa wrote:
>>>> On Fri, Jun 26, 2020 at 2:15 AM Helen Koike <helen.koike@collabora.com> wrote:
>>>>> On 6/18/20 4:25 PM, Tomasz Figa wrote:
>>>>>> On Thu, Jun 18, 2020 at 08:54:34PM +0200, Dafna Hirschfeld wrote:
>>>>>>> On 18.06.20 20:49, Tomasz Figa wrote:
>>>>>>>> On Thu, Jun 18, 2020 at 08:26:33PM +0200, Dafna Hirschfeld wrote:
>>>>>>>>> On 18.06.20 19:27, Tomasz Figa wrote:
>>>>>>>>>> On Wed, Jun 10, 2020 at 2:08 PM Tomasz Figa wrote:
>>>>>>>>>>> On Thu, Jun 4, 2020 at 9:11 PM Dafna Hirschfeld wrote:
>>>>>>>>>>>> On 04.06.20 19:54, Tomasz Figa wrote:
>>>>>>>>>>>>> On Thu, Apr 16, 2020 at 04:56:05PM +0200, Dafna Hirschfeld wrote:
>>>>>>>>>>>>>> The isp entity has a hardware support to force full range quantization
>>>>>>>>>>>>>> for YUV formats. Use the new API to indicate userspace that
>>>>>>>>>>>>>> quantization conversion is supported by adding the flag
>>>>>>>>>>>>>> V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION during media code enumeration.
>>>>>>>>>>>>>> Then uppon s_fmt on the video source pad, we assign the
>>>>>>>>>>>>>> quantization from userspace for YUV formats.
>>>>>>>>>>>>>> Also in the capture and resizer entities we retrieve the colorspace
>>>>>>>>>>>>>> from the isp entity.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>      drivers/staging/media/rkisp1/rkisp1-capture.c | 28 ++++++-------
>>>>>>>>>>>>>>      drivers/staging/media/rkisp1/rkisp1-common.h  |  2 +
>>>>>>>>>>>>>>      drivers/staging/media/rkisp1/rkisp1-isp.c     | 39 +++++++++++++++++--
>>>>>>>>>>>>>>      drivers/staging/media/rkisp1/rkisp1-resizer.c | 13 +++++++
>>>>>>>>>>>>>>      4 files changed, 65 insertions(+), 17 deletions(-)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>>>>>>>>>> index fbf62399fe3d..aca0f93bc772 100644
>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>>>>>>>>>> @@ -1066,14 +1066,13 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
>>>>>>>>>>>>>>                            const struct v4l2_format_info **fmt_info)
>>>>>>>>>>>>>>      {
>>>>>>>>>>>>>>         const struct rkisp1_capture_config *config = cap->config;
>>>>>>>>>>>>>> -    struct rkisp1_capture *other_cap =
>>>>>>>>>>>>>> -                    &cap->rkisp1->capture_devs[cap->id ^ 1];
>>>>>>>>>>>>>>         const struct rkisp1_capture_fmt_cfg *fmt;
>>>>>>>>>>>>>>         const struct v4l2_format_info *info;
>>>>>>>>>>>>>>         const unsigned int max_widths[] = { RKISP1_RSZ_MP_SRC_MAX_WIDTH,
>>>>>>>>>>>>>>                                             RKISP1_RSZ_SP_SRC_MAX_WIDTH };
>>>>>>>>>>>>>>         const unsigned int max_heights[] = { RKISP1_RSZ_MP_SRC_MAX_HEIGHT,
>>>>>>>>>>>>>>                                              RKISP1_RSZ_SP_SRC_MAX_HEIGHT};
>>>>>>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>         fmt = rkisp1_find_fmt_cfg(cap, pixm->pixelformat);
>>>>>>>>>>>>>>         if (!fmt) {
>>>>>>>>>>>>>> @@ -1081,24 +1080,27 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
>>>>>>>>>>>>>>                 pixm->pixelformat = fmt->fourcc;
>>>>>>>>>>>>>>         }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> +    rkisp1_get_isp_src_fmt(cap->rkisp1, &isp_sd_fmt);
>>>>>>>>>>>>>> +    pixm->field = isp_sd_fmt.format.field;
>>>>>>>>>>>>>> +    pixm->colorspace = isp_sd_fmt.format.colorspace;
>>>>>>>>>>>>>> +    pixm->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
>>>>>>>>>>>>>> +    pixm->xfer_func = isp_sd_fmt.format.xfer_func;
>>>>>>>>>>>>>> +
>>>>>>>>>>>>>> +    /*
>>>>>>>>>>>>>> +     * isp has a feature to set full range quantization for yuv formats.
>>>>>>>>>>>>>
>>>>>>>>>>>>> How about "select between limited and full range for YUV formats"?
>>>>>>>>>>>>>
>>>>>>>>>>>>>> +     * so we need to get the format from the isp.
>>>>>>>>>>>>>> +     */
>>>>>>>>>>>>>> +    pixm->quantization = isp_sd_fmt.format.quantization;
>>>>>>>>>>>>>> +    if (!v4l2_is_format_yuv(cap->pix.info))
>>>>>>>>>>>>>> +            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>         pixm->width = clamp_t(u32, pixm->width,
>>>>>>>>>>>>>>                               RKISP1_RSZ_SRC_MIN_WIDTH, max_widths[cap->id]);
>>>>>>>>>>>>>>         pixm->height = clamp_t(u32, pixm->height,
>>>>>>>>>>>>>>                                RKISP1_RSZ_SRC_MIN_HEIGHT, max_heights[cap->id]);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -    pixm->field = V4L2_FIELD_NONE;
>>>>>>>>>>>>>> -    pixm->colorspace = V4L2_COLORSPACE_DEFAULT;
>>>>>>>>>>>>>> -    pixm->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>         info = rkisp1_fill_pixfmt(pixm, cap->id);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -    /* can not change quantization when stream-on */
>>>>>>>>>>>>>> -    if (other_cap->is_streaming)
>>>>>>>>>>>>>> -            pixm->quantization = other_cap->pix.fmt.quantization;
>>>>>>>>>>>>>> -    /* output full range by default, take effect in params */
>>>>>>>>>>>>>> -    else if (!pixm->quantization ||
>>>>>>>>>>>>>> -             pixm->quantization > V4L2_QUANTIZATION_LIM_RANGE)
>>>>>>>>>>>>>> -            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>         if (fmt_cfg)
>>>>>>>>>>>>>>                 *fmt_cfg = fmt;
>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-common.h b/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>>>>>>>>>> index 2d7b7e078636..7a5576fa14c9 100644
>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>>>>>>>>>> @@ -300,6 +300,8 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
>>>>>>>>>>>>>>                         struct v4l2_device *v4l2_dev);
>>>>>>>>>>>>>>      void rkisp1_isp_unregister(struct rkisp1_device *rkisp1);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
>>>>>>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt);
>>>>>>>>>>>>>>      const struct rkisp1_isp_mbus_info *rkisp1_isp_mbus_info_get(u32 mbus_code);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>      irqreturn_t rkisp1_isp_isr(struct rkisp1_device *rkisp1);
>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-isp.c b/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>>>>>>>>>> index dee8e96f3900..6fdf5ed0b6b1 100644
>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>>>>>>>>>> @@ -613,6 +613,10 @@ static int rkisp1_isp_enum_mbus_code(struct v4l2_subdev *sd,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                 if (code->index == pos - 1) {
>>>>>>>>>>>>>>                         code->code = fmt->mbus_code;
>>>>>>>>>>>>>> +                    if (fmt->pixel_enc == V4L2_PIXEL_ENC_YUV &&
>>>>>>>>>>>>>> +                        dir == RKISP1_DIR_SRC)
>>>>>>>>>>>>>> +                            code->flags =
>>>>>>>>>>>>>> +                                    V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION;
>>>>>>>>>>>>>>                         return 0;
>>>>>>>>>>>>>>                 }
>>>>>>>>>>>>>>         }
>>>>>>>>>>>>>> @@ -639,12 +643,21 @@ static int rkisp1_isp_init_config(struct v4l2_subdev *sd,
>>>>>>>>>>>>>>         sink_crop->height = RKISP1_DEFAULT_HEIGHT;
>>>>>>>>>>>>>>         sink_crop->left = 0;
>>>>>>>>>>>>>>         sink_crop->top = 0;
>>>>>>>>>>>>>> +    sink_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>>>>>>>>>> +    sink_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(sink_fmt->colorspace);
>>>>>>>>>>>>>> +    sink_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(sink_fmt->colorspace);
>>>>>>>>>>>>>> +    sink_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>         src_fmt = v4l2_subdev_get_try_format(sd, cfg,
>>>>>>>>>>>>>>                                              RKISP1_ISP_PAD_SOURCE_VIDEO);
>>>>>>>>>>>>>>         *src_fmt = *sink_fmt;
>>>>>>>>>>>>>>         src_fmt->code = RKISP1_DEF_SRC_PAD_FMT;
>>>>>>>>>>>>>> -    src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
>>>>>>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
>>>>>>>>>>>>>> +    src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>         src_crop = v4l2_subdev_get_try_crop(sd, cfg,
>>>>>>>>>>>>>>                                             RKISP1_ISP_PAD_SOURCE_VIDEO);
>>>>>>>>>>>>>> @@ -687,10 +700,17 @@ static void rkisp1_isp_set_src_fmt(struct rkisp1_isp *isp,
>>>>>>>>>>>>>>                 isp->src_fmt = mbus_info;
>>>>>>>>>>>>>>         src_fmt->width  = src_crop->width;
>>>>>>>>>>>>>>         src_fmt->height = src_crop->height;
>>>>>>>>>>>>>> -    src_fmt->quantization = format->quantization;
>>>>>>>>>>>>>> -    /* full range by default */
>>>>>>>>>>>>>> -    if (!src_fmt->quantization)
>>>>>>>>>>>>>> +
>>>>>>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
>>>>>>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
>>>>>>>>>>>>>> +
>>>>>>>>>>>>>> +    if (mbus_info->pixel_enc == V4L2_PIXEL_ENC_BAYER)
>>>>>>>>>>>>>>                 src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>> +    else if (format->quantization == V4L2_QUANTIZATION_DEFAULT)
>>>>>>>>>>>>>> +            src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
>>>>>>>>>>>>>> +    else
>>>>>>>>>>>>>> +            src_fmt->quantization = format->quantization;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>         *format = *src_fmt;
>>>>>>>>>>>>>>      }
>>>>>>>>>>>>>> @@ -1068,6 +1088,17 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
>>>>>>>>>>>>>>         return ret;
>>>>>>>>>>>>>>      }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
>>>>>>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt)
>>>>>>>>>>>>>> +{
>>>>>>>>>>>>>> +    struct rkisp1_isp *isp = &rkisp1->isp;
>>>>>>>>>>>>>> +
>>>>>>>>>>>>>> +    sd_fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
>>>>>>>>>>>>>> +    sd_fmt->pad = RKISP1_ISP_PAD_SOURCE_VIDEO;
>>>>>>>>>>>>>> +
>>>>>>>>>>>>>> +    return v4l2_subdev_call(&isp->sd, pad, get_fmt, NULL, sd_fmt);
>>>>>>>>>>>>>
>>>>>>>>>>>>> Do we need to get through the external API to access data that is
>>>>>>>>>>>>> driver-internal anyway?
>>>>>>>>>>>>>
>>>>>>>>>>>>>> +}
>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>      void rkisp1_isp_unregister(struct rkisp1_device *rkisp1)
>>>>>>>>>>>>>>      {
>>>>>>>>>>>>>>         struct v4l2_subdev *sd = &rkisp1->isp.sd;
>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-resizer.c b/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>>>>>>>>>> index 7b6b7ddd4169..8705b133de68 100644
>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>>>>>>>>>> @@ -525,6 +525,7 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
>>>>>>>>>>>>>>         const struct rkisp1_isp_mbus_info *mbus_info;
>>>>>>>>>>>>>>         struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
>>>>>>>>>>>>>>         struct v4l2_rect *sink_crop;
>>>>>>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>         sink_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SINK, which);
>>>>>>>>>>>>>>         src_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SRC, which);
>>>>>>>>>>>>>> @@ -539,8 +540,20 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
>>>>>>>>>>>>>>         if (which == V4L2_SUBDEV_FORMAT_ACTIVE)
>>>>>>>>>>>>>>                 rsz->pixel_enc = mbus_info->pixel_enc;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> +    rkisp1_get_isp_src_fmt(rsz->rkisp1, &isp_sd_fmt);
>>>>>>>>>>>>>> +
>>>>>>>>>>>>>
>>>>>>>>>>>>> Is this necessary? My understanding was that in the subdev model, it was
>>>>>>>>>>>>> the userspace responsibility to propagate any configuration changes through
>>>>>>>>>>>>> the graph.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Also, doing this only here wouldn't fully maintain the
>>>>>>>>>>>>> consistency of the state. For example, if one sets the ISP subdev format
>>>>>>>>>>>>> first, then the resizer subdev and then the ISP subdev again, wouldn't the
>>>>>>>>>>>>> resizer subdev end up with a wrong format?
>>>>>>>>>>>>
>>>>>>>>>>>> yes, this is indeed a bug, I am preparing v4 now.
>>>>>>>>>>>> What I thought to do is adding quantization conversion
>>>>>>>>>>>> support also on ther resizer and capture entities.
>>>>>>>>>>>> Then in the 'link_validation' callbacks, there
>>>>>>>>>>>> is a validation that the quantization fields matches.
>>>>>>>>>>>
>>>>>>>>>>> My understanding is that, if we have the following topology
>>>>>>>>>>>
>>>>>>>>>>> [ ISP ] -> [ Resizer ] -> [ Video node ]
>>>>>>>>>>>
>>>>>>>>>>> then the ISP source pad would have the csc capability, while resizer
>>>>>>>>>>> and video node would just accept whatever is configured on their sink
>>>>>>>>>>> pads (no need for csc capability for that) and propagate that to their
>>>>>>>>>>> outputs, i.e. resizer source pad and video node CAPTURE format.
>>>>>>>>>>>
>>>>>>>>>>> Is this what you were going to implement?
>>>>>>>>> Hi, I sent a v4 where the CSC capability is set on the reiszer and capture as well.
>>>>>>>>> I can send a v5 implementing it the way you suggest. Currently the doc says that the colorspace fields
>>>>>>>>> must be set by the driver for capture streams. This implies that userspace can set it
>>>>>>>>> only if the CSC is supported.
>>>>>>>>
>>>>>>>> Why would the userspace have to set it for the capture stream on the
>>>>>>>> resizer and video nodes? Couldn't the userspace set it to DEFAULT and
>>>>>>>> then the driver override to whatever it received on the corresponding
>>>>>>>> sink?
>>>>>
>>>>> According to the documentation [1]:
>>>>>
>>>>> "the default quantization encoding as defined by the colorspace"
>>>>>
>>>>> Then, for instance, the default for V4L2_COLORSPACE_SRGB is limited range [2].
>>>>>
>>>>> Unless we change the definition of DEFAULT to let it accept whatever it receives on the corresponding sink.
>>>>>
>>>>> Then the way I understand is:
>>>>>
>>>>> [sensor]->[isp]->[resizer]->[capture]
>>>>>
>>>>> Userspace should set quantization on:
>>>>> * sensor source pad
>>>>> * isp sink pad
>>>>> * resizer sink pad
>>>>> * capture node
>>>>>
>>>>> The remaining source pads would get values propagated from their sink pads.
>>>>>
>>>>> [1] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-defs.html#c.v4l2_quantization
>>>>> [2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-details.html#col-srgb
>>>>
>>>> After discussing this offline, we concluded that:
>>>> - only the source pad of the ISP entity should expose the CSC capability,
>>>> - the CAPTURE video node should expose only the formats/color spaces
>>>> compatible with the current setting of the resizer source,
>>>> - consistency between the resizer source format and CAPTURE format
>>>> should be implicitly maintained by the driver.
>>>
>>> Do you mean it should be validated at streamon time ?
>>
>> Nope. Any V4L2 state must be kept valid all the time, as mandated by
>> the general V4L2 principles. Wrong configuration of the video node
>> must not be a reason for STREAMON failing.
>>
>>> In MC-based
>>> drivers, entities are isolated from each other from a configuration
>>> point of view, and the validity of the configuration across the whole
>>> pipeline is checked at streamon time. I don't think CSC parameters
>>> should be treated differently than formats in this regard.
>>
>> Indeed, the ISP entity and the resizer entity are isolated from each
>> other and they need to be validated separately. However, the resizer
>> entity is not isolated from the video node, as there is no
>> configuration exposed for the video node in the MC API. If MC exposed
>> an explicit format control for the video interface sink pad, then the
>> resizer would have to be validated separately indeed, but still that
>> video interface sink pad would be tied to the rest of the
>> configuration of the video node (e.g. the CAPTURE format).
> 
> Two questions:
> 
> The validation callback between the resizer and the isp has to include check
> that the quantization values matches ?


To sumarize:

The current quantization fields from VIDIOC_SUBDEV_{S,G}_FMT and VIDEO_{SET,GET}_FORMAT
are filled by the driver (read-only by userspace).

This RFC exposes an API that allows the quantization to be a read-write configuration.
The question is where we expose the read-write capability.

We have two main design choices here.

1) Expose read-write quantization in all pads and video nodes.
And userspace must set matching quantization in all of them.
Which makes userspace more complicated, but userspace already needs to configure
the whole topology anyway.
This wouldn't make much sense in video nodes for metadata (params and statistics)

In this options, the validation callback should check for matching quantizations
between all links, and it would failed on STREAMON if formats don't match.

2-A) Expose read-write quantization only on the isp source pad.
The others pads and video nodes would be keept read-only.
Which means that changing quantization on a read-write node, will implicitly
change the values on the read-only nodes.
Which maybe not be a problem, since current API (read-only) already says this field
is filled by the driver.

In this options, the validation callback doesn't need to check for matching quantizations
between links.

2-A) Expose read-write quantization only on the video node
This is basically the same as 2-A, but userspace could interpret that we
could have different quantizations in each video node.


I'd like to hear from others what would be better here, design choice 1 or 2.


> Userspace have to configure the quantizaion on the resizer sink pad without that CSC
> API beeing expose on it?

In the current API, the quantization field is read-only, so to allow userspace to configure
quantization in any pad, the CSC API needs to be exposed on the pad to make it read-write.


Regards,
Helen

> 
> 
> Thanks,
> Dafna
>  
>>
>> Best regards,
>> Tomasz
>>
>>>
>>>> Below is the rationale for that:
>>>> - In the MC model, the video node represents the DMA alone (and thus
>>>> the reason to have the resizer entities at all) and since the
>>>> colorspace is not a property of the DMA, it doesn't belong there.
>>>> - Even if we put that aside, both MP and SP can only output the same
>>>> colorspace, which would create a dependency between the configuration
>>>> of the two video nodes, which is not something well defined by V4L2.
>>>> - A video interface in the MC API doesn't expose control over its
>>>> input format, which means that the relation between the other side of
>>>> the link (resizer source) and the video node is implicit and needs to
>>>> be handled internally in the driver.
>>>> - V4L2, as opposed to MC, requires that the current state is always
>>>> consistent and valid. In this case it means that an ioctl on the video
>>>> node must not be able to alter the state in a way that would make it
>>>> inconsistent with the state of the matching resizer source.
>>>
>>> -- 
>>> Regards,
>>>
>>> Laurent Pinchart

Re: [RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad

[Laurent Pinchart]

Hi Helen,

On Mon, Jun 29, 2020 at 12:50:06PM -0300, Helen Koike wrote:
> On 6/28/20 4:34 PM, Dafna Hirschfeld wrote:
> > On 28.06.20 16:59, Tomasz Figa wrote:
> >> On Sun, Jun 28, 2020 at 3:15 PM Laurent Pinchart wrote:
> >>> On Sun, Jun 28, 2020 at 03:10:41PM +0200, Tomasz Figa wrote:
> >>>> On Fri, Jun 26, 2020 at 2:15 AM Helen Koike <helen.koike@collabora.com> wrote:
> >>>>> On 6/18/20 4:25 PM, Tomasz Figa wrote:
> >>>>>> On Thu, Jun 18, 2020 at 08:54:34PM +0200, Dafna Hirschfeld wrote:
> >>>>>>> On 18.06.20 20:49, Tomasz Figa wrote:
> >>>>>>>> On Thu, Jun 18, 2020 at 08:26:33PM +0200, Dafna Hirschfeld wrote:
> >>>>>>>>> On 18.06.20 19:27, Tomasz Figa wrote:
> >>>>>>>>>> On Wed, Jun 10, 2020 at 2:08 PM Tomasz Figa wrote:
> >>>>>>>>>>> On Thu, Jun 4, 2020 at 9:11 PM Dafna Hirschfeld wrote:
> >>>>>>>>>>>> On 04.06.20 19:54, Tomasz Figa wrote:
> >>>>>>>>>>>>> On Thu, Apr 16, 2020 at 04:56:05PM +0200, Dafna Hirschfeld wrote:
> >>>>>>>>>>>>>> The isp entity has a hardware support to force full range quantization
> >>>>>>>>>>>>>> for YUV formats. Use the new API to indicate userspace that
> >>>>>>>>>>>>>> quantization conversion is supported by adding the flag
> >>>>>>>>>>>>>> V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION during media code enumeration.
> >>>>>>>>>>>>>> Then uppon s_fmt on the video source pad, we assign the
> >>>>>>>>>>>>>> quantization from userspace for YUV formats.
> >>>>>>>>>>>>>> Also in the capture and resizer entities we retrieve the colorspace
> >>>>>>>>>>>>>> from the isp entity.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
> >>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>      drivers/staging/media/rkisp1/rkisp1-capture.c | 28 ++++++-------
> >>>>>>>>>>>>>>      drivers/staging/media/rkisp1/rkisp1-common.h  |  2 +
> >>>>>>>>>>>>>>      drivers/staging/media/rkisp1/rkisp1-isp.c     | 39 +++++++++++++++++--
> >>>>>>>>>>>>>>      drivers/staging/media/rkisp1/rkisp1-resizer.c | 13 +++++++
> >>>>>>>>>>>>>>      4 files changed, 65 insertions(+), 17 deletions(-)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c
> >>>>>>>>>>>>>> index fbf62399fe3d..aca0f93bc772 100644
> >>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-capture.c
> >>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-capture.c
> >>>>>>>>>>>>>> @@ -1066,14 +1066,13 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
> >>>>>>>>>>>>>>                            const struct v4l2_format_info **fmt_info)
> >>>>>>>>>>>>>>      {
> >>>>>>>>>>>>>>         const struct rkisp1_capture_config *config = cap->config;
> >>>>>>>>>>>>>> -    struct rkisp1_capture *other_cap =
> >>>>>>>>>>>>>> -                    &cap->rkisp1->capture_devs[cap->id ^ 1];
> >>>>>>>>>>>>>>         const struct rkisp1_capture_fmt_cfg *fmt;
> >>>>>>>>>>>>>>         const struct v4l2_format_info *info;
> >>>>>>>>>>>>>>         const unsigned int max_widths[] = { RKISP1_RSZ_MP_SRC_MAX_WIDTH,
> >>>>>>>>>>>>>>                                             RKISP1_RSZ_SP_SRC_MAX_WIDTH };
> >>>>>>>>>>>>>>         const unsigned int max_heights[] = { RKISP1_RSZ_MP_SRC_MAX_HEIGHT,
> >>>>>>>>>>>>>>                                              RKISP1_RSZ_SP_SRC_MAX_HEIGHT};
> >>>>>>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>         fmt = rkisp1_find_fmt_cfg(cap, pixm->pixelformat);
> >>>>>>>>>>>>>>         if (!fmt) {
> >>>>>>>>>>>>>> @@ -1081,24 +1080,27 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
> >>>>>>>>>>>>>>                 pixm->pixelformat = fmt->fourcc;
> >>>>>>>>>>>>>>         }
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> +    rkisp1_get_isp_src_fmt(cap->rkisp1, &isp_sd_fmt);
> >>>>>>>>>>>>>> +    pixm->field = isp_sd_fmt.format.field;
> >>>>>>>>>>>>>> +    pixm->colorspace = isp_sd_fmt.format.colorspace;
> >>>>>>>>>>>>>> +    pixm->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
> >>>>>>>>>>>>>> +    pixm->xfer_func = isp_sd_fmt.format.xfer_func;
> >>>>>>>>>>>>>> +
> >>>>>>>>>>>>>> +    /*
> >>>>>>>>>>>>>> +     * isp has a feature to set full range quantization for yuv formats.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> How about "select between limited and full range for YUV formats"?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> +     * so we need to get the format from the isp.
> >>>>>>>>>>>>>> +     */
> >>>>>>>>>>>>>> +    pixm->quantization = isp_sd_fmt.format.quantization;
> >>>>>>>>>>>>>> +    if (!v4l2_is_format_yuv(cap->pix.info))
> >>>>>>>>>>>>>> +            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>         pixm->width = clamp_t(u32, pixm->width,
> >>>>>>>>>>>>>>                               RKISP1_RSZ_SRC_MIN_WIDTH, max_widths[cap->id]);
> >>>>>>>>>>>>>>         pixm->height = clamp_t(u32, pixm->height,
> >>>>>>>>>>>>>>                                RKISP1_RSZ_SRC_MIN_HEIGHT, max_heights[cap->id]);
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -    pixm->field = V4L2_FIELD_NONE;
> >>>>>>>>>>>>>> -    pixm->colorspace = V4L2_COLORSPACE_DEFAULT;
> >>>>>>>>>>>>>> -    pixm->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> >>>>>>>>>>>>>> -
> >>>>>>>>>>>>>>         info = rkisp1_fill_pixfmt(pixm, cap->id);
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -    /* can not change quantization when stream-on */
> >>>>>>>>>>>>>> -    if (other_cap->is_streaming)
> >>>>>>>>>>>>>> -            pixm->quantization = other_cap->pix.fmt.quantization;
> >>>>>>>>>>>>>> -    /* output full range by default, take effect in params */
> >>>>>>>>>>>>>> -    else if (!pixm->quantization ||
> >>>>>>>>>>>>>> -             pixm->quantization > V4L2_QUANTIZATION_LIM_RANGE)
> >>>>>>>>>>>>>> -            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>         if (fmt_cfg)
> >>>>>>>>>>>>>>                 *fmt_cfg = fmt;
> >>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-common.h b/drivers/staging/media/rkisp1/rkisp1-common.h
> >>>>>>>>>>>>>> index 2d7b7e078636..7a5576fa14c9 100644
> >>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-common.h
> >>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-common.h
> >>>>>>>>>>>>>> @@ -300,6 +300,8 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
> >>>>>>>>>>>>>>                         struct v4l2_device *v4l2_dev);
> >>>>>>>>>>>>>>      void rkisp1_isp_unregister(struct rkisp1_device *rkisp1);
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
> >>>>>>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt);
> >>>>>>>>>>>>>>      const struct rkisp1_isp_mbus_info *rkisp1_isp_mbus_info_get(u32 mbus_code);
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>      irqreturn_t rkisp1_isp_isr(struct rkisp1_device *rkisp1);
> >>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-isp.c b/drivers/staging/media/rkisp1/rkisp1-isp.c
> >>>>>>>>>>>>>> index dee8e96f3900..6fdf5ed0b6b1 100644
> >>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-isp.c
> >>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-isp.c
> >>>>>>>>>>>>>> @@ -613,6 +613,10 @@ static int rkisp1_isp_enum_mbus_code(struct v4l2_subdev *sd,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>                 if (code->index == pos - 1) {
> >>>>>>>>>>>>>>                         code->code = fmt->mbus_code;
> >>>>>>>>>>>>>> +                    if (fmt->pixel_enc == V4L2_PIXEL_ENC_YUV &&
> >>>>>>>>>>>>>> +                        dir == RKISP1_DIR_SRC)
> >>>>>>>>>>>>>> +                            code->flags =
> >>>>>>>>>>>>>> +                                    V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION;
> >>>>>>>>>>>>>>                         return 0;
> >>>>>>>>>>>>>>                 }
> >>>>>>>>>>>>>>         }
> >>>>>>>>>>>>>> @@ -639,12 +643,21 @@ static int rkisp1_isp_init_config(struct v4l2_subdev *sd,
> >>>>>>>>>>>>>>         sink_crop->height = RKISP1_DEFAULT_HEIGHT;
> >>>>>>>>>>>>>>         sink_crop->left = 0;
> >>>>>>>>>>>>>>         sink_crop->top = 0;
> >>>>>>>>>>>>>> +    sink_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >>>>>>>>>>>>>> +    sink_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(sink_fmt->colorspace);
> >>>>>>>>>>>>>> +    sink_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(sink_fmt->colorspace);
> >>>>>>>>>>>>>> +    sink_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>         src_fmt = v4l2_subdev_get_try_format(sd, cfg,
> >>>>>>>>>>>>>>                                              RKISP1_ISP_PAD_SOURCE_VIDEO);
> >>>>>>>>>>>>>>         *src_fmt = *sink_fmt;
> >>>>>>>>>>>>>>         src_fmt->code = RKISP1_DEF_SRC_PAD_FMT;
> >>>>>>>>>>>>>> -    src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >>>>>>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>>>>>>> +    src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
> >>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>         src_crop = v4l2_subdev_get_try_crop(sd, cfg,
> >>>>>>>>>>>>>>                                             RKISP1_ISP_PAD_SOURCE_VIDEO);
> >>>>>>>>>>>>>> @@ -687,10 +700,17 @@ static void rkisp1_isp_set_src_fmt(struct rkisp1_isp *isp,
> >>>>>>>>>>>>>>                 isp->src_fmt = mbus_info;
> >>>>>>>>>>>>>>         src_fmt->width  = src_crop->width;
> >>>>>>>>>>>>>>         src_fmt->height = src_crop->height;
> >>>>>>>>>>>>>> -    src_fmt->quantization = format->quantization;
> >>>>>>>>>>>>>> -    /* full range by default */
> >>>>>>>>>>>>>> -    if (!src_fmt->quantization)
> >>>>>>>>>>>>>> +
> >>>>>>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >>>>>>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>>>>>>> +
> >>>>>>>>>>>>>> +    if (mbus_info->pixel_enc == V4L2_PIXEL_ENC_BAYER)
> >>>>>>>>>>>>>>                 src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>>> +    else if (format->quantization == V4L2_QUANTIZATION_DEFAULT)
> >>>>>>>>>>>>>> +            src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
> >>>>>>>>>>>>>> +    else
> >>>>>>>>>>>>>> +            src_fmt->quantization = format->quantization;
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>         *format = *src_fmt;
> >>>>>>>>>>>>>>      }
> >>>>>>>>>>>>>> @@ -1068,6 +1088,17 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
> >>>>>>>>>>>>>>         return ret;
> >>>>>>>>>>>>>>      }
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
> >>>>>>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt)
> >>>>>>>>>>>>>> +{
> >>>>>>>>>>>>>> +    struct rkisp1_isp *isp = &rkisp1->isp;
> >>>>>>>>>>>>>> +
> >>>>>>>>>>>>>> +    sd_fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
> >>>>>>>>>>>>>> +    sd_fmt->pad = RKISP1_ISP_PAD_SOURCE_VIDEO;
> >>>>>>>>>>>>>> +
> >>>>>>>>>>>>>> +    return v4l2_subdev_call(&isp->sd, pad, get_fmt, NULL, sd_fmt);
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Do we need to get through the external API to access data that is
> >>>>>>>>>>>>> driver-internal anyway?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> +}
> >>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>      void rkisp1_isp_unregister(struct rkisp1_device *rkisp1)
> >>>>>>>>>>>>>>      {
> >>>>>>>>>>>>>>         struct v4l2_subdev *sd = &rkisp1->isp.sd;
> >>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-resizer.c b/drivers/staging/media/rkisp1/rkisp1-resizer.c
> >>>>>>>>>>>>>> index 7b6b7ddd4169..8705b133de68 100644
> >>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-resizer.c
> >>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-resizer.c
> >>>>>>>>>>>>>> @@ -525,6 +525,7 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
> >>>>>>>>>>>>>>         const struct rkisp1_isp_mbus_info *mbus_info;
> >>>>>>>>>>>>>>         struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
> >>>>>>>>>>>>>>         struct v4l2_rect *sink_crop;
> >>>>>>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>         sink_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SINK, which);
> >>>>>>>>>>>>>>         src_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SRC, which);
> >>>>>>>>>>>>>> @@ -539,8 +540,20 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
> >>>>>>>>>>>>>>         if (which == V4L2_SUBDEV_FORMAT_ACTIVE)
> >>>>>>>>>>>>>>                 rsz->pixel_enc = mbus_info->pixel_enc;
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> +    rkisp1_get_isp_src_fmt(rsz->rkisp1, &isp_sd_fmt);
> >>>>>>>>>>>>>> +
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Is this necessary? My understanding was that in the subdev model, it was
> >>>>>>>>>>>>> the userspace responsibility to propagate any configuration changes through
> >>>>>>>>>>>>> the graph.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Also, doing this only here wouldn't fully maintain the
> >>>>>>>>>>>>> consistency of the state. For example, if one sets the ISP subdev format
> >>>>>>>>>>>>> first, then the resizer subdev and then the ISP subdev again, wouldn't the
> >>>>>>>>>>>>> resizer subdev end up with a wrong format?
> >>>>>>>>>>>>
> >>>>>>>>>>>> yes, this is indeed a bug, I am preparing v4 now.
> >>>>>>>>>>>> What I thought to do is adding quantization conversion
> >>>>>>>>>>>> support also on ther resizer and capture entities.
> >>>>>>>>>>>> Then in the 'link_validation' callbacks, there
> >>>>>>>>>>>> is a validation that the quantization fields matches.
> >>>>>>>>>>>
> >>>>>>>>>>> My understanding is that, if we have the following topology
> >>>>>>>>>>>
> >>>>>>>>>>> [ ISP ] -> [ Resizer ] -> [ Video node ]
> >>>>>>>>>>>
> >>>>>>>>>>> then the ISP source pad would have the csc capability, while resizer
> >>>>>>>>>>> and video node would just accept whatever is configured on their sink
> >>>>>>>>>>> pads (no need for csc capability for that) and propagate that to their
> >>>>>>>>>>> outputs, i.e. resizer source pad and video node CAPTURE format.
> >>>>>>>>>>>
> >>>>>>>>>>> Is this what you were going to implement?
> >>>>>>>>> Hi, I sent a v4 where the CSC capability is set on the reiszer and capture as well.
> >>>>>>>>> I can send a v5 implementing it the way you suggest. Currently the doc says that the colorspace fields
> >>>>>>>>> must be set by the driver for capture streams. This implies that userspace can set it
> >>>>>>>>> only if the CSC is supported.
> >>>>>>>>
> >>>>>>>> Why would the userspace have to set it for the capture stream on the
> >>>>>>>> resizer and video nodes? Couldn't the userspace set it to DEFAULT and
> >>>>>>>> then the driver override to whatever it received on the corresponding
> >>>>>>>> sink?
> >>>>>
> >>>>> According to the documentation [1]:
> >>>>>
> >>>>> "the default quantization encoding as defined by the colorspace"
> >>>>>
> >>>>> Then, for instance, the default for V4L2_COLORSPACE_SRGB is limited range [2].
> >>>>>
> >>>>> Unless we change the definition of DEFAULT to let it accept whatever it receives on the corresponding sink.
> >>>>>
> >>>>> Then the way I understand is:
> >>>>>
> >>>>> [sensor]->[isp]->[resizer]->[capture]
> >>>>>
> >>>>> Userspace should set quantization on:
> >>>>> * sensor source pad
> >>>>> * isp sink pad
> >>>>> * resizer sink pad
> >>>>> * capture node
> >>>>>
> >>>>> The remaining source pads would get values propagated from their sink pads.
> >>>>>
> >>>>> [1] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-defs.html#c.v4l2_quantization
> >>>>> [2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-details.html#col-srgb
> >>>>
> >>>> After discussing this offline, we concluded that:
> >>>> - only the source pad of the ISP entity should expose the CSC capability,
> >>>> - the CAPTURE video node should expose only the formats/color spaces
> >>>> compatible with the current setting of the resizer source,
> >>>> - consistency between the resizer source format and CAPTURE format
> >>>> should be implicitly maintained by the driver.
> >>>
> >>> Do you mean it should be validated at streamon time ?
> >>
> >> Nope. Any V4L2 state must be kept valid all the time, as mandated by
> >> the general V4L2 principles. Wrong configuration of the video node
> >> must not be a reason for STREAMON failing.
> >>
> >>> In MC-based
> >>> drivers, entities are isolated from each other from a configuration
> >>> point of view, and the validity of the configuration across the whole
> >>> pipeline is checked at streamon time. I don't think CSC parameters
> >>> should be treated differently than formats in this regard.
> >>
> >> Indeed, the ISP entity and the resizer entity are isolated from each
> >> other and they need to be validated separately. However, the resizer
> >> entity is not isolated from the video node, as there is no
> >> configuration exposed for the video node in the MC API. If MC exposed
> >> an explicit format control for the video interface sink pad, then the
> >> resizer would have to be validated separately indeed, but still that
> >> video interface sink pad would be tied to the rest of the
> >> configuration of the video node (e.g. the CAPTURE format).
> > 
> > Two questions:
> > 
> > The validation callback between the resizer and the isp has to include check
> > that the quantization values matches ?
> 
> To sumarize:
> 
> The current quantization fields from VIDIOC_SUBDEV_{S,G}_FMT and VIDEO_{SET,GET}_FORMAT
> are filled by the driver (read-only by userspace).
> 
> This RFC exposes an API that allows the quantization to be a read-write configuration.
> The question is where we expose the read-write capability.
> 
> We have two main design choices here.
> 
> 1) Expose read-write quantization in all pads and video nodes.
> And userspace must set matching quantization in all of them.
> Which makes userspace more complicated, but userspace already needs to configure
> the whole topology anyway.
> This wouldn't make much sense in video nodes for metadata (params and statistics)
> 
> In this options, the validation callback should check for matching quantizations
> between all links, and it would failed on STREAMON if formats don't match.
> 
> 2-A) Expose read-write quantization only on the isp source pad.
> The others pads and video nodes would be keept read-only.
> Which means that changing quantization on a read-write node, will implicitly
> change the values on the read-only nodes.
> Which maybe not be a problem, since current API (read-only) already says this field
> is filled by the driver.
> 
> In this options, the validation callback doesn't need to check for matching quantizations
> between links.
> 
> 2-A) Expose read-write quantization only on the video node
> This is basically the same as 2-A, but userspace could interpret that we
> could have different quantizations in each video node.
> 
> 
> I'd like to hear from others what would be better here, design choice 1 or 2.

To keep things simple, I'd go for 2-A, and I would ignore the colorspace
fields completely on the video nodes.

In an MC-centric driver there should be no automatic in-kernel
propagation of userspace-visible configuration from a source pad to the
connected sink pad (regardless of whether the sink is a subdev or a
video node). It's userspace that is responsible for propagating the
configuration, and the kernel is supposed to validate it at stream-on
time.

In this case, the colorspace information is only relevant to the ISP and
nobody else cares (neither the resizer nor the DMA engines) - please
correct me if I'm wrong. Given that userspace will be specific to this
particular hardware, I would not enforce matching colorspace beyond the
output of the ISP. Userspace will deal with colorspace at the output of
the ISP and ignore it on the video node.

> > Userspace have to configure the quantizaion on the resizer sink pad without that CSC
> > API beeing expose on it?
> 
> In the current API, the quantization field is read-only, so to allow userspace to configure
> quantization in any pad, the CSC API needs to be exposed on the pad to make it read-write.
> 
> >>>> Below is the rationale for that:
> >>>> - In the MC model, the video node represents the DMA alone (and thus
> >>>> the reason to have the resizer entities at all) and since the
> >>>> colorspace is not a property of the DMA, it doesn't belong there.
> >>>> - Even if we put that aside, both MP and SP can only output the same
> >>>> colorspace, which would create a dependency between the configuration
> >>>> of the two video nodes, which is not something well defined by V4L2.
> >>>> - A video interface in the MC API doesn't expose control over its
> >>>> input format, which means that the relation between the other side of
> >>>> the link (resizer source) and the video node is implicit and needs to
> >>>> be handled internally in the driver.
> >>>> - V4L2, as opposed to MC, requires that the current state is always
> >>>> consistent and valid. In this case it means that an ioctl on the video
> >>>> node must not be able to alter the state in a way that would make it
> >>>> inconsistent with the state of the matching resizer source.

-- 
Regards,

Laurent Pinchart

Re: [RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad

[Dafna Hirschfeld]

On 30.06.20 01:08, Laurent Pinchart wrote:
> Hi Helen,
> 
> On Mon, Jun 29, 2020 at 12:50:06PM -0300, Helen Koike wrote:
>> On 6/28/20 4:34 PM, Dafna Hirschfeld wrote:
>>> On 28.06.20 16:59, Tomasz Figa wrote:
>>>> On Sun, Jun 28, 2020 at 3:15 PM Laurent Pinchart wrote:
>>>>> On Sun, Jun 28, 2020 at 03:10:41PM +0200, Tomasz Figa wrote:
>>>>>> On Fri, Jun 26, 2020 at 2:15 AM Helen Koike <helen.koike@collabora.com> wrote:
>>>>>>> On 6/18/20 4:25 PM, Tomasz Figa wrote:
>>>>>>>> On Thu, Jun 18, 2020 at 08:54:34PM +0200, Dafna Hirschfeld wrote:
>>>>>>>>> On 18.06.20 20:49, Tomasz Figa wrote:
>>>>>>>>>> On Thu, Jun 18, 2020 at 08:26:33PM +0200, Dafna Hirschfeld wrote:
>>>>>>>>>>> On 18.06.20 19:27, Tomasz Figa wrote:
>>>>>>>>>>>> On Wed, Jun 10, 2020 at 2:08 PM Tomasz Figa wrote:
>>>>>>>>>>>>> On Thu, Jun 4, 2020 at 9:11 PM Dafna Hirschfeld wrote:
>>>>>>>>>>>>>> On 04.06.20 19:54, Tomasz Figa wrote:
>>>>>>>>>>>>>>> On Thu, Apr 16, 2020 at 04:56:05PM +0200, Dafna Hirschfeld wrote:
>>>>>>>>>>>>>>>> The isp entity has a hardware support to force full range quantization
>>>>>>>>>>>>>>>> for YUV formats. Use the new API to indicate userspace that
>>>>>>>>>>>>>>>> quantization conversion is supported by adding the flag
>>>>>>>>>>>>>>>> V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION during media code enumeration.
>>>>>>>>>>>>>>>> Then uppon s_fmt on the video source pad, we assign the
>>>>>>>>>>>>>>>> quantization from userspace for YUV formats.
>>>>>>>>>>>>>>>> Also in the capture and resizer entities we retrieve the colorspace
>>>>>>>>>>>>>>>> from the isp entity.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>       drivers/staging/media/rkisp1/rkisp1-capture.c | 28 ++++++-------
>>>>>>>>>>>>>>>>       drivers/staging/media/rkisp1/rkisp1-common.h  |  2 +
>>>>>>>>>>>>>>>>       drivers/staging/media/rkisp1/rkisp1-isp.c     | 39 +++++++++++++++++--
>>>>>>>>>>>>>>>>       drivers/staging/media/rkisp1/rkisp1-resizer.c | 13 +++++++
>>>>>>>>>>>>>>>>       4 files changed, 65 insertions(+), 17 deletions(-)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>>>>>>>>>>>> index fbf62399fe3d..aca0f93bc772 100644
>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>>>>>>>>>>>> @@ -1066,14 +1066,13 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
>>>>>>>>>>>>>>>>                             const struct v4l2_format_info **fmt_info)
>>>>>>>>>>>>>>>>       {
>>>>>>>>>>>>>>>>          const struct rkisp1_capture_config *config = cap->config;
>>>>>>>>>>>>>>>> -    struct rkisp1_capture *other_cap =
>>>>>>>>>>>>>>>> -                    &cap->rkisp1->capture_devs[cap->id ^ 1];
>>>>>>>>>>>>>>>>          const struct rkisp1_capture_fmt_cfg *fmt;
>>>>>>>>>>>>>>>>          const struct v4l2_format_info *info;
>>>>>>>>>>>>>>>>          const unsigned int max_widths[] = { RKISP1_RSZ_MP_SRC_MAX_WIDTH,
>>>>>>>>>>>>>>>>                                              RKISP1_RSZ_SP_SRC_MAX_WIDTH };
>>>>>>>>>>>>>>>>          const unsigned int max_heights[] = { RKISP1_RSZ_MP_SRC_MAX_HEIGHT,
>>>>>>>>>>>>>>>>                                               RKISP1_RSZ_SP_SRC_MAX_HEIGHT};
>>>>>>>>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>          fmt = rkisp1_find_fmt_cfg(cap, pixm->pixelformat);
>>>>>>>>>>>>>>>>          if (!fmt) {
>>>>>>>>>>>>>>>> @@ -1081,24 +1080,27 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
>>>>>>>>>>>>>>>>                  pixm->pixelformat = fmt->fourcc;
>>>>>>>>>>>>>>>>          }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> +    rkisp1_get_isp_src_fmt(cap->rkisp1, &isp_sd_fmt);
>>>>>>>>>>>>>>>> +    pixm->field = isp_sd_fmt.format.field;
>>>>>>>>>>>>>>>> +    pixm->colorspace = isp_sd_fmt.format.colorspace;
>>>>>>>>>>>>>>>> +    pixm->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
>>>>>>>>>>>>>>>> +    pixm->xfer_func = isp_sd_fmt.format.xfer_func;
>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>> +    /*
>>>>>>>>>>>>>>>> +     * isp has a feature to set full range quantization for yuv formats.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> How about "select between limited and full range for YUV formats"?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> +     * so we need to get the format from the isp.
>>>>>>>>>>>>>>>> +     */
>>>>>>>>>>>>>>>> +    pixm->quantization = isp_sd_fmt.format.quantization;
>>>>>>>>>>>>>>>> +    if (!v4l2_is_format_yuv(cap->pix.info))
>>>>>>>>>>>>>>>> +            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>          pixm->width = clamp_t(u32, pixm->width,
>>>>>>>>>>>>>>>>                                RKISP1_RSZ_SRC_MIN_WIDTH, max_widths[cap->id]);
>>>>>>>>>>>>>>>>          pixm->height = clamp_t(u32, pixm->height,
>>>>>>>>>>>>>>>>                                 RKISP1_RSZ_SRC_MIN_HEIGHT, max_heights[cap->id]);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -    pixm->field = V4L2_FIELD_NONE;
>>>>>>>>>>>>>>>> -    pixm->colorspace = V4L2_COLORSPACE_DEFAULT;
>>>>>>>>>>>>>>>> -    pixm->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>          info = rkisp1_fill_pixfmt(pixm, cap->id);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -    /* can not change quantization when stream-on */
>>>>>>>>>>>>>>>> -    if (other_cap->is_streaming)
>>>>>>>>>>>>>>>> -            pixm->quantization = other_cap->pix.fmt.quantization;
>>>>>>>>>>>>>>>> -    /* output full range by default, take effect in params */
>>>>>>>>>>>>>>>> -    else if (!pixm->quantization ||
>>>>>>>>>>>>>>>> -             pixm->quantization > V4L2_QUANTIZATION_LIM_RANGE)
>>>>>>>>>>>>>>>> -            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>          if (fmt_cfg)
>>>>>>>>>>>>>>>>                  *fmt_cfg = fmt;
>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-common.h b/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>>>>>>>>>>>> index 2d7b7e078636..7a5576fa14c9 100644
>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>>>>>>>>>>>> @@ -300,6 +300,8 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
>>>>>>>>>>>>>>>>                          struct v4l2_device *v4l2_dev);
>>>>>>>>>>>>>>>>       void rkisp1_isp_unregister(struct rkisp1_device *rkisp1);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
>>>>>>>>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt);
>>>>>>>>>>>>>>>>       const struct rkisp1_isp_mbus_info *rkisp1_isp_mbus_info_get(u32 mbus_code);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>       irqreturn_t rkisp1_isp_isr(struct rkisp1_device *rkisp1);
>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-isp.c b/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>>>>>>>>>>>> index dee8e96f3900..6fdf5ed0b6b1 100644
>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>>>>>>>>>>>> @@ -613,6 +613,10 @@ static int rkisp1_isp_enum_mbus_code(struct v4l2_subdev *sd,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                  if (code->index == pos - 1) {
>>>>>>>>>>>>>>>>                          code->code = fmt->mbus_code;
>>>>>>>>>>>>>>>> +                    if (fmt->pixel_enc == V4L2_PIXEL_ENC_YUV &&
>>>>>>>>>>>>>>>> +                        dir == RKISP1_DIR_SRC)
>>>>>>>>>>>>>>>> +                            code->flags =
>>>>>>>>>>>>>>>> +                                    V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION;
>>>>>>>>>>>>>>>>                          return 0;
>>>>>>>>>>>>>>>>                  }
>>>>>>>>>>>>>>>>          }
>>>>>>>>>>>>>>>> @@ -639,12 +643,21 @@ static int rkisp1_isp_init_config(struct v4l2_subdev *sd,
>>>>>>>>>>>>>>>>          sink_crop->height = RKISP1_DEFAULT_HEIGHT;
>>>>>>>>>>>>>>>>          sink_crop->left = 0;
>>>>>>>>>>>>>>>>          sink_crop->top = 0;
>>>>>>>>>>>>>>>> +    sink_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>>>>>>>>>>>> +    sink_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(sink_fmt->colorspace);
>>>>>>>>>>>>>>>> +    sink_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(sink_fmt->colorspace);
>>>>>>>>>>>>>>>> +    sink_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>          src_fmt = v4l2_subdev_get_try_format(sd, cfg,
>>>>>>>>>>>>>>>>                                               RKISP1_ISP_PAD_SOURCE_VIDEO);
>>>>>>>>>>>>>>>>          *src_fmt = *sink_fmt;
>>>>>>>>>>>>>>>>          src_fmt->code = RKISP1_DEF_SRC_PAD_FMT;
>>>>>>>>>>>>>>>> -    src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>>>>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
>>>>>>>>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
>>>>>>>>>>>>>>>> +    src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>          src_crop = v4l2_subdev_get_try_crop(sd, cfg,
>>>>>>>>>>>>>>>>                                              RKISP1_ISP_PAD_SOURCE_VIDEO);
>>>>>>>>>>>>>>>> @@ -687,10 +700,17 @@ static void rkisp1_isp_set_src_fmt(struct rkisp1_isp *isp,
>>>>>>>>>>>>>>>>                  isp->src_fmt = mbus_info;
>>>>>>>>>>>>>>>>          src_fmt->width  = src_crop->width;
>>>>>>>>>>>>>>>>          src_fmt->height = src_crop->height;
>>>>>>>>>>>>>>>> -    src_fmt->quantization = format->quantization;
>>>>>>>>>>>>>>>> -    /* full range by default */
>>>>>>>>>>>>>>>> -    if (!src_fmt->quantization)
>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>>>>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
>>>>>>>>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>> +    if (mbus_info->pixel_enc == V4L2_PIXEL_ENC_BAYER)
>>>>>>>>>>>>>>>>                  src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>>>> +    else if (format->quantization == V4L2_QUANTIZATION_DEFAULT)
>>>>>>>>>>>>>>>> +            src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
>>>>>>>>>>>>>>>> +    else
>>>>>>>>>>>>>>>> +            src_fmt->quantization = format->quantization;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>          *format = *src_fmt;
>>>>>>>>>>>>>>>>       }
>>>>>>>>>>>>>>>> @@ -1068,6 +1088,17 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
>>>>>>>>>>>>>>>>          return ret;
>>>>>>>>>>>>>>>>       }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
>>>>>>>>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt)
>>>>>>>>>>>>>>>> +{
>>>>>>>>>>>>>>>> +    struct rkisp1_isp *isp = &rkisp1->isp;
>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>> +    sd_fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
>>>>>>>>>>>>>>>> +    sd_fmt->pad = RKISP1_ISP_PAD_SOURCE_VIDEO;
>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>> +    return v4l2_subdev_call(&isp->sd, pad, get_fmt, NULL, sd_fmt);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Do we need to get through the external API to access data that is
>>>>>>>>>>>>>>> driver-internal anyway?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> +}
>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>       void rkisp1_isp_unregister(struct rkisp1_device *rkisp1)
>>>>>>>>>>>>>>>>       {
>>>>>>>>>>>>>>>>          struct v4l2_subdev *sd = &rkisp1->isp.sd;
>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-resizer.c b/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>>>>>>>>>>>> index 7b6b7ddd4169..8705b133de68 100644
>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>>>>>>>>>>>> @@ -525,6 +525,7 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
>>>>>>>>>>>>>>>>          const struct rkisp1_isp_mbus_info *mbus_info;
>>>>>>>>>>>>>>>>          struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
>>>>>>>>>>>>>>>>          struct v4l2_rect *sink_crop;
>>>>>>>>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>          sink_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SINK, which);
>>>>>>>>>>>>>>>>          src_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SRC, which);
>>>>>>>>>>>>>>>> @@ -539,8 +540,20 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
>>>>>>>>>>>>>>>>          if (which == V4L2_SUBDEV_FORMAT_ACTIVE)
>>>>>>>>>>>>>>>>                  rsz->pixel_enc = mbus_info->pixel_enc;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> +    rkisp1_get_isp_src_fmt(rsz->rkisp1, &isp_sd_fmt);
>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Is this necessary? My understanding was that in the subdev model, it was
>>>>>>>>>>>>>>> the userspace responsibility to propagate any configuration changes through
>>>>>>>>>>>>>>> the graph.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Also, doing this only here wouldn't fully maintain the
>>>>>>>>>>>>>>> consistency of the state. For example, if one sets the ISP subdev format
>>>>>>>>>>>>>>> first, then the resizer subdev and then the ISP subdev again, wouldn't the
>>>>>>>>>>>>>>> resizer subdev end up with a wrong format?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> yes, this is indeed a bug, I am preparing v4 now.
>>>>>>>>>>>>>> What I thought to do is adding quantization conversion
>>>>>>>>>>>>>> support also on ther resizer and capture entities.
>>>>>>>>>>>>>> Then in the 'link_validation' callbacks, there
>>>>>>>>>>>>>> is a validation that the quantization fields matches.
>>>>>>>>>>>>>
>>>>>>>>>>>>> My understanding is that, if we have the following topology
>>>>>>>>>>>>>
>>>>>>>>>>>>> [ ISP ] -> [ Resizer ] -> [ Video node ]
>>>>>>>>>>>>>
>>>>>>>>>>>>> then the ISP source pad would have the csc capability, while resizer
>>>>>>>>>>>>> and video node would just accept whatever is configured on their sink
>>>>>>>>>>>>> pads (no need for csc capability for that) and propagate that to their
>>>>>>>>>>>>> outputs, i.e. resizer source pad and video node CAPTURE format.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Is this what you were going to implement?
>>>>>>>>>>> Hi, I sent a v4 where the CSC capability is set on the reiszer and capture as well.
>>>>>>>>>>> I can send a v5 implementing it the way you suggest. Currently the doc says that the colorspace fields
>>>>>>>>>>> must be set by the driver for capture streams. This implies that userspace can set it
>>>>>>>>>>> only if the CSC is supported.
>>>>>>>>>>
>>>>>>>>>> Why would the userspace have to set it for the capture stream on the
>>>>>>>>>> resizer and video nodes? Couldn't the userspace set it to DEFAULT and
>>>>>>>>>> then the driver override to whatever it received on the corresponding
>>>>>>>>>> sink?
>>>>>>>
>>>>>>> According to the documentation [1]:
>>>>>>>
>>>>>>> "the default quantization encoding as defined by the colorspace"
>>>>>>>
>>>>>>> Then, for instance, the default for V4L2_COLORSPACE_SRGB is limited range [2].
>>>>>>>
>>>>>>> Unless we change the definition of DEFAULT to let it accept whatever it receives on the corresponding sink.
>>>>>>>
>>>>>>> Then the way I understand is:
>>>>>>>
>>>>>>> [sensor]->[isp]->[resizer]->[capture]
>>>>>>>
>>>>>>> Userspace should set quantization on:
>>>>>>> * sensor source pad
>>>>>>> * isp sink pad
>>>>>>> * resizer sink pad
>>>>>>> * capture node
>>>>>>>
>>>>>>> The remaining source pads would get values propagated from their sink pads.
>>>>>>>
>>>>>>> [1] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-defs.html#c.v4l2_quantization
>>>>>>> [2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-details.html#col-srgb
>>>>>>
>>>>>> After discussing this offline, we concluded that:
>>>>>> - only the source pad of the ISP entity should expose the CSC capability,
>>>>>> - the CAPTURE video node should expose only the formats/color spaces
>>>>>> compatible with the current setting of the resizer source,
>>>>>> - consistency between the resizer source format and CAPTURE format
>>>>>> should be implicitly maintained by the driver.
>>>>>
>>>>> Do you mean it should be validated at streamon time ?
>>>>
>>>> Nope. Any V4L2 state must be kept valid all the time, as mandated by
>>>> the general V4L2 principles. Wrong configuration of the video node
>>>> must not be a reason for STREAMON failing.
>>>>
>>>>> In MC-based
>>>>> drivers, entities are isolated from each other from a configuration
>>>>> point of view, and the validity of the configuration across the whole
>>>>> pipeline is checked at streamon time. I don't think CSC parameters
>>>>> should be treated differently than formats in this regard.
>>>>
>>>> Indeed, the ISP entity and the resizer entity are isolated from each
>>>> other and they need to be validated separately. However, the resizer
>>>> entity is not isolated from the video node, as there is no
>>>> configuration exposed for the video node in the MC API. If MC exposed
>>>> an explicit format control for the video interface sink pad, then the
>>>> resizer would have to be validated separately indeed, but still that
>>>> video interface sink pad would be tied to the rest of the
>>>> configuration of the video node (e.g. the CAPTURE format).
>>>
>>> Two questions:
>>>
>>> The validation callback between the resizer and the isp has to include check
>>> that the quantization values matches ?
>>
>> To sumarize:
>>
>> The current quantization fields from VIDIOC_SUBDEV_{S,G}_FMT and VIDEO_{SET,GET}_FORMAT
>> are filled by the driver (read-only by userspace).
>>
>> This RFC exposes an API that allows the quantization to be a read-write configuration.
>> The question is where we expose the read-write capability.
>>
>> We have two main design choices here.
>>
>> 1) Expose read-write quantization in all pads and video nodes.
>> And userspace must set matching quantization in all of them.
>> Which makes userspace more complicated, but userspace already needs to configure
>> the whole topology anyway.
>> This wouldn't make much sense in video nodes for metadata (params and statistics)
>>
>> In this options, the validation callback should check for matching quantizations
>> between all links, and it would failed on STREAMON if formats don't match.
>>
>> 2-A) Expose read-write quantization only on the isp source pad.
>> The others pads and video nodes would be keept read-only.
>> Which means that changing quantization on a read-write node, will implicitly
>> change the values on the read-only nodes.
>> Which maybe not be a problem, since current API (read-only) already says this field
>> is filled by the driver.
>>
>> In this options, the validation callback doesn't need to check for matching quantizations
>> between links.
>>
>> 2-A) Expose read-write quantization only on the video node
>> This is basically the same as 2-A, but userspace could interpret that we
>> could have different quantizations in each video node.
>>
>>
>> I'd like to hear from others what would be better here, design choice 1 or 2.
> 
> To keep things simple, I'd go for 2-A, and I would ignore the colorspace
> fields completely on the video nodes.
> 
> In an MC-centric driver there should be no automatic in-kernel
> propagation of userspace-visible configuration from a source pad to the
> connected sink pad (regardless of whether the sink is a subdev or a
> video node). It's userspace that is responsible for propagating the
> configuration, and the kernel is supposed to validate it at stream-on
> time.
> 
> In this case, the colorspace information is only relevant to the ISP and
> nobody else cares (neither the resizer nor the DMA engines) - please
> correct me if I'm wrong. Given that userspace will be specific to this
> particular hardware, I would not enforce matching colorspace beyond the
> output of the ISP. Userspace will deal with colorspace at the output of
> the ISP and ignore it on the video node.

So you mean that userspace should know that the quantization of the video
is the one configured on the isp, and not 'Default Quantization' as would
be reported by the resizers captures?

Thanks,
Dafna

> 
>>> Userspace have to configure the quantizaion on the resizer sink pad without that CSC
>>> API beeing expose on it?
>>
>> In the current API, the quantization field is read-only, so to allow userspace to configure
>> quantization in any pad, the CSC API needs to be exposed on the pad to make it read-write.
>>
>>>>>> Below is the rationale for that:
>>>>>> - In the MC model, the video node represents the DMA alone (and thus
>>>>>> the reason to have the resizer entities at all) and since the
>>>>>> colorspace is not a property of the DMA, it doesn't belong there.
>>>>>> - Even if we put that aside, both MP and SP can only output the same
>>>>>> colorspace, which would create a dependency between the configuration
>>>>>> of the two video nodes, which is not something well defined by V4L2.
>>>>>> - A video interface in the MC API doesn't expose control over its
>>>>>> input format, which means that the relation between the other side of
>>>>>> the link (resizer source) and the video node is implicit and needs to
>>>>>> be handled internally in the driver.
>>>>>> - V4L2, as opposed to MC, requires that the current state is always
>>>>>> consistent and valid. In this case it means that an ioctl on the video
>>>>>> node must not be able to alter the state in a way that would make it
>>>>>> inconsistent with the state of the matching resizer source.
> 

Re: [RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad

[Laurent Pinchart]

Hi Dafna,

On Tue, Jun 30, 2020 at 11:36:35AM +0200, Dafna Hirschfeld wrote:
> On 30.06.20 01:08, Laurent Pinchart wrote:
> > On Mon, Jun 29, 2020 at 12:50:06PM -0300, Helen Koike wrote:
> >> On 6/28/20 4:34 PM, Dafna Hirschfeld wrote:
> >>> On 28.06.20 16:59, Tomasz Figa wrote:
> >>>> On Sun, Jun 28, 2020 at 3:15 PM Laurent Pinchart wrote:
> >>>>> On Sun, Jun 28, 2020 at 03:10:41PM +0200, Tomasz Figa wrote:
> >>>>>> On Fri, Jun 26, 2020 at 2:15 AM Helen Koike <helen.koike@collabora.com> wrote:
> >>>>>>> On 6/18/20 4:25 PM, Tomasz Figa wrote:
> >>>>>>>> On Thu, Jun 18, 2020 at 08:54:34PM +0200, Dafna Hirschfeld wrote:
> >>>>>>>>> On 18.06.20 20:49, Tomasz Figa wrote:
> >>>>>>>>>> On Thu, Jun 18, 2020 at 08:26:33PM +0200, Dafna Hirschfeld wrote:
> >>>>>>>>>>> On 18.06.20 19:27, Tomasz Figa wrote:
> >>>>>>>>>>>> On Wed, Jun 10, 2020 at 2:08 PM Tomasz Figa wrote:
> >>>>>>>>>>>>> On Thu, Jun 4, 2020 at 9:11 PM Dafna Hirschfeld wrote:
> >>>>>>>>>>>>>> On 04.06.20 19:54, Tomasz Figa wrote:
> >>>>>>>>>>>>>>> On Thu, Apr 16, 2020 at 04:56:05PM +0200, Dafna Hirschfeld wrote:
> >>>>>>>>>>>>>>>> The isp entity has a hardware support to force full range quantization
> >>>>>>>>>>>>>>>> for YUV formats. Use the new API to indicate userspace that
> >>>>>>>>>>>>>>>> quantization conversion is supported by adding the flag
> >>>>>>>>>>>>>>>> V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION during media code enumeration.
> >>>>>>>>>>>>>>>> Then uppon s_fmt on the video source pad, we assign the
> >>>>>>>>>>>>>>>> quantization from userspace for YUV formats.
> >>>>>>>>>>>>>>>> Also in the capture and resizer entities we retrieve the colorspace
> >>>>>>>>>>>>>>>> from the isp entity.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
> >>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>       drivers/staging/media/rkisp1/rkisp1-capture.c | 28 ++++++-------
> >>>>>>>>>>>>>>>>       drivers/staging/media/rkisp1/rkisp1-common.h  |  2 +
> >>>>>>>>>>>>>>>>       drivers/staging/media/rkisp1/rkisp1-isp.c     | 39 +++++++++++++++++--
> >>>>>>>>>>>>>>>>       drivers/staging/media/rkisp1/rkisp1-resizer.c | 13 +++++++
> >>>>>>>>>>>>>>>>       4 files changed, 65 insertions(+), 17 deletions(-)
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c
> >>>>>>>>>>>>>>>> index fbf62399fe3d..aca0f93bc772 100644
> >>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-capture.c
> >>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-capture.c
> >>>>>>>>>>>>>>>> @@ -1066,14 +1066,13 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
> >>>>>>>>>>>>>>>>                             const struct v4l2_format_info **fmt_info)
> >>>>>>>>>>>>>>>>       {
> >>>>>>>>>>>>>>>>          const struct rkisp1_capture_config *config = cap->config;
> >>>>>>>>>>>>>>>> -    struct rkisp1_capture *other_cap =
> >>>>>>>>>>>>>>>> -                    &cap->rkisp1->capture_devs[cap->id ^ 1];
> >>>>>>>>>>>>>>>>          const struct rkisp1_capture_fmt_cfg *fmt;
> >>>>>>>>>>>>>>>>          const struct v4l2_format_info *info;
> >>>>>>>>>>>>>>>>          const unsigned int max_widths[] = { RKISP1_RSZ_MP_SRC_MAX_WIDTH,
> >>>>>>>>>>>>>>>>                                              RKISP1_RSZ_SP_SRC_MAX_WIDTH };
> >>>>>>>>>>>>>>>>          const unsigned int max_heights[] = { RKISP1_RSZ_MP_SRC_MAX_HEIGHT,
> >>>>>>>>>>>>>>>>                                               RKISP1_RSZ_SP_SRC_MAX_HEIGHT};
> >>>>>>>>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>          fmt = rkisp1_find_fmt_cfg(cap, pixm->pixelformat);
> >>>>>>>>>>>>>>>>          if (!fmt) {
> >>>>>>>>>>>>>>>> @@ -1081,24 +1080,27 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
> >>>>>>>>>>>>>>>>                  pixm->pixelformat = fmt->fourcc;
> >>>>>>>>>>>>>>>>          }
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> +    rkisp1_get_isp_src_fmt(cap->rkisp1, &isp_sd_fmt);
> >>>>>>>>>>>>>>>> +    pixm->field = isp_sd_fmt.format.field;
> >>>>>>>>>>>>>>>> +    pixm->colorspace = isp_sd_fmt.format.colorspace;
> >>>>>>>>>>>>>>>> +    pixm->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
> >>>>>>>>>>>>>>>> +    pixm->xfer_func = isp_sd_fmt.format.xfer_func;
> >>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>> +    /*
> >>>>>>>>>>>>>>>> +     * isp has a feature to set full range quantization for yuv formats.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> How about "select between limited and full range for YUV formats"?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> +     * so we need to get the format from the isp.
> >>>>>>>>>>>>>>>> +     */
> >>>>>>>>>>>>>>>> +    pixm->quantization = isp_sd_fmt.format.quantization;
> >>>>>>>>>>>>>>>> +    if (!v4l2_is_format_yuv(cap->pix.info))
> >>>>>>>>>>>>>>>> +            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>          pixm->width = clamp_t(u32, pixm->width,
> >>>>>>>>>>>>>>>>                                RKISP1_RSZ_SRC_MIN_WIDTH, max_widths[cap->id]);
> >>>>>>>>>>>>>>>>          pixm->height = clamp_t(u32, pixm->height,
> >>>>>>>>>>>>>>>>                                 RKISP1_RSZ_SRC_MIN_HEIGHT, max_heights[cap->id]);
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> -    pixm->field = V4L2_FIELD_NONE;
> >>>>>>>>>>>>>>>> -    pixm->colorspace = V4L2_COLORSPACE_DEFAULT;
> >>>>>>>>>>>>>>>> -    pixm->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> >>>>>>>>>>>>>>>> -
> >>>>>>>>>>>>>>>>          info = rkisp1_fill_pixfmt(pixm, cap->id);
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> -    /* can not change quantization when stream-on */
> >>>>>>>>>>>>>>>> -    if (other_cap->is_streaming)
> >>>>>>>>>>>>>>>> -            pixm->quantization = other_cap->pix.fmt.quantization;
> >>>>>>>>>>>>>>>> -    /* output full range by default, take effect in params */
> >>>>>>>>>>>>>>>> -    else if (!pixm->quantization ||
> >>>>>>>>>>>>>>>> -             pixm->quantization > V4L2_QUANTIZATION_LIM_RANGE)
> >>>>>>>>>>>>>>>> -            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>          if (fmt_cfg)
> >>>>>>>>>>>>>>>>                  *fmt_cfg = fmt;
> >>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-common.h b/drivers/staging/media/rkisp1/rkisp1-common.h
> >>>>>>>>>>>>>>>> index 2d7b7e078636..7a5576fa14c9 100644
> >>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-common.h
> >>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-common.h
> >>>>>>>>>>>>>>>> @@ -300,6 +300,8 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
> >>>>>>>>>>>>>>>>                          struct v4l2_device *v4l2_dev);
> >>>>>>>>>>>>>>>>       void rkisp1_isp_unregister(struct rkisp1_device *rkisp1);
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
> >>>>>>>>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt);
> >>>>>>>>>>>>>>>>       const struct rkisp1_isp_mbus_info *rkisp1_isp_mbus_info_get(u32 mbus_code);
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>       irqreturn_t rkisp1_isp_isr(struct rkisp1_device *rkisp1);
> >>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-isp.c b/drivers/staging/media/rkisp1/rkisp1-isp.c
> >>>>>>>>>>>>>>>> index dee8e96f3900..6fdf5ed0b6b1 100644
> >>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-isp.c
> >>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-isp.c
> >>>>>>>>>>>>>>>> @@ -613,6 +613,10 @@ static int rkisp1_isp_enum_mbus_code(struct v4l2_subdev *sd,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>                  if (code->index == pos - 1) {
> >>>>>>>>>>>>>>>>                          code->code = fmt->mbus_code;
> >>>>>>>>>>>>>>>> +                    if (fmt->pixel_enc == V4L2_PIXEL_ENC_YUV &&
> >>>>>>>>>>>>>>>> +                        dir == RKISP1_DIR_SRC)
> >>>>>>>>>>>>>>>> +                            code->flags =
> >>>>>>>>>>>>>>>> +                                    V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION;
> >>>>>>>>>>>>>>>>                          return 0;
> >>>>>>>>>>>>>>>>                  }
> >>>>>>>>>>>>>>>>          }
> >>>>>>>>>>>>>>>> @@ -639,12 +643,21 @@ static int rkisp1_isp_init_config(struct v4l2_subdev *sd,
> >>>>>>>>>>>>>>>>          sink_crop->height = RKISP1_DEFAULT_HEIGHT;
> >>>>>>>>>>>>>>>>          sink_crop->left = 0;
> >>>>>>>>>>>>>>>>          sink_crop->top = 0;
> >>>>>>>>>>>>>>>> +    sink_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >>>>>>>>>>>>>>>> +    sink_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(sink_fmt->colorspace);
> >>>>>>>>>>>>>>>> +    sink_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(sink_fmt->colorspace);
> >>>>>>>>>>>>>>>> +    sink_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>          src_fmt = v4l2_subdev_get_try_format(sd, cfg,
> >>>>>>>>>>>>>>>>                                               RKISP1_ISP_PAD_SOURCE_VIDEO);
> >>>>>>>>>>>>>>>>          *src_fmt = *sink_fmt;
> >>>>>>>>>>>>>>>>          src_fmt->code = RKISP1_DEF_SRC_PAD_FMT;
> >>>>>>>>>>>>>>>> -    src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >>>>>>>>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>>>>>>>>> +    src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
> >>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>          src_crop = v4l2_subdev_get_try_crop(sd, cfg,
> >>>>>>>>>>>>>>>>                                              RKISP1_ISP_PAD_SOURCE_VIDEO);
> >>>>>>>>>>>>>>>> @@ -687,10 +700,17 @@ static void rkisp1_isp_set_src_fmt(struct rkisp1_isp *isp,
> >>>>>>>>>>>>>>>>                  isp->src_fmt = mbus_info;
> >>>>>>>>>>>>>>>>          src_fmt->width  = src_crop->width;
> >>>>>>>>>>>>>>>>          src_fmt->height = src_crop->height;
> >>>>>>>>>>>>>>>> -    src_fmt->quantization = format->quantization;
> >>>>>>>>>>>>>>>> -    /* full range by default */
> >>>>>>>>>>>>>>>> -    if (!src_fmt->quantization)
> >>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >>>>>>>>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>> +    if (mbus_info->pixel_enc == V4L2_PIXEL_ENC_BAYER)
> >>>>>>>>>>>>>>>>                  src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>>>>> +    else if (format->quantization == V4L2_QUANTIZATION_DEFAULT)
> >>>>>>>>>>>>>>>> +            src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
> >>>>>>>>>>>>>>>> +    else
> >>>>>>>>>>>>>>>> +            src_fmt->quantization = format->quantization;
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>          *format = *src_fmt;
> >>>>>>>>>>>>>>>>       }
> >>>>>>>>>>>>>>>> @@ -1068,6 +1088,17 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
> >>>>>>>>>>>>>>>>          return ret;
> >>>>>>>>>>>>>>>>       }
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
> >>>>>>>>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt)
> >>>>>>>>>>>>>>>> +{
> >>>>>>>>>>>>>>>> +    struct rkisp1_isp *isp = &rkisp1->isp;
> >>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>> +    sd_fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
> >>>>>>>>>>>>>>>> +    sd_fmt->pad = RKISP1_ISP_PAD_SOURCE_VIDEO;
> >>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>> +    return v4l2_subdev_call(&isp->sd, pad, get_fmt, NULL, sd_fmt);
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Do we need to get through the external API to access data that is
> >>>>>>>>>>>>>>> driver-internal anyway?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> +}
> >>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>       void rkisp1_isp_unregister(struct rkisp1_device *rkisp1)
> >>>>>>>>>>>>>>>>       {
> >>>>>>>>>>>>>>>>          struct v4l2_subdev *sd = &rkisp1->isp.sd;
> >>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-resizer.c b/drivers/staging/media/rkisp1/rkisp1-resizer.c
> >>>>>>>>>>>>>>>> index 7b6b7ddd4169..8705b133de68 100644
> >>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-resizer.c
> >>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-resizer.c
> >>>>>>>>>>>>>>>> @@ -525,6 +525,7 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
> >>>>>>>>>>>>>>>>          const struct rkisp1_isp_mbus_info *mbus_info;
> >>>>>>>>>>>>>>>>          struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
> >>>>>>>>>>>>>>>>          struct v4l2_rect *sink_crop;
> >>>>>>>>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>          sink_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SINK, which);
> >>>>>>>>>>>>>>>>          src_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SRC, which);
> >>>>>>>>>>>>>>>> @@ -539,8 +540,20 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
> >>>>>>>>>>>>>>>>          if (which == V4L2_SUBDEV_FORMAT_ACTIVE)
> >>>>>>>>>>>>>>>>                  rsz->pixel_enc = mbus_info->pixel_enc;
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> +    rkisp1_get_isp_src_fmt(rsz->rkisp1, &isp_sd_fmt);
> >>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Is this necessary? My understanding was that in the subdev model, it was
> >>>>>>>>>>>>>>> the userspace responsibility to propagate any configuration changes through
> >>>>>>>>>>>>>>> the graph.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Also, doing this only here wouldn't fully maintain the
> >>>>>>>>>>>>>>> consistency of the state. For example, if one sets the ISP subdev format
> >>>>>>>>>>>>>>> first, then the resizer subdev and then the ISP subdev again, wouldn't the
> >>>>>>>>>>>>>>> resizer subdev end up with a wrong format?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> yes, this is indeed a bug, I am preparing v4 now.
> >>>>>>>>>>>>>> What I thought to do is adding quantization conversion
> >>>>>>>>>>>>>> support also on ther resizer and capture entities.
> >>>>>>>>>>>>>> Then in the 'link_validation' callbacks, there
> >>>>>>>>>>>>>> is a validation that the quantization fields matches.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> My understanding is that, if we have the following topology
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> [ ISP ] -> [ Resizer ] -> [ Video node ]
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> then the ISP source pad would have the csc capability, while resizer
> >>>>>>>>>>>>> and video node would just accept whatever is configured on their sink
> >>>>>>>>>>>>> pads (no need for csc capability for that) and propagate that to their
> >>>>>>>>>>>>> outputs, i.e. resizer source pad and video node CAPTURE format.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Is this what you were going to implement?
> >>>>>>>>>>> Hi, I sent a v4 where the CSC capability is set on the reiszer and capture as well.
> >>>>>>>>>>> I can send a v5 implementing it the way you suggest. Currently the doc says that the colorspace fields
> >>>>>>>>>>> must be set by the driver for capture streams. This implies that userspace can set it
> >>>>>>>>>>> only if the CSC is supported.
> >>>>>>>>>>
> >>>>>>>>>> Why would the userspace have to set it for the capture stream on the
> >>>>>>>>>> resizer and video nodes? Couldn't the userspace set it to DEFAULT and
> >>>>>>>>>> then the driver override to whatever it received on the corresponding
> >>>>>>>>>> sink?
> >>>>>>>
> >>>>>>> According to the documentation [1]:
> >>>>>>>
> >>>>>>> "the default quantization encoding as defined by the colorspace"
> >>>>>>>
> >>>>>>> Then, for instance, the default for V4L2_COLORSPACE_SRGB is limited range [2].
> >>>>>>>
> >>>>>>> Unless we change the definition of DEFAULT to let it accept whatever it receives on the corresponding sink.
> >>>>>>>
> >>>>>>> Then the way I understand is:
> >>>>>>>
> >>>>>>> [sensor]->[isp]->[resizer]->[capture]
> >>>>>>>
> >>>>>>> Userspace should set quantization on:
> >>>>>>> * sensor source pad
> >>>>>>> * isp sink pad
> >>>>>>> * resizer sink pad
> >>>>>>> * capture node
> >>>>>>>
> >>>>>>> The remaining source pads would get values propagated from their sink pads.
> >>>>>>>
> >>>>>>> [1] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-defs.html#c.v4l2_quantization
> >>>>>>> [2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-details.html#col-srgb
> >>>>>>
> >>>>>> After discussing this offline, we concluded that:
> >>>>>> - only the source pad of the ISP entity should expose the CSC capability,
> >>>>>> - the CAPTURE video node should expose only the formats/color spaces
> >>>>>> compatible with the current setting of the resizer source,
> >>>>>> - consistency between the resizer source format and CAPTURE format
> >>>>>> should be implicitly maintained by the driver.
> >>>>>
> >>>>> Do you mean it should be validated at streamon time ?
> >>>>
> >>>> Nope. Any V4L2 state must be kept valid all the time, as mandated by
> >>>> the general V4L2 principles. Wrong configuration of the video node
> >>>> must not be a reason for STREAMON failing.
> >>>>
> >>>>> In MC-based
> >>>>> drivers, entities are isolated from each other from a configuration
> >>>>> point of view, and the validity of the configuration across the whole
> >>>>> pipeline is checked at streamon time. I don't think CSC parameters
> >>>>> should be treated differently than formats in this regard.
> >>>>
> >>>> Indeed, the ISP entity and the resizer entity are isolated from each
> >>>> other and they need to be validated separately. However, the resizer
> >>>> entity is not isolated from the video node, as there is no
> >>>> configuration exposed for the video node in the MC API. If MC exposed
> >>>> an explicit format control for the video interface sink pad, then the
> >>>> resizer would have to be validated separately indeed, but still that
> >>>> video interface sink pad would be tied to the rest of the
> >>>> configuration of the video node (e.g. the CAPTURE format).
> >>>
> >>> Two questions:
> >>>
> >>> The validation callback between the resizer and the isp has to include check
> >>> that the quantization values matches ?
> >>
> >> To sumarize:
> >>
> >> The current quantization fields from VIDIOC_SUBDEV_{S,G}_FMT and VIDEO_{SET,GET}_FORMAT
> >> are filled by the driver (read-only by userspace).
> >>
> >> This RFC exposes an API that allows the quantization to be a read-write configuration.
> >> The question is where we expose the read-write capability.
> >>
> >> We have two main design choices here.
> >>
> >> 1) Expose read-write quantization in all pads and video nodes.
> >> And userspace must set matching quantization in all of them.
> >> Which makes userspace more complicated, but userspace already needs to configure
> >> the whole topology anyway.
> >> This wouldn't make much sense in video nodes for metadata (params and statistics)
> >>
> >> In this options, the validation callback should check for matching quantizations
> >> between all links, and it would failed on STREAMON if formats don't match.
> >>
> >> 2-A) Expose read-write quantization only on the isp source pad.
> >> The others pads and video nodes would be keept read-only.
> >> Which means that changing quantization on a read-write node, will implicitly
> >> change the values on the read-only nodes.
> >> Which maybe not be a problem, since current API (read-only) already says this field
> >> is filled by the driver.
> >>
> >> In this options, the validation callback doesn't need to check for matching quantizations
> >> between links.
> >>
> >> 2-A) Expose read-write quantization only on the video node
> >> This is basically the same as 2-A, but userspace could interpret that we
> >> could have different quantizations in each video node.
> >>
> >>
> >> I'd like to hear from others what would be better here, design choice 1 or 2.
> > 
> > To keep things simple, I'd go for 2-A, and I would ignore the colorspace
> > fields completely on the video nodes.
> > 
> > In an MC-centric driver there should be no automatic in-kernel
> > propagation of userspace-visible configuration from a source pad to the
> > connected sink pad (regardless of whether the sink is a subdev or a
> > video node). It's userspace that is responsible for propagating the
> > configuration, and the kernel is supposed to validate it at stream-on
> > time.
> > 
> > In this case, the colorspace information is only relevant to the ISP and
> > nobody else cares (neither the resizer nor the DMA engines) - please
> > correct me if I'm wrong. Given that userspace will be specific to this
> > particular hardware, I would not enforce matching colorspace beyond the
> > output of the ISP. Userspace will deal with colorspace at the output of
> > the ISP and ignore it on the video node.
> 
> So you mean that userspace should know that the quantization of the video
> is the one configured on the isp, and not 'Default Quantization' as would
> be reported by the resizers captures?

Yes. The Rockchip ISP requires device-specific userspace, so requiring
it to know about the driver doesn't seem a problem to me. I however
expect some people to disagree :-) We should however record this in
documentation somewhere, requiring userspace developers to read the
rkisp1 documentation is one thing, requiring them to read the driver
source code to figure out how to use it would be going one step too far
I think.

> >>> Userspace have to configure the quantizaion on the resizer sink pad without that CSC
> >>> API beeing expose on it?
> >>
> >> In the current API, the quantization field is read-only, so to allow userspace to configure
> >> quantization in any pad, the CSC API needs to be exposed on the pad to make it read-write.
> >>
> >>>>>> Below is the rationale for that:
> >>>>>> - In the MC model, the video node represents the DMA alone (and thus
> >>>>>> the reason to have the resizer entities at all) and since the
> >>>>>> colorspace is not a property of the DMA, it doesn't belong there.
> >>>>>> - Even if we put that aside, both MP and SP can only output the same
> >>>>>> colorspace, which would create a dependency between the configuration
> >>>>>> of the two video nodes, which is not something well defined by V4L2.
> >>>>>> - A video interface in the MC API doesn't expose control over its
> >>>>>> input format, which means that the relation between the other side of
> >>>>>> the link (resizer source) and the video node is implicit and needs to
> >>>>>> be handled internally in the driver.
> >>>>>> - V4L2, as opposed to MC, requires that the current state is always
> >>>>>> consistent and valid. In this case it means that an ioctl on the video
> >>>>>> node must not be able to alter the state in a way that would make it
> >>>>>> inconsistent with the state of the matching resizer source.

-- 
Regards,

Laurent Pinchart

Re: [RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad

[Dafna Hirschfeld]

On 01.07.20 14:52, Laurent Pinchart wrote:
> Hi Dafna,
> 
> On Tue, Jun 30, 2020 at 11:36:35AM +0200, Dafna Hirschfeld wrote:
>> On 30.06.20 01:08, Laurent Pinchart wrote:
>>> On Mon, Jun 29, 2020 at 12:50:06PM -0300, Helen Koike wrote:
>>>> On 6/28/20 4:34 PM, Dafna Hirschfeld wrote:
>>>>> On 28.06.20 16:59, Tomasz Figa wrote:
>>>>>> On Sun, Jun 28, 2020 at 3:15 PM Laurent Pinchart wrote:
>>>>>>> On Sun, Jun 28, 2020 at 03:10:41PM +0200, Tomasz Figa wrote:
>>>>>>>> On Fri, Jun 26, 2020 at 2:15 AM Helen Koike <helen.koike@collabora.com> wrote:
>>>>>>>>> On 6/18/20 4:25 PM, Tomasz Figa wrote:
>>>>>>>>>> On Thu, Jun 18, 2020 at 08:54:34PM +0200, Dafna Hirschfeld wrote:
>>>>>>>>>>> On 18.06.20 20:49, Tomasz Figa wrote:
>>>>>>>>>>>> On Thu, Jun 18, 2020 at 08:26:33PM +0200, Dafna Hirschfeld wrote:
>>>>>>>>>>>>> On 18.06.20 19:27, Tomasz Figa wrote:
>>>>>>>>>>>>>> On Wed, Jun 10, 2020 at 2:08 PM Tomasz Figa wrote:
>>>>>>>>>>>>>>> On Thu, Jun 4, 2020 at 9:11 PM Dafna Hirschfeld wrote:
>>>>>>>>>>>>>>>> On 04.06.20 19:54, Tomasz Figa wrote:
>>>>>>>>>>>>>>>>> On Thu, Apr 16, 2020 at 04:56:05PM +0200, Dafna Hirschfeld wrote:
>>>>>>>>>>>>>>>>>> The isp entity has a hardware support to force full range quantization
>>>>>>>>>>>>>>>>>> for YUV formats. Use the new API to indicate userspace that
>>>>>>>>>>>>>>>>>> quantization conversion is supported by adding the flag
>>>>>>>>>>>>>>>>>> V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION during media code enumeration.
>>>>>>>>>>>>>>>>>> Then uppon s_fmt on the video source pad, we assign the
>>>>>>>>>>>>>>>>>> quantization from userspace for YUV formats.
>>>>>>>>>>>>>>>>>> Also in the capture and resizer entities we retrieve the colorspace
>>>>>>>>>>>>>>>>>> from the isp entity.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>        drivers/staging/media/rkisp1/rkisp1-capture.c | 28 ++++++-------
>>>>>>>>>>>>>>>>>>        drivers/staging/media/rkisp1/rkisp1-common.h  |  2 +
>>>>>>>>>>>>>>>>>>        drivers/staging/media/rkisp1/rkisp1-isp.c     | 39 +++++++++++++++++--
>>>>>>>>>>>>>>>>>>        drivers/staging/media/rkisp1/rkisp1-resizer.c | 13 +++++++
>>>>>>>>>>>>>>>>>>        4 files changed, 65 insertions(+), 17 deletions(-)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>>>>>>>>>>>>>> index fbf62399fe3d..aca0f93bc772 100644
>>>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>>>>>>>>>>>>>> @@ -1066,14 +1066,13 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
>>>>>>>>>>>>>>>>>>                              const struct v4l2_format_info **fmt_info)
>>>>>>>>>>>>>>>>>>        {
>>>>>>>>>>>>>>>>>>           const struct rkisp1_capture_config *config = cap->config;
>>>>>>>>>>>>>>>>>> -    struct rkisp1_capture *other_cap =
>>>>>>>>>>>>>>>>>> -                    &cap->rkisp1->capture_devs[cap->id ^ 1];
>>>>>>>>>>>>>>>>>>           const struct rkisp1_capture_fmt_cfg *fmt;
>>>>>>>>>>>>>>>>>>           const struct v4l2_format_info *info;
>>>>>>>>>>>>>>>>>>           const unsigned int max_widths[] = { RKISP1_RSZ_MP_SRC_MAX_WIDTH,
>>>>>>>>>>>>>>>>>>                                               RKISP1_RSZ_SP_SRC_MAX_WIDTH };
>>>>>>>>>>>>>>>>>>           const unsigned int max_heights[] = { RKISP1_RSZ_MP_SRC_MAX_HEIGHT,
>>>>>>>>>>>>>>>>>>                                                RKISP1_RSZ_SP_SRC_MAX_HEIGHT};
>>>>>>>>>>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>           fmt = rkisp1_find_fmt_cfg(cap, pixm->pixelformat);
>>>>>>>>>>>>>>>>>>           if (!fmt) {
>>>>>>>>>>>>>>>>>> @@ -1081,24 +1080,27 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
>>>>>>>>>>>>>>>>>>                   pixm->pixelformat = fmt->fourcc;
>>>>>>>>>>>>>>>>>>           }
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> +    rkisp1_get_isp_src_fmt(cap->rkisp1, &isp_sd_fmt);
>>>>>>>>>>>>>>>>>> +    pixm->field = isp_sd_fmt.format.field;
>>>>>>>>>>>>>>>>>> +    pixm->colorspace = isp_sd_fmt.format.colorspace;
>>>>>>>>>>>>>>>>>> +    pixm->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
>>>>>>>>>>>>>>>>>> +    pixm->xfer_func = isp_sd_fmt.format.xfer_func;
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>> +    /*
>>>>>>>>>>>>>>>>>> +     * isp has a feature to set full range quantization for yuv formats.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> How about "select between limited and full range for YUV formats"?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> +     * so we need to get the format from the isp.
>>>>>>>>>>>>>>>>>> +     */
>>>>>>>>>>>>>>>>>> +    pixm->quantization = isp_sd_fmt.format.quantization;
>>>>>>>>>>>>>>>>>> +    if (!v4l2_is_format_yuv(cap->pix.info))
>>>>>>>>>>>>>>>>>> +            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>           pixm->width = clamp_t(u32, pixm->width,
>>>>>>>>>>>>>>>>>>                                 RKISP1_RSZ_SRC_MIN_WIDTH, max_widths[cap->id]);
>>>>>>>>>>>>>>>>>>           pixm->height = clamp_t(u32, pixm->height,
>>>>>>>>>>>>>>>>>>                                  RKISP1_RSZ_SRC_MIN_HEIGHT, max_heights[cap->id]);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -    pixm->field = V4L2_FIELD_NONE;
>>>>>>>>>>>>>>>>>> -    pixm->colorspace = V4L2_COLORSPACE_DEFAULT;
>>>>>>>>>>>>>>>>>> -    pixm->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>>>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>>>           info = rkisp1_fill_pixfmt(pixm, cap->id);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -    /* can not change quantization when stream-on */
>>>>>>>>>>>>>>>>>> -    if (other_cap->is_streaming)
>>>>>>>>>>>>>>>>>> -            pixm->quantization = other_cap->pix.fmt.quantization;
>>>>>>>>>>>>>>>>>> -    /* output full range by default, take effect in params */
>>>>>>>>>>>>>>>>>> -    else if (!pixm->quantization ||
>>>>>>>>>>>>>>>>>> -             pixm->quantization > V4L2_QUANTIZATION_LIM_RANGE)
>>>>>>>>>>>>>>>>>> -            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>           if (fmt_cfg)
>>>>>>>>>>>>>>>>>>                   *fmt_cfg = fmt;
>>>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-common.h b/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>>>>>>>>>>>>>> index 2d7b7e078636..7a5576fa14c9 100644
>>>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>>>>>>>>>>>>>> @@ -300,6 +300,8 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
>>>>>>>>>>>>>>>>>>                           struct v4l2_device *v4l2_dev);
>>>>>>>>>>>>>>>>>>        void rkisp1_isp_unregister(struct rkisp1_device *rkisp1);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
>>>>>>>>>>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt);
>>>>>>>>>>>>>>>>>>        const struct rkisp1_isp_mbus_info *rkisp1_isp_mbus_info_get(u32 mbus_code);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>        irqreturn_t rkisp1_isp_isr(struct rkisp1_device *rkisp1);
>>>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-isp.c b/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>>>>>>>>>>>>>> index dee8e96f3900..6fdf5ed0b6b1 100644
>>>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>>>>>>>>>>>>>> @@ -613,6 +613,10 @@ static int rkisp1_isp_enum_mbus_code(struct v4l2_subdev *sd,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                   if (code->index == pos - 1) {
>>>>>>>>>>>>>>>>>>                           code->code = fmt->mbus_code;
>>>>>>>>>>>>>>>>>> +                    if (fmt->pixel_enc == V4L2_PIXEL_ENC_YUV &&
>>>>>>>>>>>>>>>>>> +                        dir == RKISP1_DIR_SRC)
>>>>>>>>>>>>>>>>>> +                            code->flags =
>>>>>>>>>>>>>>>>>> +                                    V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION;
>>>>>>>>>>>>>>>>>>                           return 0;
>>>>>>>>>>>>>>>>>>                   }
>>>>>>>>>>>>>>>>>>           }
>>>>>>>>>>>>>>>>>> @@ -639,12 +643,21 @@ static int rkisp1_isp_init_config(struct v4l2_subdev *sd,
>>>>>>>>>>>>>>>>>>           sink_crop->height = RKISP1_DEFAULT_HEIGHT;
>>>>>>>>>>>>>>>>>>           sink_crop->left = 0;
>>>>>>>>>>>>>>>>>>           sink_crop->top = 0;
>>>>>>>>>>>>>>>>>> +    sink_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>>>>>>>>>>>>>> +    sink_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(sink_fmt->colorspace);
>>>>>>>>>>>>>>>>>> +    sink_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(sink_fmt->colorspace);
>>>>>>>>>>>>>>>>>> +    sink_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>           src_fmt = v4l2_subdev_get_try_format(sd, cfg,
>>>>>>>>>>>>>>>>>>                                                RKISP1_ISP_PAD_SOURCE_VIDEO);
>>>>>>>>>>>>>>>>>>           *src_fmt = *sink_fmt;
>>>>>>>>>>>>>>>>>>           src_fmt->code = RKISP1_DEF_SRC_PAD_FMT;
>>>>>>>>>>>>>>>>>> -    src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>>>>>>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
>>>>>>>>>>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
>>>>>>>>>>>>>>>>>> +    src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>           src_crop = v4l2_subdev_get_try_crop(sd, cfg,
>>>>>>>>>>>>>>>>>>                                               RKISP1_ISP_PAD_SOURCE_VIDEO);
>>>>>>>>>>>>>>>>>> @@ -687,10 +700,17 @@ static void rkisp1_isp_set_src_fmt(struct rkisp1_isp *isp,
>>>>>>>>>>>>>>>>>>                   isp->src_fmt = mbus_info;
>>>>>>>>>>>>>>>>>>           src_fmt->width  = src_crop->width;
>>>>>>>>>>>>>>>>>>           src_fmt->height = src_crop->height;
>>>>>>>>>>>>>>>>>> -    src_fmt->quantization = format->quantization;
>>>>>>>>>>>>>>>>>> -    /* full range by default */
>>>>>>>>>>>>>>>>>> -    if (!src_fmt->quantization)
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>>>>>>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
>>>>>>>>>>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>> +    if (mbus_info->pixel_enc == V4L2_PIXEL_ENC_BAYER)
>>>>>>>>>>>>>>>>>>                   src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>>>>>> +    else if (format->quantization == V4L2_QUANTIZATION_DEFAULT)
>>>>>>>>>>>>>>>>>> +            src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
>>>>>>>>>>>>>>>>>> +    else
>>>>>>>>>>>>>>>>>> +            src_fmt->quantization = format->quantization;
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>           *format = *src_fmt;
>>>>>>>>>>>>>>>>>>        }
>>>>>>>>>>>>>>>>>> @@ -1068,6 +1088,17 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
>>>>>>>>>>>>>>>>>>           return ret;
>>>>>>>>>>>>>>>>>>        }
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
>>>>>>>>>>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt)
>>>>>>>>>>>>>>>>>> +{
>>>>>>>>>>>>>>>>>> +    struct rkisp1_isp *isp = &rkisp1->isp;
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>> +    sd_fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
>>>>>>>>>>>>>>>>>> +    sd_fmt->pad = RKISP1_ISP_PAD_SOURCE_VIDEO;
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>> +    return v4l2_subdev_call(&isp->sd, pad, get_fmt, NULL, sd_fmt);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Do we need to get through the external API to access data that is
>>>>>>>>>>>>>>>>> driver-internal anyway?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> +}
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>        void rkisp1_isp_unregister(struct rkisp1_device *rkisp1)
>>>>>>>>>>>>>>>>>>        {
>>>>>>>>>>>>>>>>>>           struct v4l2_subdev *sd = &rkisp1->isp.sd;
>>>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-resizer.c b/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>>>>>>>>>>>>>> index 7b6b7ddd4169..8705b133de68 100644
>>>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>>>>>>>>>>>>>> @@ -525,6 +525,7 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
>>>>>>>>>>>>>>>>>>           const struct rkisp1_isp_mbus_info *mbus_info;
>>>>>>>>>>>>>>>>>>           struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
>>>>>>>>>>>>>>>>>>           struct v4l2_rect *sink_crop;
>>>>>>>>>>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>           sink_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SINK, which);
>>>>>>>>>>>>>>>>>>           src_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SRC, which);
>>>>>>>>>>>>>>>>>> @@ -539,8 +540,20 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
>>>>>>>>>>>>>>>>>>           if (which == V4L2_SUBDEV_FORMAT_ACTIVE)
>>>>>>>>>>>>>>>>>>                   rsz->pixel_enc = mbus_info->pixel_enc;
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> +    rkisp1_get_isp_src_fmt(rsz->rkisp1, &isp_sd_fmt);
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Is this necessary? My understanding was that in the subdev model, it was
>>>>>>>>>>>>>>>>> the userspace responsibility to propagate any configuration changes through
>>>>>>>>>>>>>>>>> the graph.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Also, doing this only here wouldn't fully maintain the
>>>>>>>>>>>>>>>>> consistency of the state. For example, if one sets the ISP subdev format
>>>>>>>>>>>>>>>>> first, then the resizer subdev and then the ISP subdev again, wouldn't the
>>>>>>>>>>>>>>>>> resizer subdev end up with a wrong format?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> yes, this is indeed a bug, I am preparing v4 now.
>>>>>>>>>>>>>>>> What I thought to do is adding quantization conversion
>>>>>>>>>>>>>>>> support also on ther resizer and capture entities.
>>>>>>>>>>>>>>>> Then in the 'link_validation' callbacks, there
>>>>>>>>>>>>>>>> is a validation that the quantization fields matches.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> My understanding is that, if we have the following topology
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [ ISP ] -> [ Resizer ] -> [ Video node ]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> then the ISP source pad would have the csc capability, while resizer
>>>>>>>>>>>>>>> and video node would just accept whatever is configured on their sink
>>>>>>>>>>>>>>> pads (no need for csc capability for that) and propagate that to their
>>>>>>>>>>>>>>> outputs, i.e. resizer source pad and video node CAPTURE format.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Is this what you were going to implement?
>>>>>>>>>>>>> Hi, I sent a v4 where the CSC capability is set on the reiszer and capture as well.
>>>>>>>>>>>>> I can send a v5 implementing it the way you suggest. Currently the doc says that the colorspace fields
>>>>>>>>>>>>> must be set by the driver for capture streams. This implies that userspace can set it
>>>>>>>>>>>>> only if the CSC is supported.
>>>>>>>>>>>>
>>>>>>>>>>>> Why would the userspace have to set it for the capture stream on the
>>>>>>>>>>>> resizer and video nodes? Couldn't the userspace set it to DEFAULT and
>>>>>>>>>>>> then the driver override to whatever it received on the corresponding
>>>>>>>>>>>> sink?
>>>>>>>>>
>>>>>>>>> According to the documentation [1]:
>>>>>>>>>
>>>>>>>>> "the default quantization encoding as defined by the colorspace"
>>>>>>>>>
>>>>>>>>> Then, for instance, the default for V4L2_COLORSPACE_SRGB is limited range [2].
>>>>>>>>>
>>>>>>>>> Unless we change the definition of DEFAULT to let it accept whatever it receives on the corresponding sink.
>>>>>>>>>
>>>>>>>>> Then the way I understand is:
>>>>>>>>>
>>>>>>>>> [sensor]->[isp]->[resizer]->[capture]
>>>>>>>>>
>>>>>>>>> Userspace should set quantization on:
>>>>>>>>> * sensor source pad
>>>>>>>>> * isp sink pad
>>>>>>>>> * resizer sink pad
>>>>>>>>> * capture node
>>>>>>>>>
>>>>>>>>> The remaining source pads would get values propagated from their sink pads.
>>>>>>>>>
>>>>>>>>> [1] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-defs.html#c.v4l2_quantization
>>>>>>>>> [2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-details.html#col-srgb
>>>>>>>>
>>>>>>>> After discussing this offline, we concluded that:
>>>>>>>> - only the source pad of the ISP entity should expose the CSC capability,
>>>>>>>> - the CAPTURE video node should expose only the formats/color spaces
>>>>>>>> compatible with the current setting of the resizer source,
>>>>>>>> - consistency between the resizer source format and CAPTURE format
>>>>>>>> should be implicitly maintained by the driver.
>>>>>>>
>>>>>>> Do you mean it should be validated at streamon time ?
>>>>>>
>>>>>> Nope. Any V4L2 state must be kept valid all the time, as mandated by
>>>>>> the general V4L2 principles. Wrong configuration of the video node
>>>>>> must not be a reason for STREAMON failing.
>>>>>>
>>>>>>> In MC-based
>>>>>>> drivers, entities are isolated from each other from a configuration
>>>>>>> point of view, and the validity of the configuration across the whole
>>>>>>> pipeline is checked at streamon time. I don't think CSC parameters
>>>>>>> should be treated differently than formats in this regard.
>>>>>>
>>>>>> Indeed, the ISP entity and the resizer entity are isolated from each
>>>>>> other and they need to be validated separately. However, the resizer
>>>>>> entity is not isolated from the video node, as there is no
>>>>>> configuration exposed for the video node in the MC API. If MC exposed
>>>>>> an explicit format control for the video interface sink pad, then the
>>>>>> resizer would have to be validated separately indeed, but still that
>>>>>> video interface sink pad would be tied to the rest of the
>>>>>> configuration of the video node (e.g. the CAPTURE format).
>>>>>
>>>>> Two questions:
>>>>>
>>>>> The validation callback between the resizer and the isp has to include check
>>>>> that the quantization values matches ?
>>>>
>>>> To sumarize:
>>>>
>>>> The current quantization fields from VIDIOC_SUBDEV_{S,G}_FMT and VIDEO_{SET,GET}_FORMAT
>>>> are filled by the driver (read-only by userspace).
>>>>
>>>> This RFC exposes an API that allows the quantization to be a read-write configuration.
>>>> The question is where we expose the read-write capability.
>>>>
>>>> We have two main design choices here.
>>>>
>>>> 1) Expose read-write quantization in all pads and video nodes.
>>>> And userspace must set matching quantization in all of them.
>>>> Which makes userspace more complicated, but userspace already needs to configure
>>>> the whole topology anyway.
>>>> This wouldn't make much sense in video nodes for metadata (params and statistics)
>>>>
>>>> In this options, the validation callback should check for matching quantizations
>>>> between all links, and it would failed on STREAMON if formats don't match.
>>>>
>>>> 2-A) Expose read-write quantization only on the isp source pad.
>>>> The others pads and video nodes would be keept read-only.
>>>> Which means that changing quantization on a read-write node, will implicitly
>>>> change the values on the read-only nodes.
>>>> Which maybe not be a problem, since current API (read-only) already says this field
>>>> is filled by the driver.
>>>>
>>>> In this options, the validation callback doesn't need to check for matching quantizations
>>>> between links.
>>>>
>>>> 2-A) Expose read-write quantization only on the video node
>>>> This is basically the same as 2-A, but userspace could interpret that we
>>>> could have different quantizations in each video node.
>>>>
>>>>
>>>> I'd like to hear from others what would be better here, design choice 1 or 2.
>>>
>>> To keep things simple, I'd go for 2-A, and I would ignore the colorspace
>>> fields completely on the video nodes.
>>>
>>> In an MC-centric driver there should be no automatic in-kernel
>>> propagation of userspace-visible configuration from a source pad to the
>>> connected sink pad (regardless of whether the sink is a subdev or a
>>> video node). It's userspace that is responsible for propagating the
>>> configuration, and the kernel is supposed to validate it at stream-on
>>> time.
>>>
>>> In this case, the colorspace information is only relevant to the ISP and
>>> nobody else cares (neither the resizer nor the DMA engines) - please
>>> correct me if I'm wrong. Given that userspace will be specific to this
>>> particular hardware, I would not enforce matching colorspace beyond the
>>> output of the ISP. Userspace will deal with colorspace at the output of
>>> the ISP and ignore it on the video node.
>>
>> So you mean that userspace should know that the quantization of the video
>> is the one configured on the isp, and not 'Default Quantization' as would
>> be reported by the resizers captures?
> 
> Yes. The Rockchip ISP requires device-specific userspace, so requiring
> it to know about the driver doesn't seem a problem to me. I however
> expect some people to disagree :-) We should however record this in
> documentation somewhere, requiring userspace developers to read the
> rkisp1 documentation is one thing, requiring them to read the driver
> source code to figure out how to use it would be going one step too far
> I think.
Hi, Sorry for keep questioning, I just want to make sure I understood correctly what
you mean. So when userspace set the quantization on the src pad of the isp, should
the quantization value then also be updated (propagated) to the resizer and the capture?

Thanks,
Dafna

> 
>>>>> Userspace have to configure the quantizaion on the resizer sink pad without that CSC
>>>>> API beeing expose on it?
>>>>
>>>> In the current API, the quantization field is read-only, so to allow userspace to configure
>>>> quantization in any pad, the CSC API needs to be exposed on the pad to make it read-write.
>>>>
>>>>>>>> Below is the rationale for that:
>>>>>>>> - In the MC model, the video node represents the DMA alone (and thus
>>>>>>>> the reason to have the resizer entities at all) and since the
>>>>>>>> colorspace is not a property of the DMA, it doesn't belong there.
>>>>>>>> - Even if we put that aside, both MP and SP can only output the same
>>>>>>>> colorspace, which would create a dependency between the configuration
>>>>>>>> of the two video nodes, which is not something well defined by V4L2.
>>>>>>>> - A video interface in the MC API doesn't expose control over its
>>>>>>>> input format, which means that the relation between the other side of
>>>>>>>> the link (resizer source) and the video node is implicit and needs to
>>>>>>>> be handled internally in the driver.
>>>>>>>> - V4L2, as opposed to MC, requires that the current state is always
>>>>>>>> consistent and valid. In this case it means that an ioctl on the video
>>>>>>>> node must not be able to alter the state in a way that would make it
>>>>>>>> inconsistent with the state of the matching resizer source.
> 

Re: [RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad

[Laurent Pinchart]

Hi Dafna,

On Wed, Jul 01, 2020 at 04:41:15PM +0200, Dafna Hirschfeld wrote:
> On 01.07.20 14:52, Laurent Pinchart wrote:
> > On Tue, Jun 30, 2020 at 11:36:35AM +0200, Dafna Hirschfeld wrote:
> >> On 30.06.20 01:08, Laurent Pinchart wrote:
> >>> On Mon, Jun 29, 2020 at 12:50:06PM -0300, Helen Koike wrote:
> >>>> On 6/28/20 4:34 PM, Dafna Hirschfeld wrote:
> >>>>> On 28.06.20 16:59, Tomasz Figa wrote:
> >>>>>> On Sun, Jun 28, 2020 at 3:15 PM Laurent Pinchart wrote:
> >>>>>>> On Sun, Jun 28, 2020 at 03:10:41PM +0200, Tomasz Figa wrote:
> >>>>>>>> On Fri, Jun 26, 2020 at 2:15 AM Helen Koike <helen.koike@collabora.com> wrote:
> >>>>>>>>> On 6/18/20 4:25 PM, Tomasz Figa wrote:
> >>>>>>>>>> On Thu, Jun 18, 2020 at 08:54:34PM +0200, Dafna Hirschfeld wrote:
> >>>>>>>>>>> On 18.06.20 20:49, Tomasz Figa wrote:
> >>>>>>>>>>>> On Thu, Jun 18, 2020 at 08:26:33PM +0200, Dafna Hirschfeld wrote:
> >>>>>>>>>>>>> On 18.06.20 19:27, Tomasz Figa wrote:
> >>>>>>>>>>>>>> On Wed, Jun 10, 2020 at 2:08 PM Tomasz Figa wrote:
> >>>>>>>>>>>>>>> On Thu, Jun 4, 2020 at 9:11 PM Dafna Hirschfeld wrote:
> >>>>>>>>>>>>>>>> On 04.06.20 19:54, Tomasz Figa wrote:
> >>>>>>>>>>>>>>>>> On Thu, Apr 16, 2020 at 04:56:05PM +0200, Dafna Hirschfeld wrote:
> >>>>>>>>>>>>>>>>>> The isp entity has a hardware support to force full range quantization
> >>>>>>>>>>>>>>>>>> for YUV formats. Use the new API to indicate userspace that
> >>>>>>>>>>>>>>>>>> quantization conversion is supported by adding the flag
> >>>>>>>>>>>>>>>>>> V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION during media code enumeration.
> >>>>>>>>>>>>>>>>>> Then uppon s_fmt on the video source pad, we assign the
> >>>>>>>>>>>>>>>>>> quantization from userspace for YUV formats.
> >>>>>>>>>>>>>>>>>> Also in the capture and resizer entities we retrieve the colorspace
> >>>>>>>>>>>>>>>>>> from the isp entity.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
> >>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>        drivers/staging/media/rkisp1/rkisp1-capture.c | 28 ++++++-------
> >>>>>>>>>>>>>>>>>>        drivers/staging/media/rkisp1/rkisp1-common.h  |  2 +
> >>>>>>>>>>>>>>>>>>        drivers/staging/media/rkisp1/rkisp1-isp.c     | 39 +++++++++++++++++--
> >>>>>>>>>>>>>>>>>>        drivers/staging/media/rkisp1/rkisp1-resizer.c | 13 +++++++
> >>>>>>>>>>>>>>>>>>        4 files changed, 65 insertions(+), 17 deletions(-)
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c
> >>>>>>>>>>>>>>>>>> index fbf62399fe3d..aca0f93bc772 100644
> >>>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-capture.c
> >>>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-capture.c
> >>>>>>>>>>>>>>>>>> @@ -1066,14 +1066,13 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
> >>>>>>>>>>>>>>>>>>                              const struct v4l2_format_info **fmt_info)
> >>>>>>>>>>>>>>>>>>        {
> >>>>>>>>>>>>>>>>>>           const struct rkisp1_capture_config *config = cap->config;
> >>>>>>>>>>>>>>>>>> -    struct rkisp1_capture *other_cap =
> >>>>>>>>>>>>>>>>>> -                    &cap->rkisp1->capture_devs[cap->id ^ 1];
> >>>>>>>>>>>>>>>>>>           const struct rkisp1_capture_fmt_cfg *fmt;
> >>>>>>>>>>>>>>>>>>           const struct v4l2_format_info *info;
> >>>>>>>>>>>>>>>>>>           const unsigned int max_widths[] = { RKISP1_RSZ_MP_SRC_MAX_WIDTH,
> >>>>>>>>>>>>>>>>>>                                               RKISP1_RSZ_SP_SRC_MAX_WIDTH };
> >>>>>>>>>>>>>>>>>>           const unsigned int max_heights[] = { RKISP1_RSZ_MP_SRC_MAX_HEIGHT,
> >>>>>>>>>>>>>>>>>>                                                RKISP1_RSZ_SP_SRC_MAX_HEIGHT};
> >>>>>>>>>>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>           fmt = rkisp1_find_fmt_cfg(cap, pixm->pixelformat);
> >>>>>>>>>>>>>>>>>>           if (!fmt) {
> >>>>>>>>>>>>>>>>>> @@ -1081,24 +1080,27 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
> >>>>>>>>>>>>>>>>>>                   pixm->pixelformat = fmt->fourcc;
> >>>>>>>>>>>>>>>>>>           }
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> +    rkisp1_get_isp_src_fmt(cap->rkisp1, &isp_sd_fmt);
> >>>>>>>>>>>>>>>>>> +    pixm->field = isp_sd_fmt.format.field;
> >>>>>>>>>>>>>>>>>> +    pixm->colorspace = isp_sd_fmt.format.colorspace;
> >>>>>>>>>>>>>>>>>> +    pixm->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
> >>>>>>>>>>>>>>>>>> +    pixm->xfer_func = isp_sd_fmt.format.xfer_func;
> >>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>> +    /*
> >>>>>>>>>>>>>>>>>> +     * isp has a feature to set full range quantization for yuv formats.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> How about "select between limited and full range for YUV formats"?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> +     * so we need to get the format from the isp.
> >>>>>>>>>>>>>>>>>> +     */
> >>>>>>>>>>>>>>>>>> +    pixm->quantization = isp_sd_fmt.format.quantization;
> >>>>>>>>>>>>>>>>>> +    if (!v4l2_is_format_yuv(cap->pix.info))
> >>>>>>>>>>>>>>>>>> +            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>>           pixm->width = clamp_t(u32, pixm->width,
> >>>>>>>>>>>>>>>>>>                                 RKISP1_RSZ_SRC_MIN_WIDTH, max_widths[cap->id]);
> >>>>>>>>>>>>>>>>>>           pixm->height = clamp_t(u32, pixm->height,
> >>>>>>>>>>>>>>>>>>                                  RKISP1_RSZ_SRC_MIN_HEIGHT, max_heights[cap->id]);
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> -    pixm->field = V4L2_FIELD_NONE;
> >>>>>>>>>>>>>>>>>> -    pixm->colorspace = V4L2_COLORSPACE_DEFAULT;
> >>>>>>>>>>>>>>>>>> -    pixm->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> >>>>>>>>>>>>>>>>>> -
> >>>>>>>>>>>>>>>>>>           info = rkisp1_fill_pixfmt(pixm, cap->id);
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> -    /* can not change quantization when stream-on */
> >>>>>>>>>>>>>>>>>> -    if (other_cap->is_streaming)
> >>>>>>>>>>>>>>>>>> -            pixm->quantization = other_cap->pix.fmt.quantization;
> >>>>>>>>>>>>>>>>>> -    /* output full range by default, take effect in params */
> >>>>>>>>>>>>>>>>>> -    else if (!pixm->quantization ||
> >>>>>>>>>>>>>>>>>> -             pixm->quantization > V4L2_QUANTIZATION_LIM_RANGE)
> >>>>>>>>>>>>>>>>>> -            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>           if (fmt_cfg)
> >>>>>>>>>>>>>>>>>>                   *fmt_cfg = fmt;
> >>>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-common.h b/drivers/staging/media/rkisp1/rkisp1-common.h
> >>>>>>>>>>>>>>>>>> index 2d7b7e078636..7a5576fa14c9 100644
> >>>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-common.h
> >>>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-common.h
> >>>>>>>>>>>>>>>>>> @@ -300,6 +300,8 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
> >>>>>>>>>>>>>>>>>>                           struct v4l2_device *v4l2_dev);
> >>>>>>>>>>>>>>>>>>        void rkisp1_isp_unregister(struct rkisp1_device *rkisp1);
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
> >>>>>>>>>>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt);
> >>>>>>>>>>>>>>>>>>        const struct rkisp1_isp_mbus_info *rkisp1_isp_mbus_info_get(u32 mbus_code);
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>        irqreturn_t rkisp1_isp_isr(struct rkisp1_device *rkisp1);
> >>>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-isp.c b/drivers/staging/media/rkisp1/rkisp1-isp.c
> >>>>>>>>>>>>>>>>>> index dee8e96f3900..6fdf5ed0b6b1 100644
> >>>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-isp.c
> >>>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-isp.c
> >>>>>>>>>>>>>>>>>> @@ -613,6 +613,10 @@ static int rkisp1_isp_enum_mbus_code(struct v4l2_subdev *sd,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>                   if (code->index == pos - 1) {
> >>>>>>>>>>>>>>>>>>                           code->code = fmt->mbus_code;
> >>>>>>>>>>>>>>>>>> +                    if (fmt->pixel_enc == V4L2_PIXEL_ENC_YUV &&
> >>>>>>>>>>>>>>>>>> +                        dir == RKISP1_DIR_SRC)
> >>>>>>>>>>>>>>>>>> +                            code->flags =
> >>>>>>>>>>>>>>>>>> +                                    V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION;
> >>>>>>>>>>>>>>>>>>                           return 0;
> >>>>>>>>>>>>>>>>>>                   }
> >>>>>>>>>>>>>>>>>>           }
> >>>>>>>>>>>>>>>>>> @@ -639,12 +643,21 @@ static int rkisp1_isp_init_config(struct v4l2_subdev *sd,
> >>>>>>>>>>>>>>>>>>           sink_crop->height = RKISP1_DEFAULT_HEIGHT;
> >>>>>>>>>>>>>>>>>>           sink_crop->left = 0;
> >>>>>>>>>>>>>>>>>>           sink_crop->top = 0;
> >>>>>>>>>>>>>>>>>> +    sink_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >>>>>>>>>>>>>>>>>> +    sink_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(sink_fmt->colorspace);
> >>>>>>>>>>>>>>>>>> +    sink_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(sink_fmt->colorspace);
> >>>>>>>>>>>>>>>>>> +    sink_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>           src_fmt = v4l2_subdev_get_try_format(sd, cfg,
> >>>>>>>>>>>>>>>>>>                                                RKISP1_ISP_PAD_SOURCE_VIDEO);
> >>>>>>>>>>>>>>>>>>           *src_fmt = *sink_fmt;
> >>>>>>>>>>>>>>>>>>           src_fmt->code = RKISP1_DEF_SRC_PAD_FMT;
> >>>>>>>>>>>>>>>>>> -    src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >>>>>>>>>>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>>>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>>>>>>>>>>> +    src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
> >>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>           src_crop = v4l2_subdev_get_try_crop(sd, cfg,
> >>>>>>>>>>>>>>>>>>                                               RKISP1_ISP_PAD_SOURCE_VIDEO);
> >>>>>>>>>>>>>>>>>> @@ -687,10 +700,17 @@ static void rkisp1_isp_set_src_fmt(struct rkisp1_isp *isp,
> >>>>>>>>>>>>>>>>>>                   isp->src_fmt = mbus_info;
> >>>>>>>>>>>>>>>>>>           src_fmt->width  = src_crop->width;
> >>>>>>>>>>>>>>>>>>           src_fmt->height = src_crop->height;
> >>>>>>>>>>>>>>>>>> -    src_fmt->quantization = format->quantization;
> >>>>>>>>>>>>>>>>>> -    /* full range by default */
> >>>>>>>>>>>>>>>>>> -    if (!src_fmt->quantization)
> >>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >>>>>>>>>>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>>>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>> +    if (mbus_info->pixel_enc == V4L2_PIXEL_ENC_BAYER)
> >>>>>>>>>>>>>>>>>>                   src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>>>>>>> +    else if (format->quantization == V4L2_QUANTIZATION_DEFAULT)
> >>>>>>>>>>>>>>>>>> +            src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
> >>>>>>>>>>>>>>>>>> +    else
> >>>>>>>>>>>>>>>>>> +            src_fmt->quantization = format->quantization;
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>           *format = *src_fmt;
> >>>>>>>>>>>>>>>>>>        }
> >>>>>>>>>>>>>>>>>> @@ -1068,6 +1088,17 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
> >>>>>>>>>>>>>>>>>>           return ret;
> >>>>>>>>>>>>>>>>>>        }
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
> >>>>>>>>>>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt)
> >>>>>>>>>>>>>>>>>> +{
> >>>>>>>>>>>>>>>>>> +    struct rkisp1_isp *isp = &rkisp1->isp;
> >>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>> +    sd_fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
> >>>>>>>>>>>>>>>>>> +    sd_fmt->pad = RKISP1_ISP_PAD_SOURCE_VIDEO;
> >>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>> +    return v4l2_subdev_call(&isp->sd, pad, get_fmt, NULL, sd_fmt);
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Do we need to get through the external API to access data that is
> >>>>>>>>>>>>>>>>> driver-internal anyway?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> +}
> >>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>>        void rkisp1_isp_unregister(struct rkisp1_device *rkisp1)
> >>>>>>>>>>>>>>>>>>        {
> >>>>>>>>>>>>>>>>>>           struct v4l2_subdev *sd = &rkisp1->isp.sd;
> >>>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-resizer.c b/drivers/staging/media/rkisp1/rkisp1-resizer.c
> >>>>>>>>>>>>>>>>>> index 7b6b7ddd4169..8705b133de68 100644
> >>>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-resizer.c
> >>>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-resizer.c
> >>>>>>>>>>>>>>>>>> @@ -525,6 +525,7 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
> >>>>>>>>>>>>>>>>>>           const struct rkisp1_isp_mbus_info *mbus_info;
> >>>>>>>>>>>>>>>>>>           struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
> >>>>>>>>>>>>>>>>>>           struct v4l2_rect *sink_crop;
> >>>>>>>>>>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>           sink_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SINK, which);
> >>>>>>>>>>>>>>>>>>           src_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SRC, which);
> >>>>>>>>>>>>>>>>>> @@ -539,8 +540,20 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
> >>>>>>>>>>>>>>>>>>           if (which == V4L2_SUBDEV_FORMAT_ACTIVE)
> >>>>>>>>>>>>>>>>>>                   rsz->pixel_enc = mbus_info->pixel_enc;
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> +    rkisp1_get_isp_src_fmt(rsz->rkisp1, &isp_sd_fmt);
> >>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Is this necessary? My understanding was that in the subdev model, it was
> >>>>>>>>>>>>>>>>> the userspace responsibility to propagate any configuration changes through
> >>>>>>>>>>>>>>>>> the graph.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Also, doing this only here wouldn't fully maintain the
> >>>>>>>>>>>>>>>>> consistency of the state. For example, if one sets the ISP subdev format
> >>>>>>>>>>>>>>>>> first, then the resizer subdev and then the ISP subdev again, wouldn't the
> >>>>>>>>>>>>>>>>> resizer subdev end up with a wrong format?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> yes, this is indeed a bug, I am preparing v4 now.
> >>>>>>>>>>>>>>>> What I thought to do is adding quantization conversion
> >>>>>>>>>>>>>>>> support also on ther resizer and capture entities.
> >>>>>>>>>>>>>>>> Then in the 'link_validation' callbacks, there
> >>>>>>>>>>>>>>>> is a validation that the quantization fields matches.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> My understanding is that, if we have the following topology
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> [ ISP ] -> [ Resizer ] -> [ Video node ]
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> then the ISP source pad would have the csc capability, while resizer
> >>>>>>>>>>>>>>> and video node would just accept whatever is configured on their sink
> >>>>>>>>>>>>>>> pads (no need for csc capability for that) and propagate that to their
> >>>>>>>>>>>>>>> outputs, i.e. resizer source pad and video node CAPTURE format.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Is this what you were going to implement?
> >>>>>>>>>>>>> Hi, I sent a v4 where the CSC capability is set on the reiszer and capture as well.
> >>>>>>>>>>>>> I can send a v5 implementing it the way you suggest. Currently the doc says that the colorspace fields
> >>>>>>>>>>>>> must be set by the driver for capture streams. This implies that userspace can set it
> >>>>>>>>>>>>> only if the CSC is supported.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Why would the userspace have to set it for the capture stream on the
> >>>>>>>>>>>> resizer and video nodes? Couldn't the userspace set it to DEFAULT and
> >>>>>>>>>>>> then the driver override to whatever it received on the corresponding
> >>>>>>>>>>>> sink?
> >>>>>>>>>
> >>>>>>>>> According to the documentation [1]:
> >>>>>>>>>
> >>>>>>>>> "the default quantization encoding as defined by the colorspace"
> >>>>>>>>>
> >>>>>>>>> Then, for instance, the default for V4L2_COLORSPACE_SRGB is limited range [2].
> >>>>>>>>>
> >>>>>>>>> Unless we change the definition of DEFAULT to let it accept whatever it receives on the corresponding sink.
> >>>>>>>>>
> >>>>>>>>> Then the way I understand is:
> >>>>>>>>>
> >>>>>>>>> [sensor]->[isp]->[resizer]->[capture]
> >>>>>>>>>
> >>>>>>>>> Userspace should set quantization on:
> >>>>>>>>> * sensor source pad
> >>>>>>>>> * isp sink pad
> >>>>>>>>> * resizer sink pad
> >>>>>>>>> * capture node
> >>>>>>>>>
> >>>>>>>>> The remaining source pads would get values propagated from their sink pads.
> >>>>>>>>>
> >>>>>>>>> [1] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-defs.html#c.v4l2_quantization
> >>>>>>>>> [2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-details.html#col-srgb
> >>>>>>>>
> >>>>>>>> After discussing this offline, we concluded that:
> >>>>>>>> - only the source pad of the ISP entity should expose the CSC capability,
> >>>>>>>> - the CAPTURE video node should expose only the formats/color spaces
> >>>>>>>> compatible with the current setting of the resizer source,
> >>>>>>>> - consistency between the resizer source format and CAPTURE format
> >>>>>>>> should be implicitly maintained by the driver.
> >>>>>>>
> >>>>>>> Do you mean it should be validated at streamon time ?
> >>>>>>
> >>>>>> Nope. Any V4L2 state must be kept valid all the time, as mandated by
> >>>>>> the general V4L2 principles. Wrong configuration of the video node
> >>>>>> must not be a reason for STREAMON failing.
> >>>>>>
> >>>>>>> In MC-based
> >>>>>>> drivers, entities are isolated from each other from a configuration
> >>>>>>> point of view, and the validity of the configuration across the whole
> >>>>>>> pipeline is checked at streamon time. I don't think CSC parameters
> >>>>>>> should be treated differently than formats in this regard.
> >>>>>>
> >>>>>> Indeed, the ISP entity and the resizer entity are isolated from each
> >>>>>> other and they need to be validated separately. However, the resizer
> >>>>>> entity is not isolated from the video node, as there is no
> >>>>>> configuration exposed for the video node in the MC API. If MC exposed
> >>>>>> an explicit format control for the video interface sink pad, then the
> >>>>>> resizer would have to be validated separately indeed, but still that
> >>>>>> video interface sink pad would be tied to the rest of the
> >>>>>> configuration of the video node (e.g. the CAPTURE format).
> >>>>>
> >>>>> Two questions:
> >>>>>
> >>>>> The validation callback between the resizer and the isp has to include check
> >>>>> that the quantization values matches ?
> >>>>
> >>>> To sumarize:
> >>>>
> >>>> The current quantization fields from VIDIOC_SUBDEV_{S,G}_FMT and VIDEO_{SET,GET}_FORMAT
> >>>> are filled by the driver (read-only by userspace).
> >>>>
> >>>> This RFC exposes an API that allows the quantization to be a read-write configuration.
> >>>> The question is where we expose the read-write capability.
> >>>>
> >>>> We have two main design choices here.
> >>>>
> >>>> 1) Expose read-write quantization in all pads and video nodes.
> >>>> And userspace must set matching quantization in all of them.
> >>>> Which makes userspace more complicated, but userspace already needs to configure
> >>>> the whole topology anyway.
> >>>> This wouldn't make much sense in video nodes for metadata (params and statistics)
> >>>>
> >>>> In this options, the validation callback should check for matching quantizations
> >>>> between all links, and it would failed on STREAMON if formats don't match.
> >>>>
> >>>> 2-A) Expose read-write quantization only on the isp source pad.
> >>>> The others pads and video nodes would be keept read-only.
> >>>> Which means that changing quantization on a read-write node, will implicitly
> >>>> change the values on the read-only nodes.
> >>>> Which maybe not be a problem, since current API (read-only) already says this field
> >>>> is filled by the driver.
> >>>>
> >>>> In this options, the validation callback doesn't need to check for matching quantizations
> >>>> between links.
> >>>>
> >>>> 2-A) Expose read-write quantization only on the video node
> >>>> This is basically the same as 2-A, but userspace could interpret that we
> >>>> could have different quantizations in each video node.
> >>>>
> >>>>
> >>>> I'd like to hear from others what would be better here, design choice 1 or 2.
> >>>
> >>> To keep things simple, I'd go for 2-A, and I would ignore the colorspace
> >>> fields completely on the video nodes.
> >>>
> >>> In an MC-centric driver there should be no automatic in-kernel
> >>> propagation of userspace-visible configuration from a source pad to the
> >>> connected sink pad (regardless of whether the sink is a subdev or a
> >>> video node). It's userspace that is responsible for propagating the
> >>> configuration, and the kernel is supposed to validate it at stream-on
> >>> time.
> >>>
> >>> In this case, the colorspace information is only relevant to the ISP and
> >>> nobody else cares (neither the resizer nor the DMA engines) - please
> >>> correct me if I'm wrong. Given that userspace will be specific to this
> >>> particular hardware, I would not enforce matching colorspace beyond the
> >>> output of the ISP. Userspace will deal with colorspace at the output of
> >>> the ISP and ignore it on the video node.
> >>
> >> So you mean that userspace should know that the quantization of the video
> >> is the one configured on the isp, and not 'Default Quantization' as would
> >> be reported by the resizers captures?
> > 
> > Yes. The Rockchip ISP requires device-specific userspace, so requiring
> > it to know about the driver doesn't seem a problem to me. I however
> > expect some people to disagree :-) We should however record this in
> > documentation somewhere, requiring userspace developers to read the
> > rkisp1 documentation is one thing, requiring them to read the driver
> > source code to figure out how to use it would be going one step too far
> > I think.
>
> Hi, Sorry for keep questioning, I just want to make sure I understood
> correctly what you mean. So when userspace set the quantization on the
> src pad of the isp, should the quantization value then also be updated
> (propagated) to the resizer and the capture?

In MC-based drivers, kernel drivers should not propagate those values
between subdevs or from subdev to video node. Userspace handles the
propagation. In this specific case, userspace may propagate the values
from the ISP source to the resizer sink, and the driver may validate
that across links at streamon time, but I don't think there's any value
in enforcing it.

> >>>>> Userspace have to configure the quantizaion on the resizer sink pad without that CSC
> >>>>> API beeing expose on it?
> >>>>
> >>>> In the current API, the quantization field is read-only, so to allow userspace to configure
> >>>> quantization in any pad, the CSC API needs to be exposed on the pad to make it read-write.
> >>>>
> >>>>>>>> Below is the rationale for that:
> >>>>>>>> - In the MC model, the video node represents the DMA alone (and thus
> >>>>>>>> the reason to have the resizer entities at all) and since the
> >>>>>>>> colorspace is not a property of the DMA, it doesn't belong there.
> >>>>>>>> - Even if we put that aside, both MP and SP can only output the same
> >>>>>>>> colorspace, which would create a dependency between the configuration
> >>>>>>>> of the two video nodes, which is not something well defined by V4L2.
> >>>>>>>> - A video interface in the MC API doesn't expose control over its
> >>>>>>>> input format, which means that the relation between the other side of
> >>>>>>>> the link (resizer source) and the video node is implicit and needs to
> >>>>>>>> be handled internally in the driver.
> >>>>>>>> - V4L2, as opposed to MC, requires that the current state is always
> >>>>>>>> consistent and valid. In this case it means that an ioctl on the video
> >>>>>>>> node must not be able to alter the state in a way that would make it
> >>>>>>>> inconsistent with the state of the matching resizer source.

-- 
Regards,

Laurent Pinchart

Re: [RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad

[Dafna Hirschfeld]

hi

Am 01.07.20 um 14:52 schrieb Laurent Pinchart:
> Hi Dafna,
> 
> On Tue, Jun 30, 2020 at 11:36:35AM +0200, Dafna Hirschfeld wrote:
>> On 30.06.20 01:08, Laurent Pinchart wrote:
>>> On Mon, Jun 29, 2020 at 12:50:06PM -0300, Helen Koike wrote:
>>>> On 6/28/20 4:34 PM, Dafna Hirschfeld wrote:
>>>>> On 28.06.20 16:59, Tomasz Figa wrote:
>>>>>> On Sun, Jun 28, 2020 at 3:15 PM Laurent Pinchart wrote:
>>>>>>> On Sun, Jun 28, 2020 at 03:10:41PM +0200, Tomasz Figa wrote:
>>>>>>>> On Fri, Jun 26, 2020 at 2:15 AM Helen Koike <helen.koike@collabora.com> wrote:
>>>>>>>>> On 6/18/20 4:25 PM, Tomasz Figa wrote:
>>>>>>>>>> On Thu, Jun 18, 2020 at 08:54:34PM +0200, Dafna Hirschfeld wrote:
>>>>>>>>>>> On 18.06.20 20:49, Tomasz Figa wrote:
>>>>>>>>>>>> On Thu, Jun 18, 2020 at 08:26:33PM +0200, Dafna Hirschfeld wrote:
>>>>>>>>>>>>> On 18.06.20 19:27, Tomasz Figa wrote:
>>>>>>>>>>>>>> On Wed, Jun 10, 2020 at 2:08 PM Tomasz Figa wrote:
>>>>>>>>>>>>>>> On Thu, Jun 4, 2020 at 9:11 PM Dafna Hirschfeld wrote:
>>>>>>>>>>>>>>>> On 04.06.20 19:54, Tomasz Figa wrote:
>>>>>>>>>>>>>>>>> On Thu, Apr 16, 2020 at 04:56:05PM +0200, Dafna Hirschfeld wrote:
>>>>>>>>>>>>>>>>>> The isp entity has a hardware support to force full range quantization
>>>>>>>>>>>>>>>>>> for YUV formats. Use the new API to indicate userspace that
>>>>>>>>>>>>>>>>>> quantization conversion is supported by adding the flag
>>>>>>>>>>>>>>>>>> V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION during media code enumeration.
>>>>>>>>>>>>>>>>>> Then uppon s_fmt on the video source pad, we assign the
>>>>>>>>>>>>>>>>>> quantization from userspace for YUV formats.
>>>>>>>>>>>>>>>>>> Also in the capture and resizer entities we retrieve the colorspace
>>>>>>>>>>>>>>>>>> from the isp entity.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>        drivers/staging/media/rkisp1/rkisp1-capture.c | 28 ++++++-------
>>>>>>>>>>>>>>>>>>        drivers/staging/media/rkisp1/rkisp1-common.h  |  2 +
>>>>>>>>>>>>>>>>>>        drivers/staging/media/rkisp1/rkisp1-isp.c     | 39 +++++++++++++++++--
>>>>>>>>>>>>>>>>>>        drivers/staging/media/rkisp1/rkisp1-resizer.c | 13 +++++++
>>>>>>>>>>>>>>>>>>        4 files changed, 65 insertions(+), 17 deletions(-)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>>>>>>>>>>>>>> index fbf62399fe3d..aca0f93bc772 100644
>>>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>>>>>>>>>>>>>> @@ -1066,14 +1066,13 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
>>>>>>>>>>>>>>>>>>                              const struct v4l2_format_info **fmt_info)
>>>>>>>>>>>>>>>>>>        {
>>>>>>>>>>>>>>>>>>           const struct rkisp1_capture_config *config = cap->config;
>>>>>>>>>>>>>>>>>> -    struct rkisp1_capture *other_cap =
>>>>>>>>>>>>>>>>>> -                    &cap->rkisp1->capture_devs[cap->id ^ 1];
>>>>>>>>>>>>>>>>>>           const struct rkisp1_capture_fmt_cfg *fmt;
>>>>>>>>>>>>>>>>>>           const struct v4l2_format_info *info;
>>>>>>>>>>>>>>>>>>           const unsigned int max_widths[] = { RKISP1_RSZ_MP_SRC_MAX_WIDTH,
>>>>>>>>>>>>>>>>>>                                               RKISP1_RSZ_SP_SRC_MAX_WIDTH };
>>>>>>>>>>>>>>>>>>           const unsigned int max_heights[] = { RKISP1_RSZ_MP_SRC_MAX_HEIGHT,
>>>>>>>>>>>>>>>>>>                                                RKISP1_RSZ_SP_SRC_MAX_HEIGHT};
>>>>>>>>>>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>           fmt = rkisp1_find_fmt_cfg(cap, pixm->pixelformat);
>>>>>>>>>>>>>>>>>>           if (!fmt) {
>>>>>>>>>>>>>>>>>> @@ -1081,24 +1080,27 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
>>>>>>>>>>>>>>>>>>                   pixm->pixelformat = fmt->fourcc;
>>>>>>>>>>>>>>>>>>           }
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> +    rkisp1_get_isp_src_fmt(cap->rkisp1, &isp_sd_fmt);
>>>>>>>>>>>>>>>>>> +    pixm->field = isp_sd_fmt.format.field;
>>>>>>>>>>>>>>>>>> +    pixm->colorspace = isp_sd_fmt.format.colorspace;
>>>>>>>>>>>>>>>>>> +    pixm->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
>>>>>>>>>>>>>>>>>> +    pixm->xfer_func = isp_sd_fmt.format.xfer_func;
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>> +    /*
>>>>>>>>>>>>>>>>>> +     * isp has a feature to set full range quantization for yuv formats.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> How about "select between limited and full range for YUV formats"?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> +     * so we need to get the format from the isp.
>>>>>>>>>>>>>>>>>> +     */
>>>>>>>>>>>>>>>>>> +    pixm->quantization = isp_sd_fmt.format.quantization;
>>>>>>>>>>>>>>>>>> +    if (!v4l2_is_format_yuv(cap->pix.info))
>>>>>>>>>>>>>>>>>> +            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>           pixm->width = clamp_t(u32, pixm->width,
>>>>>>>>>>>>>>>>>>                                 RKISP1_RSZ_SRC_MIN_WIDTH, max_widths[cap->id]);
>>>>>>>>>>>>>>>>>>           pixm->height = clamp_t(u32, pixm->height,
>>>>>>>>>>>>>>>>>>                                  RKISP1_RSZ_SRC_MIN_HEIGHT, max_heights[cap->id]);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -    pixm->field = V4L2_FIELD_NONE;
>>>>>>>>>>>>>>>>>> -    pixm->colorspace = V4L2_COLORSPACE_DEFAULT;
>>>>>>>>>>>>>>>>>> -    pixm->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>>>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>>>           info = rkisp1_fill_pixfmt(pixm, cap->id);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -    /* can not change quantization when stream-on */
>>>>>>>>>>>>>>>>>> -    if (other_cap->is_streaming)
>>>>>>>>>>>>>>>>>> -            pixm->quantization = other_cap->pix.fmt.quantization;
>>>>>>>>>>>>>>>>>> -    /* output full range by default, take effect in params */
>>>>>>>>>>>>>>>>>> -    else if (!pixm->quantization ||
>>>>>>>>>>>>>>>>>> -             pixm->quantization > V4L2_QUANTIZATION_LIM_RANGE)
>>>>>>>>>>>>>>>>>> -            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>           if (fmt_cfg)
>>>>>>>>>>>>>>>>>>                   *fmt_cfg = fmt;
>>>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-common.h b/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>>>>>>>>>>>>>> index 2d7b7e078636..7a5576fa14c9 100644
>>>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>>>>>>>>>>>>>> @@ -300,6 +300,8 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
>>>>>>>>>>>>>>>>>>                           struct v4l2_device *v4l2_dev);
>>>>>>>>>>>>>>>>>>        void rkisp1_isp_unregister(struct rkisp1_device *rkisp1);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
>>>>>>>>>>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt);
>>>>>>>>>>>>>>>>>>        const struct rkisp1_isp_mbus_info *rkisp1_isp_mbus_info_get(u32 mbus_code);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>        irqreturn_t rkisp1_isp_isr(struct rkisp1_device *rkisp1);
>>>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-isp.c b/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>>>>>>>>>>>>>> index dee8e96f3900..6fdf5ed0b6b1 100644
>>>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>>>>>>>>>>>>>> @@ -613,6 +613,10 @@ static int rkisp1_isp_enum_mbus_code(struct v4l2_subdev *sd,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                   if (code->index == pos - 1) {
>>>>>>>>>>>>>>>>>>                           code->code = fmt->mbus_code;
>>>>>>>>>>>>>>>>>> +                    if (fmt->pixel_enc == V4L2_PIXEL_ENC_YUV &&
>>>>>>>>>>>>>>>>>> +                        dir == RKISP1_DIR_SRC)
>>>>>>>>>>>>>>>>>> +                            code->flags =
>>>>>>>>>>>>>>>>>> +                                    V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION;
>>>>>>>>>>>>>>>>>>                           return 0;
>>>>>>>>>>>>>>>>>>                   }
>>>>>>>>>>>>>>>>>>           }
>>>>>>>>>>>>>>>>>> @@ -639,12 +643,21 @@ static int rkisp1_isp_init_config(struct v4l2_subdev *sd,
>>>>>>>>>>>>>>>>>>           sink_crop->height = RKISP1_DEFAULT_HEIGHT;
>>>>>>>>>>>>>>>>>>           sink_crop->left = 0;
>>>>>>>>>>>>>>>>>>           sink_crop->top = 0;
>>>>>>>>>>>>>>>>>> +    sink_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>>>>>>>>>>>>>> +    sink_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(sink_fmt->colorspace);
>>>>>>>>>>>>>>>>>> +    sink_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(sink_fmt->colorspace);
>>>>>>>>>>>>>>>>>> +    sink_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>           src_fmt = v4l2_subdev_get_try_format(sd, cfg,
>>>>>>>>>>>>>>>>>>                                                RKISP1_ISP_PAD_SOURCE_VIDEO);
>>>>>>>>>>>>>>>>>>           *src_fmt = *sink_fmt;
>>>>>>>>>>>>>>>>>>           src_fmt->code = RKISP1_DEF_SRC_PAD_FMT;
>>>>>>>>>>>>>>>>>> -    src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>>>>>>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
>>>>>>>>>>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
>>>>>>>>>>>>>>>>>> +    src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>           src_crop = v4l2_subdev_get_try_crop(sd, cfg,
>>>>>>>>>>>>>>>>>>                                               RKISP1_ISP_PAD_SOURCE_VIDEO);
>>>>>>>>>>>>>>>>>> @@ -687,10 +700,17 @@ static void rkisp1_isp_set_src_fmt(struct rkisp1_isp *isp,
>>>>>>>>>>>>>>>>>>                   isp->src_fmt = mbus_info;
>>>>>>>>>>>>>>>>>>           src_fmt->width  = src_crop->width;
>>>>>>>>>>>>>>>>>>           src_fmt->height = src_crop->height;
>>>>>>>>>>>>>>>>>> -    src_fmt->quantization = format->quantization;
>>>>>>>>>>>>>>>>>> -    /* full range by default */
>>>>>>>>>>>>>>>>>> -    if (!src_fmt->quantization)
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>>>>>>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
>>>>>>>>>>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>> +    if (mbus_info->pixel_enc == V4L2_PIXEL_ENC_BAYER)
>>>>>>>>>>>>>>>>>>                   src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>>>>>> +    else if (format->quantization == V4L2_QUANTIZATION_DEFAULT)
>>>>>>>>>>>>>>>>>> +            src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
>>>>>>>>>>>>>>>>>> +    else
>>>>>>>>>>>>>>>>>> +            src_fmt->quantization = format->quantization;
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>           *format = *src_fmt;
>>>>>>>>>>>>>>>>>>        }
>>>>>>>>>>>>>>>>>> @@ -1068,6 +1088,17 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
>>>>>>>>>>>>>>>>>>           return ret;
>>>>>>>>>>>>>>>>>>        }
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
>>>>>>>>>>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt)
>>>>>>>>>>>>>>>>>> +{
>>>>>>>>>>>>>>>>>> +    struct rkisp1_isp *isp = &rkisp1->isp;
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>> +    sd_fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
>>>>>>>>>>>>>>>>>> +    sd_fmt->pad = RKISP1_ISP_PAD_SOURCE_VIDEO;
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>> +    return v4l2_subdev_call(&isp->sd, pad, get_fmt, NULL, sd_fmt);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Do we need to get through the external API to access data that is
>>>>>>>>>>>>>>>>> driver-internal anyway?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> +}
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>        void rkisp1_isp_unregister(struct rkisp1_device *rkisp1)
>>>>>>>>>>>>>>>>>>        {
>>>>>>>>>>>>>>>>>>           struct v4l2_subdev *sd = &rkisp1->isp.sd;
>>>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-resizer.c b/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>>>>>>>>>>>>>> index 7b6b7ddd4169..8705b133de68 100644
>>>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>>>>>>>>>>>>>> @@ -525,6 +525,7 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
>>>>>>>>>>>>>>>>>>           const struct rkisp1_isp_mbus_info *mbus_info;
>>>>>>>>>>>>>>>>>>           struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
>>>>>>>>>>>>>>>>>>           struct v4l2_rect *sink_crop;
>>>>>>>>>>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>           sink_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SINK, which);
>>>>>>>>>>>>>>>>>>           src_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SRC, which);
>>>>>>>>>>>>>>>>>> @@ -539,8 +540,20 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
>>>>>>>>>>>>>>>>>>           if (which == V4L2_SUBDEV_FORMAT_ACTIVE)
>>>>>>>>>>>>>>>>>>                   rsz->pixel_enc = mbus_info->pixel_enc;
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> +    rkisp1_get_isp_src_fmt(rsz->rkisp1, &isp_sd_fmt);
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Is this necessary? My understanding was that in the subdev model, it was
>>>>>>>>>>>>>>>>> the userspace responsibility to propagate any configuration changes through
>>>>>>>>>>>>>>>>> the graph.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Also, doing this only here wouldn't fully maintain the
>>>>>>>>>>>>>>>>> consistency of the state. For example, if one sets the ISP subdev format
>>>>>>>>>>>>>>>>> first, then the resizer subdev and then the ISP subdev again, wouldn't the
>>>>>>>>>>>>>>>>> resizer subdev end up with a wrong format?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> yes, this is indeed a bug, I am preparing v4 now.
>>>>>>>>>>>>>>>> What I thought to do is adding quantization conversion
>>>>>>>>>>>>>>>> support also on ther resizer and capture entities.
>>>>>>>>>>>>>>>> Then in the 'link_validation' callbacks, there
>>>>>>>>>>>>>>>> is a validation that the quantization fields matches.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> My understanding is that, if we have the following topology
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [ ISP ] -> [ Resizer ] -> [ Video node ]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> then the ISP source pad would have the csc capability, while resizer
>>>>>>>>>>>>>>> and video node would just accept whatever is configured on their sink
>>>>>>>>>>>>>>> pads (no need for csc capability for that) and propagate that to their
>>>>>>>>>>>>>>> outputs, i.e. resizer source pad and video node CAPTURE format.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Is this what you were going to implement?
>>>>>>>>>>>>> Hi, I sent a v4 where the CSC capability is set on the reiszer and capture as well.
>>>>>>>>>>>>> I can send a v5 implementing it the way you suggest. Currently the doc says that the colorspace fields
>>>>>>>>>>>>> must be set by the driver for capture streams. This implies that userspace can set it
>>>>>>>>>>>>> only if the CSC is supported.
>>>>>>>>>>>>
>>>>>>>>>>>> Why would the userspace have to set it for the capture stream on the
>>>>>>>>>>>> resizer and video nodes? Couldn't the userspace set it to DEFAULT and
>>>>>>>>>>>> then the driver override to whatever it received on the corresponding
>>>>>>>>>>>> sink?
>>>>>>>>>
>>>>>>>>> According to the documentation [1]:
>>>>>>>>>
>>>>>>>>> "the default quantization encoding as defined by the colorspace"
>>>>>>>>>
>>>>>>>>> Then, for instance, the default for V4L2_COLORSPACE_SRGB is limited range [2].
>>>>>>>>>
>>>>>>>>> Unless we change the definition of DEFAULT to let it accept whatever it receives on the corresponding sink.
>>>>>>>>>
>>>>>>>>> Then the way I understand is:
>>>>>>>>>
>>>>>>>>> [sensor]->[isp]->[resizer]->[capture]
>>>>>>>>>
>>>>>>>>> Userspace should set quantization on:
>>>>>>>>> * sensor source pad
>>>>>>>>> * isp sink pad
>>>>>>>>> * resizer sink pad
>>>>>>>>> * capture node
>>>>>>>>>
>>>>>>>>> The remaining source pads would get values propagated from their sink pads.
>>>>>>>>>
>>>>>>>>> [1] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-defs.html#c.v4l2_quantization
>>>>>>>>> [2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-details.html#col-srgb
>>>>>>>>
>>>>>>>> After discussing this offline, we concluded that:
>>>>>>>> - only the source pad of the ISP entity should expose the CSC capability,
>>>>>>>> - the CAPTURE video node should expose only the formats/color spaces
>>>>>>>> compatible with the current setting of the resizer source,
>>>>>>>> - consistency between the resizer source format and CAPTURE format
>>>>>>>> should be implicitly maintained by the driver.
>>>>>>>
>>>>>>> Do you mean it should be validated at streamon time ?
>>>>>>
>>>>>> Nope. Any V4L2 state must be kept valid all the time, as mandated by
>>>>>> the general V4L2 principles. Wrong configuration of the video node
>>>>>> must not be a reason for STREAMON failing.
>>>>>>
>>>>>>> In MC-based
>>>>>>> drivers, entities are isolated from each other from a configuration
>>>>>>> point of view, and the validity of the configuration across the whole
>>>>>>> pipeline is checked at streamon time. I don't think CSC parameters
>>>>>>> should be treated differently than formats in this regard.
>>>>>>
>>>>>> Indeed, the ISP entity and the resizer entity are isolated from each
>>>>>> other and they need to be validated separately. However, the resizer
>>>>>> entity is not isolated from the video node, as there is no
>>>>>> configuration exposed for the video node in the MC API. If MC exposed
>>>>>> an explicit format control for the video interface sink pad, then the
>>>>>> resizer would have to be validated separately indeed, but still that
>>>>>> video interface sink pad would be tied to the rest of the
>>>>>> configuration of the video node (e.g. the CAPTURE format).
>>>>>
>>>>> Two questions:
>>>>>
>>>>> The validation callback between the resizer and the isp has to include check
>>>>> that the quantization values matches ?
>>>>
>>>> To sumarize:
>>>>
>>>> The current quantization fields from VIDIOC_SUBDEV_{S,G}_FMT and VIDEO_{SET,GET}_FORMAT
>>>> are filled by the driver (read-only by userspace).
>>>>
>>>> This RFC exposes an API that allows the quantization to be a read-write configuration.
>>>> The question is where we expose the read-write capability.
>>>>
>>>> We have two main design choices here.
>>>>
>>>> 1) Expose read-write quantization in all pads and video nodes.
>>>> And userspace must set matching quantization in all of them.
>>>> Which makes userspace more complicated, but userspace already needs to configure
>>>> the whole topology anyway.
>>>> This wouldn't make much sense in video nodes for metadata (params and statistics)
>>>>
>>>> In this options, the validation callback should check for matching quantizations
>>>> between all links, and it would failed on STREAMON if formats don't match.
>>>>
>>>> 2-A) Expose read-write quantization only on the isp source pad.
>>>> The others pads and video nodes would be keept read-only.
>>>> Which means that changing quantization on a read-write node, will implicitly
>>>> change the values on the read-only nodes.
>>>> Which maybe not be a problem, since current API (read-only) already says this field
>>>> is filled by the driver.
>>>>
>>>> In this options, the validation callback doesn't need to check for matching quantizations
>>>> between links.
>>>>
>>>> 2-A) Expose read-write quantization only on the video node
>>>> This is basically the same as 2-A, but userspace could interpret that we
>>>> could have different quantizations in each video node.
>>>>
>>>>
>>>> I'd like to hear from others what would be better here, design choice 1 or 2.
>>>
>>> To keep things simple, I'd go for 2-A, and I would ignore the colorspace
>>> fields completely on the video nodes.
>>>
>>> In an MC-centric driver there should be no automatic in-kernel
>>> propagation of userspace-visible configuration from a source pad to the
>>> connected sink pad (regardless of whether the sink is a subdev or a
>>> video node). It's userspace that is responsible for propagating the
>>> configuration, and the kernel is supposed to validate it at stream-on
>>> time.
>>>
>>> In this case, the colorspace information is only relevant to the ISP and
>>> nobody else cares (neither the resizer nor the DMA engines) - please
>>> correct me if I'm wrong. Given that userspace will be specific to this
>>> particular hardware, I would not enforce matching colorspace beyond the
>>> output of the ISP. Userspace will deal with colorspace at the output of
>>> the ISP and ignore it on the video node.
>>
>> So you mean that userspace should know that the quantization of the video
>> is the one configured on the isp, and not 'Default Quantization' as would
>> be reported by the resizers captures?
> 
> Yes. The Rockchip ISP requires device-specific userspace, so requiring
> it to know about the driver doesn't seem a problem to me. I however
> expect some people to disagree :-) We should however record this in
> documentation somewhere, requiring userspace developers to read the
> rkisp1 documentation is one thing, requiring them to read the driver
> source code to figure out how to use it would be going one step too far
> I think.

Hi, I already sent a v5 with your suggestion and already got your "reviewd-by" and Acked-by from Helen.
[1] https://patchwork.kernel.org/patch/11642383/#23505365

But thinking of it again, the purpose of adding the CSC API was to make the code compatible
by supporting it in the v4l2 API.
Now we end up with a new API feature for subdevices and the rkisp1 still have a non compatible weird behavior
that needs special documentation. It might be just a simpler solution to document the rkisp1 that although
a quantization should be set by the driver according to the core API, for this driver it can
also be set by userspace.

It is also not clear how other subdevices suppose to use the CSC API. Either they implement the
same weird behavior as rkisp1 or they will have to require userspace to propagate the
colorspace param that is set on the subdevice. It is not clear then if the CSC API should be
supported on the other subdev nodes in the topology in order to allow it.

Thanks,
Dafna

> 
>>>>> Userspace have to configure the quantizaion on the resizer sink pad without that CSC
>>>>> API beeing expose on it?
>>>>
>>>> In the current API, the quantization field is read-only, so to allow userspace to configure
>>>> quantization in any pad, the CSC API needs to be exposed on the pad to make it read-write.
>>>>
>>>>>>>> Below is the rationale for that:
>>>>>>>> - In the MC model, the video node represents the DMA alone (and thus
>>>>>>>> the reason to have the resizer entities at all) and since the
>>>>>>>> colorspace is not a property of the DMA, it doesn't belong there.
>>>>>>>> - Even if we put that aside, both MP and SP can only output the same
>>>>>>>> colorspace, which would create a dependency between the configuration
>>>>>>>> of the two video nodes, which is not something well defined by V4L2.
>>>>>>>> - A video interface in the MC API doesn't expose control over its
>>>>>>>> input format, which means that the relation between the other side of
>>>>>>>> the link (resizer source) and the video node is implicit and needs to
>>>>>>>> be handled internally in the driver.
>>>>>>>> - V4L2, as opposed to MC, requires that the current state is always
>>>>>>>> consistent and valid. In this case it means that an ioctl on the video
>>>>>>>> node must not be able to alter the state in a way that would make it
>>>>>>>> inconsistent with the state of the matching resizer source.
> 

Re: [RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad

[Dafna Hirschfeld]

Am 10.08.20 um 14:33 schrieb Dafna Hirschfeld:
> hi
> 
> Am 01.07.20 um 14:52 schrieb Laurent Pinchart:
>> Hi Dafna,
>>
>> On Tue, Jun 30, 2020 at 11:36:35AM +0200, Dafna Hirschfeld wrote:
>>> On 30.06.20 01:08, Laurent Pinchart wrote:
>>>> On Mon, Jun 29, 2020 at 12:50:06PM -0300, Helen Koike wrote:
>>>>> On 6/28/20 4:34 PM, Dafna Hirschfeld wrote:
>>>>>> On 28.06.20 16:59, Tomasz Figa wrote:
>>>>>>> On Sun, Jun 28, 2020 at 3:15 PM Laurent Pinchart wrote:
>>>>>>>> On Sun, Jun 28, 2020 at 03:10:41PM +0200, Tomasz Figa wrote:
>>>>>>>>> On Fri, Jun 26, 2020 at 2:15 AM Helen Koike <helen.koike@collabora.com> wrote:
>>>>>>>>>> On 6/18/20 4:25 PM, Tomasz Figa wrote:
>>>>>>>>>>> On Thu, Jun 18, 2020 at 08:54:34PM +0200, Dafna Hirschfeld wrote:
>>>>>>>>>>>> On 18.06.20 20:49, Tomasz Figa wrote:
>>>>>>>>>>>>> On Thu, Jun 18, 2020 at 08:26:33PM +0200, Dafna Hirschfeld wrote:
>>>>>>>>>>>>>> On 18.06.20 19:27, Tomasz Figa wrote:
>>>>>>>>>>>>>>> On Wed, Jun 10, 2020 at 2:08 PM Tomasz Figa wrote:
>>>>>>>>>>>>>>>> On Thu, Jun 4, 2020 at 9:11 PM Dafna Hirschfeld wrote:
>>>>>>>>>>>>>>>>> On 04.06.20 19:54, Tomasz Figa wrote:
>>>>>>>>>>>>>>>>>> On Thu, Apr 16, 2020 at 04:56:05PM +0200, Dafna Hirschfeld wrote:
>>>>>>>>>>>>>>>>>>> The isp entity has a hardware support to force full range quantization
>>>>>>>>>>>>>>>>>>> for YUV formats. Use the new API to indicate userspace that
>>>>>>>>>>>>>>>>>>> quantization conversion is supported by adding the flag
>>>>>>>>>>>>>>>>>>> V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION during media code enumeration.
>>>>>>>>>>>>>>>>>>> Then uppon s_fmt on the video source pad, we assign the
>>>>>>>>>>>>>>>>>>> quantization from userspace for YUV formats.
>>>>>>>>>>>>>>>>>>> Also in the capture and resizer entities we retrieve the colorspace
>>>>>>>>>>>>>>>>>>> from the isp entity.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>        drivers/staging/media/rkisp1/rkisp1-capture.c | 28 ++++++-------
>>>>>>>>>>>>>>>>>>>        drivers/staging/media/rkisp1/rkisp1-common.h  |  2 +
>>>>>>>>>>>>>>>>>>>        drivers/staging/media/rkisp1/rkisp1-isp.c     | 39 +++++++++++++++++--
>>>>>>>>>>>>>>>>>>>        drivers/staging/media/rkisp1/rkisp1-resizer.c | 13 +++++++
>>>>>>>>>>>>>>>>>>>        4 files changed, 65 insertions(+), 17 deletions(-)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>>>>>>>>>>>>>>> index fbf62399fe3d..aca0f93bc772 100644
>>>>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>>>>>>>>>>>>>>> @@ -1066,14 +1066,13 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
>>>>>>>>>>>>>>>>>>>                              const struct v4l2_format_info **fmt_info)
>>>>>>>>>>>>>>>>>>>        {
>>>>>>>>>>>>>>>>>>>           const struct rkisp1_capture_config *config = cap->config;
>>>>>>>>>>>>>>>>>>> -    struct rkisp1_capture *other_cap =
>>>>>>>>>>>>>>>>>>> -                    &cap->rkisp1->capture_devs[cap->id ^ 1];
>>>>>>>>>>>>>>>>>>>           const struct rkisp1_capture_fmt_cfg *fmt;
>>>>>>>>>>>>>>>>>>>           const struct v4l2_format_info *info;
>>>>>>>>>>>>>>>>>>>           const unsigned int max_widths[] = { RKISP1_RSZ_MP_SRC_MAX_WIDTH,
>>>>>>>>>>>>>>>>>>>                                               RKISP1_RSZ_SP_SRC_MAX_WIDTH };
>>>>>>>>>>>>>>>>>>>           const unsigned int max_heights[] = { RKISP1_RSZ_MP_SRC_MAX_HEIGHT,
>>>>>>>>>>>>>>>>>>>                                                RKISP1_RSZ_SP_SRC_MAX_HEIGHT};
>>>>>>>>>>>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>           fmt = rkisp1_find_fmt_cfg(cap, pixm->pixelformat);
>>>>>>>>>>>>>>>>>>>           if (!fmt) {
>>>>>>>>>>>>>>>>>>> @@ -1081,24 +1080,27 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
>>>>>>>>>>>>>>>>>>>                   pixm->pixelformat = fmt->fourcc;
>>>>>>>>>>>>>>>>>>>           }
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> +    rkisp1_get_isp_src_fmt(cap->rkisp1, &isp_sd_fmt);
>>>>>>>>>>>>>>>>>>> +    pixm->field = isp_sd_fmt.format.field;
>>>>>>>>>>>>>>>>>>> +    pixm->colorspace = isp_sd_fmt.format.colorspace;
>>>>>>>>>>>>>>>>>>> +    pixm->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
>>>>>>>>>>>>>>>>>>> +    pixm->xfer_func = isp_sd_fmt.format.xfer_func;
>>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>> +    /*
>>>>>>>>>>>>>>>>>>> +     * isp has a feature to set full range quantization for yuv formats.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> How about "select between limited and full range for YUV formats"?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> +     * so we need to get the format from the isp.
>>>>>>>>>>>>>>>>>>> +     */
>>>>>>>>>>>>>>>>>>> +    pixm->quantization = isp_sd_fmt.format.quantization;
>>>>>>>>>>>>>>>>>>> +    if (!v4l2_is_format_yuv(cap->pix.info))
>>>>>>>>>>>>>>>>>>> +            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>>           pixm->width = clamp_t(u32, pixm->width,
>>>>>>>>>>>>>>>>>>>                                 RKISP1_RSZ_SRC_MIN_WIDTH, max_widths[cap->id]);
>>>>>>>>>>>>>>>>>>>           pixm->height = clamp_t(u32, pixm->height,
>>>>>>>>>>>>>>>>>>>                                  RKISP1_RSZ_SRC_MIN_HEIGHT, max_heights[cap->id]);
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -    pixm->field = V4L2_FIELD_NONE;
>>>>>>>>>>>>>>>>>>> -    pixm->colorspace = V4L2_COLORSPACE_DEFAULT;
>>>>>>>>>>>>>>>>>>> -    pixm->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>>>>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>>>>           info = rkisp1_fill_pixfmt(pixm, cap->id);
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -    /* can not change quantization when stream-on */
>>>>>>>>>>>>>>>>>>> -    if (other_cap->is_streaming)
>>>>>>>>>>>>>>>>>>> -            pixm->quantization = other_cap->pix.fmt.quantization;
>>>>>>>>>>>>>>>>>>> -    /* output full range by default, take effect in params */
>>>>>>>>>>>>>>>>>>> -    else if (!pixm->quantization ||
>>>>>>>>>>>>>>>>>>> -             pixm->quantization > V4L2_QUANTIZATION_LIM_RANGE)
>>>>>>>>>>>>>>>>>>> -            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>           if (fmt_cfg)
>>>>>>>>>>>>>>>>>>>                   *fmt_cfg = fmt;
>>>>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-common.h b/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>>>>>>>>>>>>>>> index 2d7b7e078636..7a5576fa14c9 100644
>>>>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>>>>>>>>>>>>>>> @@ -300,6 +300,8 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
>>>>>>>>>>>>>>>>>>>                           struct v4l2_device *v4l2_dev);
>>>>>>>>>>>>>>>>>>>        void rkisp1_isp_unregister(struct rkisp1_device *rkisp1);
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
>>>>>>>>>>>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt);
>>>>>>>>>>>>>>>>>>>        const struct rkisp1_isp_mbus_info *rkisp1_isp_mbus_info_get(u32 mbus_code);
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>        irqreturn_t rkisp1_isp_isr(struct rkisp1_device *rkisp1);
>>>>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-isp.c b/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>>>>>>>>>>>>>>> index dee8e96f3900..6fdf5ed0b6b1 100644
>>>>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>>>>>>>>>>>>>>> @@ -613,6 +613,10 @@ static int rkisp1_isp_enum_mbus_code(struct v4l2_subdev *sd,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                   if (code->index == pos - 1) {
>>>>>>>>>>>>>>>>>>>                           code->code = fmt->mbus_code;
>>>>>>>>>>>>>>>>>>> +                    if (fmt->pixel_enc == V4L2_PIXEL_ENC_YUV &&
>>>>>>>>>>>>>>>>>>> +                        dir == RKISP1_DIR_SRC)
>>>>>>>>>>>>>>>>>>> +                            code->flags =
>>>>>>>>>>>>>>>>>>> +                                    V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION;
>>>>>>>>>>>>>>>>>>>                           return 0;
>>>>>>>>>>>>>>>>>>>                   }
>>>>>>>>>>>>>>>>>>>           }
>>>>>>>>>>>>>>>>>>> @@ -639,12 +643,21 @@ static int rkisp1_isp_init_config(struct v4l2_subdev *sd,
>>>>>>>>>>>>>>>>>>>           sink_crop->height = RKISP1_DEFAULT_HEIGHT;
>>>>>>>>>>>>>>>>>>>           sink_crop->left = 0;
>>>>>>>>>>>>>>>>>>>           sink_crop->top = 0;
>>>>>>>>>>>>>>>>>>> +    sink_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>>>>>>>>>>>>>>> +    sink_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(sink_fmt->colorspace);
>>>>>>>>>>>>>>>>>>> +    sink_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(sink_fmt->colorspace);
>>>>>>>>>>>>>>>>>>> +    sink_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>           src_fmt = v4l2_subdev_get_try_format(sd, cfg,
>>>>>>>>>>>>>>>>>>>                                                RKISP1_ISP_PAD_SOURCE_VIDEO);
>>>>>>>>>>>>>>>>>>>           *src_fmt = *sink_fmt;
>>>>>>>>>>>>>>>>>>>           src_fmt->code = RKISP1_DEF_SRC_PAD_FMT;
>>>>>>>>>>>>>>>>>>> -    src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>>>>>>>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
>>>>>>>>>>>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
>>>>>>>>>>>>>>>>>>> +    src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
>>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>           src_crop = v4l2_subdev_get_try_crop(sd, cfg,
>>>>>>>>>>>>>>>>>>>                                               RKISP1_ISP_PAD_SOURCE_VIDEO);
>>>>>>>>>>>>>>>>>>> @@ -687,10 +700,17 @@ static void rkisp1_isp_set_src_fmt(struct rkisp1_isp *isp,
>>>>>>>>>>>>>>>>>>>                   isp->src_fmt = mbus_info;
>>>>>>>>>>>>>>>>>>>           src_fmt->width  = src_crop->width;
>>>>>>>>>>>>>>>>>>>           src_fmt->height = src_crop->height;
>>>>>>>>>>>>>>>>>>> -    src_fmt->quantization = format->quantization;
>>>>>>>>>>>>>>>>>>> -    /* full range by default */
>>>>>>>>>>>>>>>>>>> -    if (!src_fmt->quantization)
>>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>>>>>>>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
>>>>>>>>>>>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
>>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>> +    if (mbus_info->pixel_enc == V4L2_PIXEL_ENC_BAYER)
>>>>>>>>>>>>>>>>>>>                   src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>>>>>>> +    else if (format->quantization == V4L2_QUANTIZATION_DEFAULT)
>>>>>>>>>>>>>>>>>>> +            src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
>>>>>>>>>>>>>>>>>>> +    else
>>>>>>>>>>>>>>>>>>> +            src_fmt->quantization = format->quantization;
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>           *format = *src_fmt;
>>>>>>>>>>>>>>>>>>>        }
>>>>>>>>>>>>>>>>>>> @@ -1068,6 +1088,17 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
>>>>>>>>>>>>>>>>>>>           return ret;
>>>>>>>>>>>>>>>>>>>        }
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
>>>>>>>>>>>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt)
>>>>>>>>>>>>>>>>>>> +{
>>>>>>>>>>>>>>>>>>> +    struct rkisp1_isp *isp = &rkisp1->isp;
>>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>> +    sd_fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
>>>>>>>>>>>>>>>>>>> +    sd_fmt->pad = RKISP1_ISP_PAD_SOURCE_VIDEO;
>>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>> +    return v4l2_subdev_call(&isp->sd, pad, get_fmt, NULL, sd_fmt);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Do we need to get through the external API to access data that is
>>>>>>>>>>>>>>>>>> driver-internal anyway?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> +}
>>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>>        void rkisp1_isp_unregister(struct rkisp1_device *rkisp1)
>>>>>>>>>>>>>>>>>>>        {
>>>>>>>>>>>>>>>>>>>           struct v4l2_subdev *sd = &rkisp1->isp.sd;
>>>>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-resizer.c b/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>>>>>>>>>>>>>>> index 7b6b7ddd4169..8705b133de68 100644
>>>>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>>>>>>>>>>>>>>> @@ -525,6 +525,7 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
>>>>>>>>>>>>>>>>>>>           const struct rkisp1_isp_mbus_info *mbus_info;
>>>>>>>>>>>>>>>>>>>           struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
>>>>>>>>>>>>>>>>>>>           struct v4l2_rect *sink_crop;
>>>>>>>>>>>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>           sink_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SINK, which);
>>>>>>>>>>>>>>>>>>>           src_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SRC, which);
>>>>>>>>>>>>>>>>>>> @@ -539,8 +540,20 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
>>>>>>>>>>>>>>>>>>>           if (which == V4L2_SUBDEV_FORMAT_ACTIVE)
>>>>>>>>>>>>>>>>>>>                   rsz->pixel_enc = mbus_info->pixel_enc;
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> +    rkisp1_get_isp_src_fmt(rsz->rkisp1, &isp_sd_fmt);
>>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Is this necessary? My understanding was that in the subdev model, it was
>>>>>>>>>>>>>>>>>> the userspace responsibility to propagate any configuration changes through
>>>>>>>>>>>>>>>>>> the graph.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Also, doing this only here wouldn't fully maintain the
>>>>>>>>>>>>>>>>>> consistency of the state. For example, if one sets the ISP subdev format
>>>>>>>>>>>>>>>>>> first, then the resizer subdev and then the ISP subdev again, wouldn't the
>>>>>>>>>>>>>>>>>> resizer subdev end up with a wrong format?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> yes, this is indeed a bug, I am preparing v4 now.
>>>>>>>>>>>>>>>>> What I thought to do is adding quantization conversion
>>>>>>>>>>>>>>>>> support also on ther resizer and capture entities.
>>>>>>>>>>>>>>>>> Then in the 'link_validation' callbacks, there
>>>>>>>>>>>>>>>>> is a validation that the quantization fields matches.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> My understanding is that, if we have the following topology
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [ ISP ] -> [ Resizer ] -> [ Video node ]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> then the ISP source pad would have the csc capability, while resizer
>>>>>>>>>>>>>>>> and video node would just accept whatever is configured on their sink
>>>>>>>>>>>>>>>> pads (no need for csc capability for that) and propagate that to their
>>>>>>>>>>>>>>>> outputs, i.e. resizer source pad and video node CAPTURE format.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Is this what you were going to implement?
>>>>>>>>>>>>>> Hi, I sent a v4 where the CSC capability is set on the reiszer and capture as well.
>>>>>>>>>>>>>> I can send a v5 implementing it the way you suggest. Currently the doc says that the colorspace fields
>>>>>>>>>>>>>> must be set by the driver for capture streams. This implies that userspace can set it
>>>>>>>>>>>>>> only if the CSC is supported.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Why would the userspace have to set it for the capture stream on the
>>>>>>>>>>>>> resizer and video nodes? Couldn't the userspace set it to DEFAULT and
>>>>>>>>>>>>> then the driver override to whatever it received on the corresponding
>>>>>>>>>>>>> sink?
>>>>>>>>>>
>>>>>>>>>> According to the documentation [1]:
>>>>>>>>>>
>>>>>>>>>> "the default quantization encoding as defined by the colorspace"
>>>>>>>>>>
>>>>>>>>>> Then, for instance, the default for V4L2_COLORSPACE_SRGB is limited range [2].
>>>>>>>>>>
>>>>>>>>>> Unless we change the definition of DEFAULT to let it accept whatever it receives on the corresponding sink.
>>>>>>>>>>
>>>>>>>>>> Then the way I understand is:
>>>>>>>>>>
>>>>>>>>>> [sensor]->[isp]->[resizer]->[capture]
>>>>>>>>>>
>>>>>>>>>> Userspace should set quantization on:
>>>>>>>>>> * sensor source pad
>>>>>>>>>> * isp sink pad
>>>>>>>>>> * resizer sink pad
>>>>>>>>>> * capture node
>>>>>>>>>>
>>>>>>>>>> The remaining source pads would get values propagated from their sink pads.
>>>>>>>>>>
>>>>>>>>>> [1] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-defs.html#c.v4l2_quantization
>>>>>>>>>> [2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-details.html#col-srgb
>>>>>>>>>
>>>>>>>>> After discussing this offline, we concluded that:
>>>>>>>>> - only the source pad of the ISP entity should expose the CSC capability,
>>>>>>>>> - the CAPTURE video node should expose only the formats/color spaces
>>>>>>>>> compatible with the current setting of the resizer source,
>>>>>>>>> - consistency between the resizer source format and CAPTURE format
>>>>>>>>> should be implicitly maintained by the driver.
>>>>>>>>
>>>>>>>> Do you mean it should be validated at streamon time ?
>>>>>>>
>>>>>>> Nope. Any V4L2 state must be kept valid all the time, as mandated by
>>>>>>> the general V4L2 principles. Wrong configuration of the video node
>>>>>>> must not be a reason for STREAMON failing.
>>>>>>>
>>>>>>>> In MC-based
>>>>>>>> drivers, entities are isolated from each other from a configuration
>>>>>>>> point of view, and the validity of the configuration across the whole
>>>>>>>> pipeline is checked at streamon time. I don't think CSC parameters
>>>>>>>> should be treated differently than formats in this regard.
>>>>>>>
>>>>>>> Indeed, the ISP entity and the resizer entity are isolated from each
>>>>>>> other and they need to be validated separately. However, the resizer
>>>>>>> entity is not isolated from the video node, as there is no
>>>>>>> configuration exposed for the video node in the MC API. If MC exposed
>>>>>>> an explicit format control for the video interface sink pad, then the
>>>>>>> resizer would have to be validated separately indeed, but still that
>>>>>>> video interface sink pad would be tied to the rest of the
>>>>>>> configuration of the video node (e.g. the CAPTURE format).
>>>>>>
>>>>>> Two questions:
>>>>>>
>>>>>> The validation callback between the resizer and the isp has to include check
>>>>>> that the quantization values matches ?
>>>>>
>>>>> To sumarize:
>>>>>
>>>>> The current quantization fields from VIDIOC_SUBDEV_{S,G}_FMT and VIDEO_{SET,GET}_FORMAT
>>>>> are filled by the driver (read-only by userspace).
>>>>>
>>>>> This RFC exposes an API that allows the quantization to be a read-write configuration.
>>>>> The question is where we expose the read-write capability.
>>>>>
>>>>> We have two main design choices here.
>>>>>
>>>>> 1) Expose read-write quantization in all pads and video nodes.
>>>>> And userspace must set matching quantization in all of them.
>>>>> Which makes userspace more complicated, but userspace already needs to configure
>>>>> the whole topology anyway.
>>>>> This wouldn't make much sense in video nodes for metadata (params and statistics)
>>>>>
>>>>> In this options, the validation callback should check for matching quantizations
>>>>> between all links, and it would failed on STREAMON if formats don't match.
>>>>>
>>>>> 2-A) Expose read-write quantization only on the isp source pad.
>>>>> The others pads and video nodes would be keept read-only.
>>>>> Which means that changing quantization on a read-write node, will implicitly
>>>>> change the values on the read-only nodes.
>>>>> Which maybe not be a problem, since current API (read-only) already says this field
>>>>> is filled by the driver.
>>>>>
>>>>> In this options, the validation callback doesn't need to check for matching quantizations
>>>>> between links.
>>>>>
>>>>> 2-A) Expose read-write quantization only on the video node
>>>>> This is basically the same as 2-A, but userspace could interpret that we
>>>>> could have different quantizations in each video node.
>>>>>
>>>>>
>>>>> I'd like to hear from others what would be better here, design choice 1 or 2.
>>>>
>>>> To keep things simple, I'd go for 2-A, and I would ignore the colorspace
>>>> fields completely on the video nodes.
>>>>
>>>> In an MC-centric driver there should be no automatic in-kernel
>>>> propagation of userspace-visible configuration from a source pad to the
>>>> connected sink pad (regardless of whether the sink is a subdev or a
>>>> video node). It's userspace that is responsible for propagating the
>>>> configuration, and the kernel is supposed to validate it at stream-on
>>>> time.
>>>>
>>>> In this case, the colorspace information is only relevant to the ISP and
>>>> nobody else cares (neither the resizer nor the DMA engines) - please
>>>> correct me if I'm wrong. Given that userspace will be specific to this
>>>> particular hardware, I would not enforce matching colorspace beyond the
>>>> output of the ISP. Userspace will deal with colorspace at the output of
>>>> the ISP and ignore it on the video node.
>>>
>>> So you mean that userspace should know that the quantization of the video
>>> is the one configured on the isp, and not 'Default Quantization' as would
>>> be reported by the resizers captures?
>>
>> Yes. The Rockchip ISP requires device-specific userspace, so requiring
>> it to know about the driver doesn't seem a problem to me. I however
>> expect some people to disagree :-) We should however record this in
>> documentation somewhere, requiring userspace developers to read the
>> rkisp1 documentation is one thing, requiring them to read the driver
>> source code to figure out how to use it would be going one step too far
>> I think.
> 
> Hi, I already sent a v5 with your suggestion and already got your "reviewd-by" and Acked-by from Helen.
> [1] https://patchwork.kernel.org/patch/11642383/#23505365
> 
> But thinking of it again, the purpose of adding the CSC API was to make the code compatible
> by supporting it in the v4l2 API.
> Now we end up with a new API feature for subdevices and the rkisp1 still have a non compatible weird behavior
> that needs special documentation. It might be just a simpler solution to document the rkisp1 that although
> a quantization should be set by the driver according to the core API, for this driver it can
> also be set by userspace.
> 
> It is also not clear how other subdevices suppose to use the CSC API. Either they implement the
> same weird behavior as rkisp1 or they will have to require userspace to propagate the
> colorspace param that is set on the subdevice. It is not clear then if the CSC API should be
> supported on the other subdev nodes in the topology in order to allow it.
> 
> Thanks,
> Dafna

Thinking about it even more, we can just make the quantization conversion a capability of the video nodes.
Userspace doesn't care which of the nodes expose this capability so I don't see a problem doing it.
The big benefits are that we don't need to add the CSC API for subdevices, we add it only for video nodes
and also the rkisp1 will be compatible with the API without this weird behavior.
The CSC for video nodes is also useful for other drivers that Philipp Zabel is working on while the CSC
for subdeivces is introduced only for the rkisp1.

Thanks,
Dafna

> 
>>
>>>>>> Userspace have to configure the quantizaion on the resizer sink pad without that CSC
>>>>>> API beeing expose on it?
>>>>>
>>>>> In the current API, the quantization field is read-only, so to allow userspace to configure
>>>>> quantization in any pad, the CSC API needs to be exposed on the pad to make it read-write.
>>>>>
>>>>>>>>> Below is the rationale for that:
>>>>>>>>> - In the MC model, the video node represents the DMA alone (and thus
>>>>>>>>> the reason to have the resizer entities at all) and since the
>>>>>>>>> colorspace is not a property of the DMA, it doesn't belong there.
>>>>>>>>> - Even if we put that aside, both MP and SP can only output the same
>>>>>>>>> colorspace, which would create a dependency between the configuration
>>>>>>>>> of the two video nodes, which is not something well defined by V4L2.
>>>>>>>>> - A video interface in the MC API doesn't expose control over its
>>>>>>>>> input format, which means that the relation between the other side of
>>>>>>>>> the link (resizer source) and the video node is implicit and needs to
>>>>>>>>> be handled internally in the driver.
>>>>>>>>> - V4L2, as opposed to MC, requires that the current state is always
>>>>>>>>> consistent and valid. In this case it means that an ioctl on the video
>>>>>>>>> node must not be able to alter the state in a way that would make it
>>>>>>>>> inconsistent with the state of the matching resizer source.
>>

Re: [RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad

[Laurent Pinchart]

Hi Dafna,

On Mon, Aug 10, 2020 at 03:15:24PM +0200, Dafna Hirschfeld wrote:
> Am 10.08.20 um 14:33 schrieb Dafna Hirschfeld:
> > Am 01.07.20 um 14:52 schrieb Laurent Pinchart:
> >> On Tue, Jun 30, 2020 at 11:36:35AM +0200, Dafna Hirschfeld wrote:
> >>> On 30.06.20 01:08, Laurent Pinchart wrote:
> >>>> On Mon, Jun 29, 2020 at 12:50:06PM -0300, Helen Koike wrote:
> >>>>> On 6/28/20 4:34 PM, Dafna Hirschfeld wrote:
> >>>>>> On 28.06.20 16:59, Tomasz Figa wrote:
> >>>>>>> On Sun, Jun 28, 2020 at 3:15 PM Laurent Pinchart wrote:
> >>>>>>>> On Sun, Jun 28, 2020 at 03:10:41PM +0200, Tomasz Figa wrote:
> >>>>>>>>> On Fri, Jun 26, 2020 at 2:15 AM Helen Koike <helen.koike@collabora.com> wrote:
> >>>>>>>>>> On 6/18/20 4:25 PM, Tomasz Figa wrote:
> >>>>>>>>>>> On Thu, Jun 18, 2020 at 08:54:34PM +0200, Dafna Hirschfeld wrote:
> >>>>>>>>>>>> On 18.06.20 20:49, Tomasz Figa wrote:
> >>>>>>>>>>>>> On Thu, Jun 18, 2020 at 08:26:33PM +0200, Dafna Hirschfeld wrote:
> >>>>>>>>>>>>>> On 18.06.20 19:27, Tomasz Figa wrote:
> >>>>>>>>>>>>>>> On Wed, Jun 10, 2020 at 2:08 PM Tomasz Figa wrote:
> >>>>>>>>>>>>>>>> On Thu, Jun 4, 2020 at 9:11 PM Dafna Hirschfeld wrote:
> >>>>>>>>>>>>>>>>> On 04.06.20 19:54, Tomasz Figa wrote:
> >>>>>>>>>>>>>>>>>> On Thu, Apr 16, 2020 at 04:56:05PM +0200, Dafna Hirschfeld wrote:
> >>>>>>>>>>>>>>>>>>> The isp entity has a hardware support to force full range quantization
> >>>>>>>>>>>>>>>>>>> for YUV formats. Use the new API to indicate userspace that
> >>>>>>>>>>>>>>>>>>> quantization conversion is supported by adding the flag
> >>>>>>>>>>>>>>>>>>> V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION during media code enumeration.
> >>>>>>>>>>>>>>>>>>> Then uppon s_fmt on the video source pad, we assign the
> >>>>>>>>>>>>>>>>>>> quantization from userspace for YUV formats.
> >>>>>>>>>>>>>>>>>>> Also in the capture and resizer entities we retrieve the colorspace
> >>>>>>>>>>>>>>>>>>> from the isp entity.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
> >>>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>>        drivers/staging/media/rkisp1/rkisp1-capture.c | 28 ++++++-------
> >>>>>>>>>>>>>>>>>>>        drivers/staging/media/rkisp1/rkisp1-common.h  |  2 +
> >>>>>>>>>>>>>>>>>>>        drivers/staging/media/rkisp1/rkisp1-isp.c     | 39 +++++++++++++++++--
> >>>>>>>>>>>>>>>>>>>        drivers/staging/media/rkisp1/rkisp1-resizer.c | 13 +++++++
> >>>>>>>>>>>>>>>>>>>        4 files changed, 65 insertions(+), 17 deletions(-)
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c
> >>>>>>>>>>>>>>>>>>> index fbf62399fe3d..aca0f93bc772 100644
> >>>>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-capture.c
> >>>>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-capture.c
> >>>>>>>>>>>>>>>>>>> @@ -1066,14 +1066,13 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
> >>>>>>>>>>>>>>>>>>>                              const struct v4l2_format_info **fmt_info)
> >>>>>>>>>>>>>>>>>>>        {
> >>>>>>>>>>>>>>>>>>>           const struct rkisp1_capture_config *config = cap->config;
> >>>>>>>>>>>>>>>>>>> -    struct rkisp1_capture *other_cap =
> >>>>>>>>>>>>>>>>>>> -                    &cap->rkisp1->capture_devs[cap->id ^ 1];
> >>>>>>>>>>>>>>>>>>>           const struct rkisp1_capture_fmt_cfg *fmt;
> >>>>>>>>>>>>>>>>>>>           const struct v4l2_format_info *info;
> >>>>>>>>>>>>>>>>>>>           const unsigned int max_widths[] = { RKISP1_RSZ_MP_SRC_MAX_WIDTH,
> >>>>>>>>>>>>>>>>>>>                                               RKISP1_RSZ_SP_SRC_MAX_WIDTH };
> >>>>>>>>>>>>>>>>>>>           const unsigned int max_heights[] = { RKISP1_RSZ_MP_SRC_MAX_HEIGHT,
> >>>>>>>>>>>>>>>>>>>                                                RKISP1_RSZ_SP_SRC_MAX_HEIGHT};
> >>>>>>>>>>>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>           fmt = rkisp1_find_fmt_cfg(cap, pixm->pixelformat);
> >>>>>>>>>>>>>>>>>>>           if (!fmt) {
> >>>>>>>>>>>>>>>>>>> @@ -1081,24 +1080,27 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
> >>>>>>>>>>>>>>>>>>>                   pixm->pixelformat = fmt->fourcc;
> >>>>>>>>>>>>>>>>>>>           }
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> +    rkisp1_get_isp_src_fmt(cap->rkisp1, &isp_sd_fmt);
> >>>>>>>>>>>>>>>>>>> +    pixm->field = isp_sd_fmt.format.field;
> >>>>>>>>>>>>>>>>>>> +    pixm->colorspace = isp_sd_fmt.format.colorspace;
> >>>>>>>>>>>>>>>>>>> +    pixm->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
> >>>>>>>>>>>>>>>>>>> +    pixm->xfer_func = isp_sd_fmt.format.xfer_func;
> >>>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>>> +    /*
> >>>>>>>>>>>>>>>>>>> +     * isp has a feature to set full range quantization for yuv formats.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> How about "select between limited and full range for YUV formats"?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> +     * so we need to get the format from the isp.
> >>>>>>>>>>>>>>>>>>> +     */
> >>>>>>>>>>>>>>>>>>> +    pixm->quantization = isp_sd_fmt.format.quantization;
> >>>>>>>>>>>>>>>>>>> +    if (!v4l2_is_format_yuv(cap->pix.info))
> >>>>>>>>>>>>>>>>>>> +            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>>>           pixm->width = clamp_t(u32, pixm->width,
> >>>>>>>>>>>>>>>>>>>                                 RKISP1_RSZ_SRC_MIN_WIDTH, max_widths[cap->id]);
> >>>>>>>>>>>>>>>>>>>           pixm->height = clamp_t(u32, pixm->height,
> >>>>>>>>>>>>>>>>>>>                                  RKISP1_RSZ_SRC_MIN_HEIGHT, max_heights[cap->id]);
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> -    pixm->field = V4L2_FIELD_NONE;
> >>>>>>>>>>>>>>>>>>> -    pixm->colorspace = V4L2_COLORSPACE_DEFAULT;
> >>>>>>>>>>>>>>>>>>> -    pixm->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> >>>>>>>>>>>>>>>>>>> -
> >>>>>>>>>>>>>>>>>>>           info = rkisp1_fill_pixfmt(pixm, cap->id);
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> -    /* can not change quantization when stream-on */
> >>>>>>>>>>>>>>>>>>> -    if (other_cap->is_streaming)
> >>>>>>>>>>>>>>>>>>> -            pixm->quantization = other_cap->pix.fmt.quantization;
> >>>>>>>>>>>>>>>>>>> -    /* output full range by default, take effect in params */
> >>>>>>>>>>>>>>>>>>> -    else if (!pixm->quantization ||
> >>>>>>>>>>>>>>>>>>> -             pixm->quantization > V4L2_QUANTIZATION_LIM_RANGE)
> >>>>>>>>>>>>>>>>>>> -            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>           if (fmt_cfg)
> >>>>>>>>>>>>>>>>>>>                   *fmt_cfg = fmt;
> >>>>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-common.h b/drivers/staging/media/rkisp1/rkisp1-common.h
> >>>>>>>>>>>>>>>>>>> index 2d7b7e078636..7a5576fa14c9 100644
> >>>>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-common.h
> >>>>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-common.h
> >>>>>>>>>>>>>>>>>>> @@ -300,6 +300,8 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
> >>>>>>>>>>>>>>>>>>>                           struct v4l2_device *v4l2_dev);
> >>>>>>>>>>>>>>>>>>>        void rkisp1_isp_unregister(struct rkisp1_device *rkisp1);
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
> >>>>>>>>>>>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt);
> >>>>>>>>>>>>>>>>>>>        const struct rkisp1_isp_mbus_info *rkisp1_isp_mbus_info_get(u32 mbus_code);
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>        irqreturn_t rkisp1_isp_isr(struct rkisp1_device *rkisp1);
> >>>>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-isp.c b/drivers/staging/media/rkisp1/rkisp1-isp.c
> >>>>>>>>>>>>>>>>>>> index dee8e96f3900..6fdf5ed0b6b1 100644
> >>>>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-isp.c
> >>>>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-isp.c
> >>>>>>>>>>>>>>>>>>> @@ -613,6 +613,10 @@ static int rkisp1_isp_enum_mbus_code(struct v4l2_subdev *sd,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>                   if (code->index == pos - 1) {
> >>>>>>>>>>>>>>>>>>>                           code->code = fmt->mbus_code;
> >>>>>>>>>>>>>>>>>>> +                    if (fmt->pixel_enc == V4L2_PIXEL_ENC_YUV &&
> >>>>>>>>>>>>>>>>>>> +                        dir == RKISP1_DIR_SRC)
> >>>>>>>>>>>>>>>>>>> +                            code->flags =
> >>>>>>>>>>>>>>>>>>> +                                    V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION;
> >>>>>>>>>>>>>>>>>>>                           return 0;
> >>>>>>>>>>>>>>>>>>>                   }
> >>>>>>>>>>>>>>>>>>>           }
> >>>>>>>>>>>>>>>>>>> @@ -639,12 +643,21 @@ static int rkisp1_isp_init_config(struct v4l2_subdev *sd,
> >>>>>>>>>>>>>>>>>>>           sink_crop->height = RKISP1_DEFAULT_HEIGHT;
> >>>>>>>>>>>>>>>>>>>           sink_crop->left = 0;
> >>>>>>>>>>>>>>>>>>>           sink_crop->top = 0;
> >>>>>>>>>>>>>>>>>>> +    sink_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >>>>>>>>>>>>>>>>>>> +    sink_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(sink_fmt->colorspace);
> >>>>>>>>>>>>>>>>>>> +    sink_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(sink_fmt->colorspace);
> >>>>>>>>>>>>>>>>>>> +    sink_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>           src_fmt = v4l2_subdev_get_try_format(sd, cfg,
> >>>>>>>>>>>>>>>>>>>                                                RKISP1_ISP_PAD_SOURCE_VIDEO);
> >>>>>>>>>>>>>>>>>>>           *src_fmt = *sink_fmt;
> >>>>>>>>>>>>>>>>>>>           src_fmt->code = RKISP1_DEF_SRC_PAD_FMT;
> >>>>>>>>>>>>>>>>>>> -    src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >>>>>>>>>>>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>>>>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>>>>>>>>>>>> +    src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
> >>>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>           src_crop = v4l2_subdev_get_try_crop(sd, cfg,
> >>>>>>>>>>>>>>>>>>>                                               RKISP1_ISP_PAD_SOURCE_VIDEO);
> >>>>>>>>>>>>>>>>>>> @@ -687,10 +700,17 @@ static void rkisp1_isp_set_src_fmt(struct rkisp1_isp *isp,
> >>>>>>>>>>>>>>>>>>>                   isp->src_fmt = mbus_info;
> >>>>>>>>>>>>>>>>>>>           src_fmt->width  = src_crop->width;
> >>>>>>>>>>>>>>>>>>>           src_fmt->height = src_crop->height;
> >>>>>>>>>>>>>>>>>>> -    src_fmt->quantization = format->quantization;
> >>>>>>>>>>>>>>>>>>> -    /* full range by default */
> >>>>>>>>>>>>>>>>>>> -    if (!src_fmt->quantization)
> >>>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >>>>>>>>>>>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>>>>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>>> +    if (mbus_info->pixel_enc == V4L2_PIXEL_ENC_BAYER)
> >>>>>>>>>>>>>>>>>>>                   src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>>>>>>>> +    else if (format->quantization == V4L2_QUANTIZATION_DEFAULT)
> >>>>>>>>>>>>>>>>>>> +            src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
> >>>>>>>>>>>>>>>>>>> +    else
> >>>>>>>>>>>>>>>>>>> +            src_fmt->quantization = format->quantization;
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>           *format = *src_fmt;
> >>>>>>>>>>>>>>>>>>>        }
> >>>>>>>>>>>>>>>>>>> @@ -1068,6 +1088,17 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
> >>>>>>>>>>>>>>>>>>>           return ret;
> >>>>>>>>>>>>>>>>>>>        }
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
> >>>>>>>>>>>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt)
> >>>>>>>>>>>>>>>>>>> +{
> >>>>>>>>>>>>>>>>>>> +    struct rkisp1_isp *isp = &rkisp1->isp;
> >>>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>>> +    sd_fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
> >>>>>>>>>>>>>>>>>>> +    sd_fmt->pad = RKISP1_ISP_PAD_SOURCE_VIDEO;
> >>>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>>> +    return v4l2_subdev_call(&isp->sd, pad, get_fmt, NULL, sd_fmt);
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Do we need to get through the external API to access data that is
> >>>>>>>>>>>>>>>>>> driver-internal anyway?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> +}
> >>>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>>>        void rkisp1_isp_unregister(struct rkisp1_device *rkisp1)
> >>>>>>>>>>>>>>>>>>>        {
> >>>>>>>>>>>>>>>>>>>           struct v4l2_subdev *sd = &rkisp1->isp.sd;
> >>>>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-resizer.c b/drivers/staging/media/rkisp1/rkisp1-resizer.c
> >>>>>>>>>>>>>>>>>>> index 7b6b7ddd4169..8705b133de68 100644
> >>>>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-resizer.c
> >>>>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-resizer.c
> >>>>>>>>>>>>>>>>>>> @@ -525,6 +525,7 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
> >>>>>>>>>>>>>>>>>>>           const struct rkisp1_isp_mbus_info *mbus_info;
> >>>>>>>>>>>>>>>>>>>           struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
> >>>>>>>>>>>>>>>>>>>           struct v4l2_rect *sink_crop;
> >>>>>>>>>>>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>           sink_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SINK, which);
> >>>>>>>>>>>>>>>>>>>           src_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SRC, which);
> >>>>>>>>>>>>>>>>>>> @@ -539,8 +540,20 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
> >>>>>>>>>>>>>>>>>>>           if (which == V4L2_SUBDEV_FORMAT_ACTIVE)
> >>>>>>>>>>>>>>>>>>>                   rsz->pixel_enc = mbus_info->pixel_enc;
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> +    rkisp1_get_isp_src_fmt(rsz->rkisp1, &isp_sd_fmt);
> >>>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Is this necessary? My understanding was that in the subdev model, it was
> >>>>>>>>>>>>>>>>>> the userspace responsibility to propagate any configuration changes through
> >>>>>>>>>>>>>>>>>> the graph.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Also, doing this only here wouldn't fully maintain the
> >>>>>>>>>>>>>>>>>> consistency of the state. For example, if one sets the ISP subdev format
> >>>>>>>>>>>>>>>>>> first, then the resizer subdev and then the ISP subdev again, wouldn't the
> >>>>>>>>>>>>>>>>>> resizer subdev end up with a wrong format?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> yes, this is indeed a bug, I am preparing v4 now.
> >>>>>>>>>>>>>>>>> What I thought to do is adding quantization conversion
> >>>>>>>>>>>>>>>>> support also on ther resizer and capture entities.
> >>>>>>>>>>>>>>>>> Then in the 'link_validation' callbacks, there
> >>>>>>>>>>>>>>>>> is a validation that the quantization fields matches.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> My understanding is that, if we have the following topology
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> [ ISP ] -> [ Resizer ] -> [ Video node ]
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> then the ISP source pad would have the csc capability, while resizer
> >>>>>>>>>>>>>>>> and video node would just accept whatever is configured on their sink
> >>>>>>>>>>>>>>>> pads (no need for csc capability for that) and propagate that to their
> >>>>>>>>>>>>>>>> outputs, i.e. resizer source pad and video node CAPTURE format.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Is this what you were going to implement?
> >>>>>>>>>>>>>> Hi, I sent a v4 where the CSC capability is set on the reiszer and capture as well.
> >>>>>>>>>>>>>> I can send a v5 implementing it the way you suggest. Currently the doc says that the colorspace fields
> >>>>>>>>>>>>>> must be set by the driver for capture streams. This implies that userspace can set it
> >>>>>>>>>>>>>> only if the CSC is supported.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Why would the userspace have to set it for the capture stream on the
> >>>>>>>>>>>>> resizer and video nodes? Couldn't the userspace set it to DEFAULT and
> >>>>>>>>>>>>> then the driver override to whatever it received on the corresponding
> >>>>>>>>>>>>> sink?
> >>>>>>>>>>
> >>>>>>>>>> According to the documentation [1]:
> >>>>>>>>>>
> >>>>>>>>>> "the default quantization encoding as defined by the colorspace"
> >>>>>>>>>>
> >>>>>>>>>> Then, for instance, the default for V4L2_COLORSPACE_SRGB is limited range [2].
> >>>>>>>>>>
> >>>>>>>>>> Unless we change the definition of DEFAULT to let it accept whatever it receives on the corresponding sink.
> >>>>>>>>>>
> >>>>>>>>>> Then the way I understand is:
> >>>>>>>>>>
> >>>>>>>>>> [sensor]->[isp]->[resizer]->[capture]
> >>>>>>>>>>
> >>>>>>>>>> Userspace should set quantization on:
> >>>>>>>>>> * sensor source pad
> >>>>>>>>>> * isp sink pad
> >>>>>>>>>> * resizer sink pad
> >>>>>>>>>> * capture node
> >>>>>>>>>>
> >>>>>>>>>> The remaining source pads would get values propagated from their sink pads.
> >>>>>>>>>>
> >>>>>>>>>> [1] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-defs.html#c.v4l2_quantization
> >>>>>>>>>> [2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-details.html#col-srgb
> >>>>>>>>>
> >>>>>>>>> After discussing this offline, we concluded that:
> >>>>>>>>> - only the source pad of the ISP entity should expose the CSC capability,
> >>>>>>>>> - the CAPTURE video node should expose only the formats/color spaces
> >>>>>>>>> compatible with the current setting of the resizer source,
> >>>>>>>>> - consistency between the resizer source format and CAPTURE format
> >>>>>>>>> should be implicitly maintained by the driver.
> >>>>>>>>
> >>>>>>>> Do you mean it should be validated at streamon time ?
> >>>>>>>
> >>>>>>> Nope. Any V4L2 state must be kept valid all the time, as mandated by
> >>>>>>> the general V4L2 principles. Wrong configuration of the video node
> >>>>>>> must not be a reason for STREAMON failing.
> >>>>>>>
> >>>>>>>> In MC-based
> >>>>>>>> drivers, entities are isolated from each other from a configuration
> >>>>>>>> point of view, and the validity of the configuration across the whole
> >>>>>>>> pipeline is checked at streamon time. I don't think CSC parameters
> >>>>>>>> should be treated differently than formats in this regard.
> >>>>>>>
> >>>>>>> Indeed, the ISP entity and the resizer entity are isolated from each
> >>>>>>> other and they need to be validated separately. However, the resizer
> >>>>>>> entity is not isolated from the video node, as there is no
> >>>>>>> configuration exposed for the video node in the MC API. If MC exposed
> >>>>>>> an explicit format control for the video interface sink pad, then the
> >>>>>>> resizer would have to be validated separately indeed, but still that
> >>>>>>> video interface sink pad would be tied to the rest of the
> >>>>>>> configuration of the video node (e.g. the CAPTURE format).
> >>>>>>
> >>>>>> Two questions:
> >>>>>>
> >>>>>> The validation callback between the resizer and the isp has to include check
> >>>>>> that the quantization values matches ?
> >>>>>
> >>>>> To sumarize:
> >>>>>
> >>>>> The current quantization fields from VIDIOC_SUBDEV_{S,G}_FMT and VIDEO_{SET,GET}_FORMAT
> >>>>> are filled by the driver (read-only by userspace).
> >>>>>
> >>>>> This RFC exposes an API that allows the quantization to be a read-write configuration.
> >>>>> The question is where we expose the read-write capability.
> >>>>>
> >>>>> We have two main design choices here.
> >>>>>
> >>>>> 1) Expose read-write quantization in all pads and video nodes.
> >>>>> And userspace must set matching quantization in all of them.
> >>>>> Which makes userspace more complicated, but userspace already needs to configure
> >>>>> the whole topology anyway.
> >>>>> This wouldn't make much sense in video nodes for metadata (params and statistics)
> >>>>>
> >>>>> In this options, the validation callback should check for matching quantizations
> >>>>> between all links, and it would failed on STREAMON if formats don't match.
> >>>>>
> >>>>> 2-A) Expose read-write quantization only on the isp source pad.
> >>>>> The others pads and video nodes would be keept read-only.
> >>>>> Which means that changing quantization on a read-write node, will implicitly
> >>>>> change the values on the read-only nodes.
> >>>>> Which maybe not be a problem, since current API (read-only) already says this field
> >>>>> is filled by the driver.
> >>>>>
> >>>>> In this options, the validation callback doesn't need to check for matching quantizations
> >>>>> between links.
> >>>>>
> >>>>> 2-A) Expose read-write quantization only on the video node
> >>>>> This is basically the same as 2-A, but userspace could interpret that we
> >>>>> could have different quantizations in each video node.
> >>>>>
> >>>>>
> >>>>> I'd like to hear from others what would be better here, design choice 1 or 2.
> >>>>
> >>>> To keep things simple, I'd go for 2-A, and I would ignore the colorspace
> >>>> fields completely on the video nodes.
> >>>>
> >>>> In an MC-centric driver there should be no automatic in-kernel
> >>>> propagation of userspace-visible configuration from a source pad to the
> >>>> connected sink pad (regardless of whether the sink is a subdev or a
> >>>> video node). It's userspace that is responsible for propagating the
> >>>> configuration, and the kernel is supposed to validate it at stream-on
> >>>> time.
> >>>>
> >>>> In this case, the colorspace information is only relevant to the ISP and
> >>>> nobody else cares (neither the resizer nor the DMA engines) - please
> >>>> correct me if I'm wrong. Given that userspace will be specific to this
> >>>> particular hardware, I would not enforce matching colorspace beyond the
> >>>> output of the ISP. Userspace will deal with colorspace at the output of
> >>>> the ISP and ignore it on the video node.
> >>>
> >>> So you mean that userspace should know that the quantization of the video
> >>> is the one configured on the isp, and not 'Default Quantization' as would
> >>> be reported by the resizers captures?
> >>
> >> Yes. The Rockchip ISP requires device-specific userspace, so requiring
> >> it to know about the driver doesn't seem a problem to me. I however
> >> expect some people to disagree :-) We should however record this in
> >> documentation somewhere, requiring userspace developers to read the
> >> rkisp1 documentation is one thing, requiring them to read the driver
> >> source code to figure out how to use it would be going one step too far
> >> I think.
> > 
> > Hi, I already sent a v5 with your suggestion and already got your "reviewd-by" and Acked-by from Helen.
> > [1] https://patchwork.kernel.org/patch/11642383/#23505365
> > 
> > But thinking of it again, the purpose of adding the CSC API was to make the code compatible
> > by supporting it in the v4l2 API.
> > Now we end up with a new API feature for subdevices and the rkisp1 still have a non compatible weird behavior
> > that needs special documentation. It might be just a simpler solution to document the rkisp1 that although
> > a quantization should be set by the driver according to the core API, for this driver it can
> > also be set by userspace.
> > 
> > It is also not clear how other subdevices suppose to use the CSC API. Either they implement the
> > same weird behavior as rkisp1 or they will have to require userspace to propagate the
> > colorspace param that is set on the subdevice. It is not clear then if the CSC API should be
> > supported on the other subdev nodes in the topology in order to allow it.
> > 
> > Thanks,
> > Dafna
> 
> Thinking about it even more, we can just make the quantization conversion a capability of the video nodes.
> Userspace doesn't care which of the nodes expose this capability so I don't see a problem doing it.
> The big benefits are that we don't need to add the CSC API for subdevices, we add it only for video nodes
> and also the rkisp1 will be compatible with the API without this weird behavior.

Please, no. For MC-centric devices, the video node controls the DMA
engine and the DMA engine only. We don't want to propagate *any*
configuration parameter from the video node to the subdevs. It's the job
of userspace to configure subdevs correctly. Otherwise we'll start by a
tiny change like this, and before we know, it will be full of hacks all
over the place. We created the MC API to remove policy decisions from
the kernel, let's not add them back.

> The CSC for video nodes is also useful for other drivers that Philipp Zabel is working on while the CSC
> for subdeivces is introduced only for the rkisp1.

I don't dispute that CSC on video nodes can be useful, just not in this
case.

> >>>>>> Userspace have to configure the quantizaion on the resizer sink pad without that CSC
> >>>>>> API beeing expose on it?
> >>>>>
> >>>>> In the current API, the quantization field is read-only, so to allow userspace to configure
> >>>>> quantization in any pad, the CSC API needs to be exposed on the pad to make it read-write.
> >>>>>
> >>>>>>>>> Below is the rationale for that:
> >>>>>>>>> - In the MC model, the video node represents the DMA alone (and thus
> >>>>>>>>> the reason to have the resizer entities at all) and since the
> >>>>>>>>> colorspace is not a property of the DMA, it doesn't belong there.
> >>>>>>>>> - Even if we put that aside, both MP and SP can only output the same
> >>>>>>>>> colorspace, which would create a dependency between the configuration
> >>>>>>>>> of the two video nodes, which is not something well defined by V4L2.
> >>>>>>>>> - A video interface in the MC API doesn't expose control over its
> >>>>>>>>> input format, which means that the relation between the other side of
> >>>>>>>>> the link (resizer source) and the video node is implicit and needs to
> >>>>>>>>> be handled internally in the driver.
> >>>>>>>>> - V4L2, as opposed to MC, requires that the current state is always
> >>>>>>>>> consistent and valid. In this case it means that an ioctl on the video
> >>>>>>>>> node must not be able to alter the state in a way that would make it
> >>>>>>>>> inconsistent with the state of the matching resizer source.

-- 
Regards,

Laurent Pinchart

Re: [RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad

[Laurent Pinchart]

Hi Tomasz,

On Sun, Jun 28, 2020 at 04:59:34PM +0200, Tomasz Figa wrote:
> On Sun, Jun 28, 2020 at 3:15 PM Laurent Pinchart wrote:
> > On Sun, Jun 28, 2020 at 03:10:41PM +0200, Tomasz Figa wrote:
> >> On Fri, Jun 26, 2020 at 2:15 AM Helen Koike <helen.koike@collabora.com> wrote:
> >>> On 6/18/20 4:25 PM, Tomasz Figa wrote:
> >>>> On Thu, Jun 18, 2020 at 08:54:34PM +0200, Dafna Hirschfeld wrote:
> >>>>> On 18.06.20 20:49, Tomasz Figa wrote:
> >>>>>> On Thu, Jun 18, 2020 at 08:26:33PM +0200, Dafna Hirschfeld wrote:
> >>>>>>> On 18.06.20 19:27, Tomasz Figa wrote:
> >>>>>>>> On Wed, Jun 10, 2020 at 2:08 PM Tomasz Figa wrote:
> >>>>>>>>> On Thu, Jun 4, 2020 at 9:11 PM Dafna Hirschfeld wrote:
> >>>>>>>>>> On 04.06.20 19:54, Tomasz Figa wrote:
> >>>>>>>>>>> On Thu, Apr 16, 2020 at 04:56:05PM +0200, Dafna Hirschfeld wrote:
> >>>>>>>>>>>> The isp entity has a hardware support to force full range quantization
> >>>>>>>>>>>> for YUV formats. Use the new API to indicate userspace that
> >>>>>>>>>>>> quantization conversion is supported by adding the flag
> >>>>>>>>>>>> V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION during media code enumeration.
> >>>>>>>>>>>> Then uppon s_fmt on the video source pad, we assign the
> >>>>>>>>>>>> quantization from userspace for YUV formats.
> >>>>>>>>>>>> Also in the capture and resizer entities we retrieve the colorspace
> >>>>>>>>>>>> from the isp entity.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
> >>>>>>>>>>>> ---
> >>>>>>>>>>>>     drivers/staging/media/rkisp1/rkisp1-capture.c | 28 ++++++-------
> >>>>>>>>>>>>     drivers/staging/media/rkisp1/rkisp1-common.h  |  2 +
> >>>>>>>>>>>>     drivers/staging/media/rkisp1/rkisp1-isp.c     | 39 +++++++++++++++++--
> >>>>>>>>>>>>     drivers/staging/media/rkisp1/rkisp1-resizer.c | 13 +++++++
> >>>>>>>>>>>>     4 files changed, 65 insertions(+), 17 deletions(-)
> >>>>>>>>>>>>
> >>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c
> >>>>>>>>>>>> index fbf62399fe3d..aca0f93bc772 100644
> >>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-capture.c
> >>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-capture.c
> >>>>>>>>>>>> @@ -1066,14 +1066,13 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
> >>>>>>>>>>>>                           const struct v4l2_format_info **fmt_info)
> >>>>>>>>>>>>     {
> >>>>>>>>>>>>        const struct rkisp1_capture_config *config = cap->config;
> >>>>>>>>>>>> -    struct rkisp1_capture *other_cap =
> >>>>>>>>>>>> -                    &cap->rkisp1->capture_devs[cap->id ^ 1];
> >>>>>>>>>>>>        const struct rkisp1_capture_fmt_cfg *fmt;
> >>>>>>>>>>>>        const struct v4l2_format_info *info;
> >>>>>>>>>>>>        const unsigned int max_widths[] = { RKISP1_RSZ_MP_SRC_MAX_WIDTH,
> >>>>>>>>>>>>                                            RKISP1_RSZ_SP_SRC_MAX_WIDTH };
> >>>>>>>>>>>>        const unsigned int max_heights[] = { RKISP1_RSZ_MP_SRC_MAX_HEIGHT,
> >>>>>>>>>>>>                                             RKISP1_RSZ_SP_SRC_MAX_HEIGHT};
> >>>>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
> >>>>>>>>>>>>
> >>>>>>>>>>>>        fmt = rkisp1_find_fmt_cfg(cap, pixm->pixelformat);
> >>>>>>>>>>>>        if (!fmt) {
> >>>>>>>>>>>> @@ -1081,24 +1080,27 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
> >>>>>>>>>>>>                pixm->pixelformat = fmt->fourcc;
> >>>>>>>>>>>>        }
> >>>>>>>>>>>>
> >>>>>>>>>>>> +    rkisp1_get_isp_src_fmt(cap->rkisp1, &isp_sd_fmt);
> >>>>>>>>>>>> +    pixm->field = isp_sd_fmt.format.field;
> >>>>>>>>>>>> +    pixm->colorspace = isp_sd_fmt.format.colorspace;
> >>>>>>>>>>>> +    pixm->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
> >>>>>>>>>>>> +    pixm->xfer_func = isp_sd_fmt.format.xfer_func;
> >>>>>>>>>>>> +
> >>>>>>>>>>>> +    /*
> >>>>>>>>>>>> +     * isp has a feature to set full range quantization for yuv formats.
> >>>>>>>>>>>
> >>>>>>>>>>> How about "select between limited and full range for YUV formats"?
> >>>>>>>>>>>
> >>>>>>>>>>>> +     * so we need to get the format from the isp.
> >>>>>>>>>>>> +     */
> >>>>>>>>>>>> +    pixm->quantization = isp_sd_fmt.format.quantization;
> >>>>>>>>>>>> +    if (!v4l2_is_format_yuv(cap->pix.info))
> >>>>>>>>>>>> +            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>> +
> >>>>>>>>>>>>        pixm->width = clamp_t(u32, pixm->width,
> >>>>>>>>>>>>                              RKISP1_RSZ_SRC_MIN_WIDTH, max_widths[cap->id]);
> >>>>>>>>>>>>        pixm->height = clamp_t(u32, pixm->height,
> >>>>>>>>>>>>                               RKISP1_RSZ_SRC_MIN_HEIGHT, max_heights[cap->id]);
> >>>>>>>>>>>>
> >>>>>>>>>>>> -    pixm->field = V4L2_FIELD_NONE;
> >>>>>>>>>>>> -    pixm->colorspace = V4L2_COLORSPACE_DEFAULT;
> >>>>>>>>>>>> -    pixm->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> >>>>>>>>>>>> -
> >>>>>>>>>>>>        info = rkisp1_fill_pixfmt(pixm, cap->id);
> >>>>>>>>>>>>
> >>>>>>>>>>>> -    /* can not change quantization when stream-on */
> >>>>>>>>>>>> -    if (other_cap->is_streaming)
> >>>>>>>>>>>> -            pixm->quantization = other_cap->pix.fmt.quantization;
> >>>>>>>>>>>> -    /* output full range by default, take effect in params */
> >>>>>>>>>>>> -    else if (!pixm->quantization ||
> >>>>>>>>>>>> -             pixm->quantization > V4L2_QUANTIZATION_LIM_RANGE)
> >>>>>>>>>>>> -            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>
> >>>>>>>>>>>>        if (fmt_cfg)
> >>>>>>>>>>>>                *fmt_cfg = fmt;
> >>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-common.h b/drivers/staging/media/rkisp1/rkisp1-common.h
> >>>>>>>>>>>> index 2d7b7e078636..7a5576fa14c9 100644
> >>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-common.h
> >>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-common.h
> >>>>>>>>>>>> @@ -300,6 +300,8 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
> >>>>>>>>>>>>                        struct v4l2_device *v4l2_dev);
> >>>>>>>>>>>>     void rkisp1_isp_unregister(struct rkisp1_device *rkisp1);
> >>>>>>>>>>>>
> >>>>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
> >>>>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt);
> >>>>>>>>>>>>     const struct rkisp1_isp_mbus_info *rkisp1_isp_mbus_info_get(u32 mbus_code);
> >>>>>>>>>>>>
> >>>>>>>>>>>>     irqreturn_t rkisp1_isp_isr(struct rkisp1_device *rkisp1);
> >>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-isp.c b/drivers/staging/media/rkisp1/rkisp1-isp.c
> >>>>>>>>>>>> index dee8e96f3900..6fdf5ed0b6b1 100644
> >>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-isp.c
> >>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-isp.c
> >>>>>>>>>>>> @@ -613,6 +613,10 @@ static int rkisp1_isp_enum_mbus_code(struct v4l2_subdev *sd,
> >>>>>>>>>>>>
> >>>>>>>>>>>>                if (code->index == pos - 1) {
> >>>>>>>>>>>>                        code->code = fmt->mbus_code;
> >>>>>>>>>>>> +                    if (fmt->pixel_enc == V4L2_PIXEL_ENC_YUV &&
> >>>>>>>>>>>> +                        dir == RKISP1_DIR_SRC)
> >>>>>>>>>>>> +                            code->flags =
> >>>>>>>>>>>> +                                    V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION;
> >>>>>>>>>>>>                        return 0;
> >>>>>>>>>>>>                }
> >>>>>>>>>>>>        }
> >>>>>>>>>>>> @@ -639,12 +643,21 @@ static int rkisp1_isp_init_config(struct v4l2_subdev *sd,
> >>>>>>>>>>>>        sink_crop->height = RKISP1_DEFAULT_HEIGHT;
> >>>>>>>>>>>>        sink_crop->left = 0;
> >>>>>>>>>>>>        sink_crop->top = 0;
> >>>>>>>>>>>> +    sink_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >>>>>>>>>>>> +    sink_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(sink_fmt->colorspace);
> >>>>>>>>>>>> +    sink_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(sink_fmt->colorspace);
> >>>>>>>>>>>> +    sink_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>> +
> >>>>>>>>>>>>
> >>>>>>>>>>>>        src_fmt = v4l2_subdev_get_try_format(sd, cfg,
> >>>>>>>>>>>>                                             RKISP1_ISP_PAD_SOURCE_VIDEO);
> >>>>>>>>>>>>        *src_fmt = *sink_fmt;
> >>>>>>>>>>>>        src_fmt->code = RKISP1_DEF_SRC_PAD_FMT;
> >>>>>>>>>>>> -    src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >>>>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>>>>> +    src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
> >>>>>>>>>>>> +
> >>>>>>>>>>>>
> >>>>>>>>>>>>        src_crop = v4l2_subdev_get_try_crop(sd, cfg,
> >>>>>>>>>>>>                                            RKISP1_ISP_PAD_SOURCE_VIDEO);
> >>>>>>>>>>>> @@ -687,10 +700,17 @@ static void rkisp1_isp_set_src_fmt(struct rkisp1_isp *isp,
> >>>>>>>>>>>>                isp->src_fmt = mbus_info;
> >>>>>>>>>>>>        src_fmt->width  = src_crop->width;
> >>>>>>>>>>>>        src_fmt->height = src_crop->height;
> >>>>>>>>>>>> -    src_fmt->quantization = format->quantization;
> >>>>>>>>>>>> -    /* full range by default */
> >>>>>>>>>>>> -    if (!src_fmt->quantization)
> >>>>>>>>>>>> +
> >>>>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >>>>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>>>>> +
> >>>>>>>>>>>> +    if (mbus_info->pixel_enc == V4L2_PIXEL_ENC_BAYER)
> >>>>>>>>>>>>                src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>> +    else if (format->quantization == V4L2_QUANTIZATION_DEFAULT)
> >>>>>>>>>>>> +            src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
> >>>>>>>>>>>> +    else
> >>>>>>>>>>>> +            src_fmt->quantization = format->quantization;
> >>>>>>>>>>>>
> >>>>>>>>>>>>        *format = *src_fmt;
> >>>>>>>>>>>>     }
> >>>>>>>>>>>> @@ -1068,6 +1088,17 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
> >>>>>>>>>>>>        return ret;
> >>>>>>>>>>>>     }
> >>>>>>>>>>>>
> >>>>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
> >>>>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt)
> >>>>>>>>>>>> +{
> >>>>>>>>>>>> +    struct rkisp1_isp *isp = &rkisp1->isp;
> >>>>>>>>>>>> +
> >>>>>>>>>>>> +    sd_fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
> >>>>>>>>>>>> +    sd_fmt->pad = RKISP1_ISP_PAD_SOURCE_VIDEO;
> >>>>>>>>>>>> +
> >>>>>>>>>>>> +    return v4l2_subdev_call(&isp->sd, pad, get_fmt, NULL, sd_fmt);
> >>>>>>>>>>>
> >>>>>>>>>>> Do we need to get through the external API to access data that is
> >>>>>>>>>>> driver-internal anyway?
> >>>>>>>>>>>
> >>>>>>>>>>>> +}
> >>>>>>>>>>>> +
> >>>>>>>>>>>>     void rkisp1_isp_unregister(struct rkisp1_device *rkisp1)
> >>>>>>>>>>>>     {
> >>>>>>>>>>>>        struct v4l2_subdev *sd = &rkisp1->isp.sd;
> >>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-resizer.c b/drivers/staging/media/rkisp1/rkisp1-resizer.c
> >>>>>>>>>>>> index 7b6b7ddd4169..8705b133de68 100644
> >>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-resizer.c
> >>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-resizer.c
> >>>>>>>>>>>> @@ -525,6 +525,7 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
> >>>>>>>>>>>>        const struct rkisp1_isp_mbus_info *mbus_info;
> >>>>>>>>>>>>        struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
> >>>>>>>>>>>>        struct v4l2_rect *sink_crop;
> >>>>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
> >>>>>>>>>>>>
> >>>>>>>>>>>>        sink_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SINK, which);
> >>>>>>>>>>>>        src_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SRC, which);
> >>>>>>>>>>>> @@ -539,8 +540,20 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
> >>>>>>>>>>>>        if (which == V4L2_SUBDEV_FORMAT_ACTIVE)
> >>>>>>>>>>>>                rsz->pixel_enc = mbus_info->pixel_enc;
> >>>>>>>>>>>>
> >>>>>>>>>>>> +    rkisp1_get_isp_src_fmt(rsz->rkisp1, &isp_sd_fmt);
> >>>>>>>>>>>> +
> >>>>>>>>>>>
> >>>>>>>>>>> Is this necessary? My understanding was that in the subdev model, it was
> >>>>>>>>>>> the userspace responsibility to propagate any configuration changes through
> >>>>>>>>>>> the graph.
> >>>>>>>>>>>
> >>>>>>>>>>> Also, doing this only here wouldn't fully maintain the
> >>>>>>>>>>> consistency of the state. For example, if one sets the ISP subdev format
> >>>>>>>>>>> first, then the resizer subdev and then the ISP subdev again, wouldn't the
> >>>>>>>>>>> resizer subdev end up with a wrong format?
> >>>>>>>>>>
> >>>>>>>>>> yes, this is indeed a bug, I am preparing v4 now.
> >>>>>>>>>> What I thought to do is adding quantization conversion
> >>>>>>>>>> support also on ther resizer and capture entities.
> >>>>>>>>>> Then in the 'link_validation' callbacks, there
> >>>>>>>>>> is a validation that the quantization fields matches.
> >>>>>>>>>
> >>>>>>>>> My understanding is that, if we have the following topology
> >>>>>>>>>
> >>>>>>>>> [ ISP ] -> [ Resizer ] -> [ Video node ]
> >>>>>>>>>
> >>>>>>>>> then the ISP source pad would have the csc capability, while resizer
> >>>>>>>>> and video node would just accept whatever is configured on their sink
> >>>>>>>>> pads (no need for csc capability for that) and propagate that to their
> >>>>>>>>> outputs, i.e. resizer source pad and video node CAPTURE format.
> >>>>>>>>>
> >>>>>>>>> Is this what you were going to implement?
> >>>>>>> Hi, I sent a v4 where the CSC capability is set on the reiszer and capture as well.
> >>>>>>> I can send a v5 implementing it the way you suggest. Currently the doc says that the colorspace fields
> >>>>>>> must be set by the driver for capture streams. This implies that userspace can set it
> >>>>>>> only if the CSC is supported.
> >>>>>>
> >>>>>> Why would the userspace have to set it for the capture stream on the
> >>>>>> resizer and video nodes? Couldn't the userspace set it to DEFAULT and
> >>>>>> then the driver override to whatever it received on the corresponding
> >>>>>> sink?
> >>>
> >>> According to the documentation [1]:
> >>>
> >>> "the default quantization encoding as defined by the colorspace"
> >>>
> >>> Then, for instance, the default for V4L2_COLORSPACE_SRGB is limited range [2].
> >>>
> >>> Unless we change the definition of DEFAULT to let it accept whatever it receives on the corresponding sink.
> >>>
> >>> Then the way I understand is:
> >>>
> >>> [sensor]->[isp]->[resizer]->[capture]
> >>>
> >>> Userspace should set quantization on:
> >>> * sensor source pad
> >>> * isp sink pad
> >>> * resizer sink pad
> >>> * capture node
> >>>
> >>> The remaining source pads would get values propagated from their sink pads.
> >>>
> >>> [1] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-defs.html#c.v4l2_quantization
> >>> [2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-details.html#col-srgb
> >>
> >> After discussing this offline, we concluded that:
> >> - only the source pad of the ISP entity should expose the CSC capability,
> >> - the CAPTURE video node should expose only the formats/color spaces
> >> compatible with the current setting of the resizer source,
> >> - consistency between the resizer source format and CAPTURE format
> >> should be implicitly maintained by the driver.
> >
> > Do you mean it should be validated at streamon time ?
> 
> Nope. Any V4L2 state must be kept valid all the time, as mandated by
> the general V4L2 principles. Wrong configuration of the video node
> must not be a reason for STREAMON failing.

That's a wrong interpretation of V4L2. On a video-node-centric device (a
device that doesn't use the subdev userspace API for configuration) this
is correct. In a MC-centric device, the video node and its connected
subdev are configured separately.

> > In MC-based
> > drivers, entities are isolated from each other from a configuration
> > point of view, and the validity of the configuration across the whole
> > pipeline is checked at streamon time. I don't think CSC parameters
> > should be treated differently than formats in this regard.
> 
> Indeed, the ISP entity and the resizer entity are isolated from each
> other and they need to be validated separately. However, the resizer
> entity is not isolated from the video node, as there is no
> configuration exposed for the video node in the MC API. If MC exposed
> an explicit format control for the video interface sink pad, then the
> resizer would have to be validated separately indeed, but still that
> video interface sink pad would be tied to the rest of the
> configuration of the video node (e.g. the CAPTURE format).
> 
> >> Below is the rationale for that:
> >> - In the MC model, the video node represents the DMA alone (and thus
> >> the reason to have the resizer entities at all) and since the
> >> colorspace is not a property of the DMA, it doesn't belong there.
> >> - Even if we put that aside, both MP and SP can only output the same
> >> colorspace, which would create a dependency between the configuration
> >> of the two video nodes, which is not something well defined by V4L2.
> >> - A video interface in the MC API doesn't expose control over its
> >> input format, which means that the relation between the other side of
> >> the link (resizer source) and the video node is implicit and needs to
> >> be handled internally in the driver.
> >> - V4L2, as opposed to MC, requires that the current state is always
> >> consistent and valid. In this case it means that an ioctl on the video
> >> node must not be able to alter the state in a way that would make it
> >> inconsistent with the state of the matching resizer source.

-- 
Regards,

Laurent Pinchart

Re: [RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad

[Tomasz Figa]

On Sun, Jun 28, 2020 at 10:27 PM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> Hi Tomasz,
>
> On Sun, Jun 28, 2020 at 04:59:34PM +0200, Tomasz Figa wrote:
> > On Sun, Jun 28, 2020 at 3:15 PM Laurent Pinchart wrote:
> > > On Sun, Jun 28, 2020 at 03:10:41PM +0200, Tomasz Figa wrote:
> > >> On Fri, Jun 26, 2020 at 2:15 AM Helen Koike <helen.koike@collabora.com> wrote:
> > >>> On 6/18/20 4:25 PM, Tomasz Figa wrote:
> > >>>> On Thu, Jun 18, 2020 at 08:54:34PM +0200, Dafna Hirschfeld wrote:
> > >>>>> On 18.06.20 20:49, Tomasz Figa wrote:
> > >>>>>> On Thu, Jun 18, 2020 at 08:26:33PM +0200, Dafna Hirschfeld wrote:
> > >>>>>>> On 18.06.20 19:27, Tomasz Figa wrote:
> > >>>>>>>> On Wed, Jun 10, 2020 at 2:08 PM Tomasz Figa wrote:
> > >>>>>>>>> On Thu, Jun 4, 2020 at 9:11 PM Dafna Hirschfeld wrote:
> > >>>>>>>>>> On 04.06.20 19:54, Tomasz Figa wrote:
> > >>>>>>>>>>> On Thu, Apr 16, 2020 at 04:56:05PM +0200, Dafna Hirschfeld wrote:
> > >>>>>>>>>>>> The isp entity has a hardware support to force full range quantization
> > >>>>>>>>>>>> for YUV formats. Use the new API to indicate userspace that
> > >>>>>>>>>>>> quantization conversion is supported by adding the flag
> > >>>>>>>>>>>> V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION during media code enumeration.
> > >>>>>>>>>>>> Then uppon s_fmt on the video source pad, we assign the
> > >>>>>>>>>>>> quantization from userspace for YUV formats.
> > >>>>>>>>>>>> Also in the capture and resizer entities we retrieve the colorspace
> > >>>>>>>>>>>> from the isp entity.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
> > >>>>>>>>>>>> ---
> > >>>>>>>>>>>>     drivers/staging/media/rkisp1/rkisp1-capture.c | 28 ++++++-------
> > >>>>>>>>>>>>     drivers/staging/media/rkisp1/rkisp1-common.h  |  2 +
> > >>>>>>>>>>>>     drivers/staging/media/rkisp1/rkisp1-isp.c     | 39 +++++++++++++++++--
> > >>>>>>>>>>>>     drivers/staging/media/rkisp1/rkisp1-resizer.c | 13 +++++++
> > >>>>>>>>>>>>     4 files changed, 65 insertions(+), 17 deletions(-)
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c
> > >>>>>>>>>>>> index fbf62399fe3d..aca0f93bc772 100644
> > >>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-capture.c
> > >>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-capture.c
> > >>>>>>>>>>>> @@ -1066,14 +1066,13 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
> > >>>>>>>>>>>>                           const struct v4l2_format_info **fmt_info)
> > >>>>>>>>>>>>     {
> > >>>>>>>>>>>>        const struct rkisp1_capture_config *config = cap->config;
> > >>>>>>>>>>>> -    struct rkisp1_capture *other_cap =
> > >>>>>>>>>>>> -                    &cap->rkisp1->capture_devs[cap->id ^ 1];
> > >>>>>>>>>>>>        const struct rkisp1_capture_fmt_cfg *fmt;
> > >>>>>>>>>>>>        const struct v4l2_format_info *info;
> > >>>>>>>>>>>>        const unsigned int max_widths[] = { RKISP1_RSZ_MP_SRC_MAX_WIDTH,
> > >>>>>>>>>>>>                                            RKISP1_RSZ_SP_SRC_MAX_WIDTH };
> > >>>>>>>>>>>>        const unsigned int max_heights[] = { RKISP1_RSZ_MP_SRC_MAX_HEIGHT,
> > >>>>>>>>>>>>                                             RKISP1_RSZ_SP_SRC_MAX_HEIGHT};
> > >>>>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>        fmt = rkisp1_find_fmt_cfg(cap, pixm->pixelformat);
> > >>>>>>>>>>>>        if (!fmt) {
> > >>>>>>>>>>>> @@ -1081,24 +1080,27 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
> > >>>>>>>>>>>>                pixm->pixelformat = fmt->fourcc;
> > >>>>>>>>>>>>        }
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> +    rkisp1_get_isp_src_fmt(cap->rkisp1, &isp_sd_fmt);
> > >>>>>>>>>>>> +    pixm->field = isp_sd_fmt.format.field;
> > >>>>>>>>>>>> +    pixm->colorspace = isp_sd_fmt.format.colorspace;
> > >>>>>>>>>>>> +    pixm->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
> > >>>>>>>>>>>> +    pixm->xfer_func = isp_sd_fmt.format.xfer_func;
> > >>>>>>>>>>>> +
> > >>>>>>>>>>>> +    /*
> > >>>>>>>>>>>> +     * isp has a feature to set full range quantization for yuv formats.
> > >>>>>>>>>>>
> > >>>>>>>>>>> How about "select between limited and full range for YUV formats"?
> > >>>>>>>>>>>
> > >>>>>>>>>>>> +     * so we need to get the format from the isp.
> > >>>>>>>>>>>> +     */
> > >>>>>>>>>>>> +    pixm->quantization = isp_sd_fmt.format.quantization;
> > >>>>>>>>>>>> +    if (!v4l2_is_format_yuv(cap->pix.info))
> > >>>>>>>>>>>> +            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> > >>>>>>>>>>>> +
> > >>>>>>>>>>>>        pixm->width = clamp_t(u32, pixm->width,
> > >>>>>>>>>>>>                              RKISP1_RSZ_SRC_MIN_WIDTH, max_widths[cap->id]);
> > >>>>>>>>>>>>        pixm->height = clamp_t(u32, pixm->height,
> > >>>>>>>>>>>>                               RKISP1_RSZ_SRC_MIN_HEIGHT, max_heights[cap->id]);
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> -    pixm->field = V4L2_FIELD_NONE;
> > >>>>>>>>>>>> -    pixm->colorspace = V4L2_COLORSPACE_DEFAULT;
> > >>>>>>>>>>>> -    pixm->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> > >>>>>>>>>>>> -
> > >>>>>>>>>>>>        info = rkisp1_fill_pixfmt(pixm, cap->id);
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> -    /* can not change quantization when stream-on */
> > >>>>>>>>>>>> -    if (other_cap->is_streaming)
> > >>>>>>>>>>>> -            pixm->quantization = other_cap->pix.fmt.quantization;
> > >>>>>>>>>>>> -    /* output full range by default, take effect in params */
> > >>>>>>>>>>>> -    else if (!pixm->quantization ||
> > >>>>>>>>>>>> -             pixm->quantization > V4L2_QUANTIZATION_LIM_RANGE)
> > >>>>>>>>>>>> -            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>        if (fmt_cfg)
> > >>>>>>>>>>>>                *fmt_cfg = fmt;
> > >>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-common.h b/drivers/staging/media/rkisp1/rkisp1-common.h
> > >>>>>>>>>>>> index 2d7b7e078636..7a5576fa14c9 100644
> > >>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-common.h
> > >>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-common.h
> > >>>>>>>>>>>> @@ -300,6 +300,8 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
> > >>>>>>>>>>>>                        struct v4l2_device *v4l2_dev);
> > >>>>>>>>>>>>     void rkisp1_isp_unregister(struct rkisp1_device *rkisp1);
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
> > >>>>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt);
> > >>>>>>>>>>>>     const struct rkisp1_isp_mbus_info *rkisp1_isp_mbus_info_get(u32 mbus_code);
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>     irqreturn_t rkisp1_isp_isr(struct rkisp1_device *rkisp1);
> > >>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-isp.c b/drivers/staging/media/rkisp1/rkisp1-isp.c
> > >>>>>>>>>>>> index dee8e96f3900..6fdf5ed0b6b1 100644
> > >>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-isp.c
> > >>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-isp.c
> > >>>>>>>>>>>> @@ -613,6 +613,10 @@ static int rkisp1_isp_enum_mbus_code(struct v4l2_subdev *sd,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>                if (code->index == pos - 1) {
> > >>>>>>>>>>>>                        code->code = fmt->mbus_code;
> > >>>>>>>>>>>> +                    if (fmt->pixel_enc == V4L2_PIXEL_ENC_YUV &&
> > >>>>>>>>>>>> +                        dir == RKISP1_DIR_SRC)
> > >>>>>>>>>>>> +                            code->flags =
> > >>>>>>>>>>>> +                                    V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION;
> > >>>>>>>>>>>>                        return 0;
> > >>>>>>>>>>>>                }
> > >>>>>>>>>>>>        }
> > >>>>>>>>>>>> @@ -639,12 +643,21 @@ static int rkisp1_isp_init_config(struct v4l2_subdev *sd,
> > >>>>>>>>>>>>        sink_crop->height = RKISP1_DEFAULT_HEIGHT;
> > >>>>>>>>>>>>        sink_crop->left = 0;
> > >>>>>>>>>>>>        sink_crop->top = 0;
> > >>>>>>>>>>>> +    sink_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> > >>>>>>>>>>>> +    sink_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(sink_fmt->colorspace);
> > >>>>>>>>>>>> +    sink_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(sink_fmt->colorspace);
> > >>>>>>>>>>>> +    sink_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> > >>>>>>>>>>>> +
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>        src_fmt = v4l2_subdev_get_try_format(sd, cfg,
> > >>>>>>>>>>>>                                             RKISP1_ISP_PAD_SOURCE_VIDEO);
> > >>>>>>>>>>>>        *src_fmt = *sink_fmt;
> > >>>>>>>>>>>>        src_fmt->code = RKISP1_DEF_SRC_PAD_FMT;
> > >>>>>>>>>>>> -    src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> > >>>>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> > >>>>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
> > >>>>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
> > >>>>>>>>>>>> +    src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
> > >>>>>>>>>>>> +
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>        src_crop = v4l2_subdev_get_try_crop(sd, cfg,
> > >>>>>>>>>>>>                                            RKISP1_ISP_PAD_SOURCE_VIDEO);
> > >>>>>>>>>>>> @@ -687,10 +700,17 @@ static void rkisp1_isp_set_src_fmt(struct rkisp1_isp *isp,
> > >>>>>>>>>>>>                isp->src_fmt = mbus_info;
> > >>>>>>>>>>>>        src_fmt->width  = src_crop->width;
> > >>>>>>>>>>>>        src_fmt->height = src_crop->height;
> > >>>>>>>>>>>> -    src_fmt->quantization = format->quantization;
> > >>>>>>>>>>>> -    /* full range by default */
> > >>>>>>>>>>>> -    if (!src_fmt->quantization)
> > >>>>>>>>>>>> +
> > >>>>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> > >>>>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
> > >>>>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
> > >>>>>>>>>>>> +
> > >>>>>>>>>>>> +    if (mbus_info->pixel_enc == V4L2_PIXEL_ENC_BAYER)
> > >>>>>>>>>>>>                src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> > >>>>>>>>>>>> +    else if (format->quantization == V4L2_QUANTIZATION_DEFAULT)
> > >>>>>>>>>>>> +            src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
> > >>>>>>>>>>>> +    else
> > >>>>>>>>>>>> +            src_fmt->quantization = format->quantization;
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>        *format = *src_fmt;
> > >>>>>>>>>>>>     }
> > >>>>>>>>>>>> @@ -1068,6 +1088,17 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
> > >>>>>>>>>>>>        return ret;
> > >>>>>>>>>>>>     }
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
> > >>>>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt)
> > >>>>>>>>>>>> +{
> > >>>>>>>>>>>> +    struct rkisp1_isp *isp = &rkisp1->isp;
> > >>>>>>>>>>>> +
> > >>>>>>>>>>>> +    sd_fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
> > >>>>>>>>>>>> +    sd_fmt->pad = RKISP1_ISP_PAD_SOURCE_VIDEO;
> > >>>>>>>>>>>> +
> > >>>>>>>>>>>> +    return v4l2_subdev_call(&isp->sd, pad, get_fmt, NULL, sd_fmt);
> > >>>>>>>>>>>
> > >>>>>>>>>>> Do we need to get through the external API to access data that is
> > >>>>>>>>>>> driver-internal anyway?
> > >>>>>>>>>>>
> > >>>>>>>>>>>> +}
> > >>>>>>>>>>>> +
> > >>>>>>>>>>>>     void rkisp1_isp_unregister(struct rkisp1_device *rkisp1)
> > >>>>>>>>>>>>     {
> > >>>>>>>>>>>>        struct v4l2_subdev *sd = &rkisp1->isp.sd;
> > >>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-resizer.c b/drivers/staging/media/rkisp1/rkisp1-resizer.c
> > >>>>>>>>>>>> index 7b6b7ddd4169..8705b133de68 100644
> > >>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-resizer.c
> > >>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-resizer.c
> > >>>>>>>>>>>> @@ -525,6 +525,7 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
> > >>>>>>>>>>>>        const struct rkisp1_isp_mbus_info *mbus_info;
> > >>>>>>>>>>>>        struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
> > >>>>>>>>>>>>        struct v4l2_rect *sink_crop;
> > >>>>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>        sink_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SINK, which);
> > >>>>>>>>>>>>        src_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SRC, which);
> > >>>>>>>>>>>> @@ -539,8 +540,20 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
> > >>>>>>>>>>>>        if (which == V4L2_SUBDEV_FORMAT_ACTIVE)
> > >>>>>>>>>>>>                rsz->pixel_enc = mbus_info->pixel_enc;
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> +    rkisp1_get_isp_src_fmt(rsz->rkisp1, &isp_sd_fmt);
> > >>>>>>>>>>>> +
> > >>>>>>>>>>>
> > >>>>>>>>>>> Is this necessary? My understanding was that in the subdev model, it was
> > >>>>>>>>>>> the userspace responsibility to propagate any configuration changes through
> > >>>>>>>>>>> the graph.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Also, doing this only here wouldn't fully maintain the
> > >>>>>>>>>>> consistency of the state. For example, if one sets the ISP subdev format
> > >>>>>>>>>>> first, then the resizer subdev and then the ISP subdev again, wouldn't the
> > >>>>>>>>>>> resizer subdev end up with a wrong format?
> > >>>>>>>>>>
> > >>>>>>>>>> yes, this is indeed a bug, I am preparing v4 now.
> > >>>>>>>>>> What I thought to do is adding quantization conversion
> > >>>>>>>>>> support also on ther resizer and capture entities.
> > >>>>>>>>>> Then in the 'link_validation' callbacks, there
> > >>>>>>>>>> is a validation that the quantization fields matches.
> > >>>>>>>>>
> > >>>>>>>>> My understanding is that, if we have the following topology
> > >>>>>>>>>
> > >>>>>>>>> [ ISP ] -> [ Resizer ] -> [ Video node ]
> > >>>>>>>>>
> > >>>>>>>>> then the ISP source pad would have the csc capability, while resizer
> > >>>>>>>>> and video node would just accept whatever is configured on their sink
> > >>>>>>>>> pads (no need for csc capability for that) and propagate that to their
> > >>>>>>>>> outputs, i.e. resizer source pad and video node CAPTURE format.
> > >>>>>>>>>
> > >>>>>>>>> Is this what you were going to implement?
> > >>>>>>> Hi, I sent a v4 where the CSC capability is set on the reiszer and capture as well.
> > >>>>>>> I can send a v5 implementing it the way you suggest. Currently the doc says that the colorspace fields
> > >>>>>>> must be set by the driver for capture streams. This implies that userspace can set it
> > >>>>>>> only if the CSC is supported.
> > >>>>>>
> > >>>>>> Why would the userspace have to set it for the capture stream on the
> > >>>>>> resizer and video nodes? Couldn't the userspace set it to DEFAULT and
> > >>>>>> then the driver override to whatever it received on the corresponding
> > >>>>>> sink?
> > >>>
> > >>> According to the documentation [1]:
> > >>>
> > >>> "the default quantization encoding as defined by the colorspace"
> > >>>
> > >>> Then, for instance, the default for V4L2_COLORSPACE_SRGB is limited range [2].
> > >>>
> > >>> Unless we change the definition of DEFAULT to let it accept whatever it receives on the corresponding sink.
> > >>>
> > >>> Then the way I understand is:
> > >>>
> > >>> [sensor]->[isp]->[resizer]->[capture]
> > >>>
> > >>> Userspace should set quantization on:
> > >>> * sensor source pad
> > >>> * isp sink pad
> > >>> * resizer sink pad
> > >>> * capture node
> > >>>
> > >>> The remaining source pads would get values propagated from their sink pads.
> > >>>
> > >>> [1] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-defs.html#c.v4l2_quantization
> > >>> [2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-details.html#col-srgb
> > >>
> > >> After discussing this offline, we concluded that:
> > >> - only the source pad of the ISP entity should expose the CSC capability,
> > >> - the CAPTURE video node should expose only the formats/color spaces
> > >> compatible with the current setting of the resizer source,
> > >> - consistency between the resizer source format and CAPTURE format
> > >> should be implicitly maintained by the driver.
> > >
> > > Do you mean it should be validated at streamon time ?
> >
> > Nope. Any V4L2 state must be kept valid all the time, as mandated by
> > the general V4L2 principles. Wrong configuration of the video node
> > must not be a reason for STREAMON failing.
>
> That's a wrong interpretation of V4L2. On a video-node-centric device (a
> device that doesn't use the subdev userspace API for configuration) this
> is correct. In a MC-centric device, the video node and its connected
> subdev are configured separately.

I don't see this documented anywhere in the specification, maybe
except a mention of EPIPE or ENOLINK being returned from STREAMON, but
they explicitly refer to pipeline configuration and not a conflict
between V4L2 and MC state.

The behavior as you're suggesting would be at least confusing for the
userspace, because it breaks one of the core principles of V4L2 and
also there is even no facility to allow the userspace handle this kind
of behavior, e.g. how to know which memory formats are supported for
given mbus formats of the related sink.

Best regards,
Tomasz

>
> > > In MC-based
> > > drivers, entities are isolated from each other from a configuration
> > > point of view, and the validity of the configuration across the whole
> > > pipeline is checked at streamon time. I don't think CSC parameters
> > > should be treated differently than formats in this regard.
> >
> > Indeed, the ISP entity and the resizer entity are isolated from each
> > other and they need to be validated separately. However, the resizer
> > entity is not isolated from the video node, as there is no
> > configuration exposed for the video node in the MC API. If MC exposed
> > an explicit format control for the video interface sink pad, then the
> > resizer would have to be validated separately indeed, but still that
> > video interface sink pad would be tied to the rest of the
> > configuration of the video node (e.g. the CAPTURE format).
> >
> > >> Below is the rationale for that:
> > >> - In the MC model, the video node represents the DMA alone (and thus
> > >> the reason to have the resizer entities at all) and since the
> > >> colorspace is not a property of the DMA, it doesn't belong there.
> > >> - Even if we put that aside, both MP and SP can only output the same
> > >> colorspace, which would create a dependency between the configuration
> > >> of the two video nodes, which is not something well defined by V4L2.
> > >> - A video interface in the MC API doesn't expose control over its
> > >> input format, which means that the relation between the other side of
> > >> the link (resizer source) and the video node is implicit and needs to
> > >> be handled internally in the driver.
> > >> - V4L2, as opposed to MC, requires that the current state is always
> > >> consistent and valid. In this case it means that an ioctl on the video
> > >> node must not be able to alter the state in a way that would make it
> > >> inconsistent with the state of the matching resizer source.
>
> --
> Regards,
>
> Laurent Pinchart

Re: [RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad

[Laurent Pinchart]

Hi Tomasz,

On Sun, Jun 28, 2020 at 11:33:30PM +0200, Tomasz Figa wrote:
> On Sun, Jun 28, 2020 at 10:27 PM Laurent Pinchart wrote:
> > On Sun, Jun 28, 2020 at 04:59:34PM +0200, Tomasz Figa wrote:
> >> On Sun, Jun 28, 2020 at 3:15 PM Laurent Pinchart wrote:
> >>> On Sun, Jun 28, 2020 at 03:10:41PM +0200, Tomasz Figa wrote:
> >>>> On Fri, Jun 26, 2020 at 2:15 AM Helen Koike <helen.koike@collabora.com> wrote:
> >>>>> On 6/18/20 4:25 PM, Tomasz Figa wrote:
> >>>>>> On Thu, Jun 18, 2020 at 08:54:34PM +0200, Dafna Hirschfeld wrote:
> >>>>>>> On 18.06.20 20:49, Tomasz Figa wrote:
> >>>>>>>> On Thu, Jun 18, 2020 at 08:26:33PM +0200, Dafna Hirschfeld wrote:
> >>>>>>>>> On 18.06.20 19:27, Tomasz Figa wrote:
> >>>>>>>>>> On Wed, Jun 10, 2020 at 2:08 PM Tomasz Figa wrote:
> >>>>>>>>>>> On Thu, Jun 4, 2020 at 9:11 PM Dafna Hirschfeld wrote:
> >>>>>>>>>>>> On 04.06.20 19:54, Tomasz Figa wrote:
> >>>>>>>>>>>>> On Thu, Apr 16, 2020 at 04:56:05PM +0200, Dafna Hirschfeld wrote:
> >>>>>>>>>>>>>> The isp entity has a hardware support to force full range quantization
> >>>>>>>>>>>>>> for YUV formats. Use the new API to indicate userspace that
> >>>>>>>>>>>>>> quantization conversion is supported by adding the flag
> >>>>>>>>>>>>>> V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION during media code enumeration.
> >>>>>>>>>>>>>> Then uppon s_fmt on the video source pad, we assign the
> >>>>>>>>>>>>>> quantization from userspace for YUV formats.
> >>>>>>>>>>>>>> Also in the capture and resizer entities we retrieve the colorspace
> >>>>>>>>>>>>>> from the isp entity.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
> >>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>     drivers/staging/media/rkisp1/rkisp1-capture.c | 28 ++++++-------
> >>>>>>>>>>>>>>     drivers/staging/media/rkisp1/rkisp1-common.h  |  2 +
> >>>>>>>>>>>>>>     drivers/staging/media/rkisp1/rkisp1-isp.c     | 39 +++++++++++++++++--
> >>>>>>>>>>>>>>     drivers/staging/media/rkisp1/rkisp1-resizer.c | 13 +++++++
> >>>>>>>>>>>>>>     4 files changed, 65 insertions(+), 17 deletions(-)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c
> >>>>>>>>>>>>>> index fbf62399fe3d..aca0f93bc772 100644
> >>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-capture.c
> >>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-capture.c
> >>>>>>>>>>>>>> @@ -1066,14 +1066,13 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
> >>>>>>>>>>>>>>                           const struct v4l2_format_info **fmt_info)
> >>>>>>>>>>>>>>     {
> >>>>>>>>>>>>>>        const struct rkisp1_capture_config *config = cap->config;
> >>>>>>>>>>>>>> -    struct rkisp1_capture *other_cap =
> >>>>>>>>>>>>>> -                    &cap->rkisp1->capture_devs[cap->id ^ 1];
> >>>>>>>>>>>>>>        const struct rkisp1_capture_fmt_cfg *fmt;
> >>>>>>>>>>>>>>        const struct v4l2_format_info *info;
> >>>>>>>>>>>>>>        const unsigned int max_widths[] = { RKISP1_RSZ_MP_SRC_MAX_WIDTH,
> >>>>>>>>>>>>>>                                            RKISP1_RSZ_SP_SRC_MAX_WIDTH };
> >>>>>>>>>>>>>>        const unsigned int max_heights[] = { RKISP1_RSZ_MP_SRC_MAX_HEIGHT,
> >>>>>>>>>>>>>>                                             RKISP1_RSZ_SP_SRC_MAX_HEIGHT};
> >>>>>>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>        fmt = rkisp1_find_fmt_cfg(cap, pixm->pixelformat);
> >>>>>>>>>>>>>>        if (!fmt) {
> >>>>>>>>>>>>>> @@ -1081,24 +1080,27 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
> >>>>>>>>>>>>>>                pixm->pixelformat = fmt->fourcc;
> >>>>>>>>>>>>>>        }
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> +    rkisp1_get_isp_src_fmt(cap->rkisp1, &isp_sd_fmt);
> >>>>>>>>>>>>>> +    pixm->field = isp_sd_fmt.format.field;
> >>>>>>>>>>>>>> +    pixm->colorspace = isp_sd_fmt.format.colorspace;
> >>>>>>>>>>>>>> +    pixm->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
> >>>>>>>>>>>>>> +    pixm->xfer_func = isp_sd_fmt.format.xfer_func;
> >>>>>>>>>>>>>> +
> >>>>>>>>>>>>>> +    /*
> >>>>>>>>>>>>>> +     * isp has a feature to set full range quantization for yuv formats.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> How about "select between limited and full range for YUV formats"?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> +     * so we need to get the format from the isp.
> >>>>>>>>>>>>>> +     */
> >>>>>>>>>>>>>> +    pixm->quantization = isp_sd_fmt.format.quantization;
> >>>>>>>>>>>>>> +    if (!v4l2_is_format_yuv(cap->pix.info))
> >>>>>>>>>>>>>> +            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>        pixm->width = clamp_t(u32, pixm->width,
> >>>>>>>>>>>>>>                              RKISP1_RSZ_SRC_MIN_WIDTH, max_widths[cap->id]);
> >>>>>>>>>>>>>>        pixm->height = clamp_t(u32, pixm->height,
> >>>>>>>>>>>>>>                               RKISP1_RSZ_SRC_MIN_HEIGHT, max_heights[cap->id]);
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -    pixm->field = V4L2_FIELD_NONE;
> >>>>>>>>>>>>>> -    pixm->colorspace = V4L2_COLORSPACE_DEFAULT;
> >>>>>>>>>>>>>> -    pixm->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> >>>>>>>>>>>>>> -
> >>>>>>>>>>>>>>        info = rkisp1_fill_pixfmt(pixm, cap->id);
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -    /* can not change quantization when stream-on */
> >>>>>>>>>>>>>> -    if (other_cap->is_streaming)
> >>>>>>>>>>>>>> -            pixm->quantization = other_cap->pix.fmt.quantization;
> >>>>>>>>>>>>>> -    /* output full range by default, take effect in params */
> >>>>>>>>>>>>>> -    else if (!pixm->quantization ||
> >>>>>>>>>>>>>> -             pixm->quantization > V4L2_QUANTIZATION_LIM_RANGE)
> >>>>>>>>>>>>>> -            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>        if (fmt_cfg)
> >>>>>>>>>>>>>>                *fmt_cfg = fmt;
> >>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-common.h b/drivers/staging/media/rkisp1/rkisp1-common.h
> >>>>>>>>>>>>>> index 2d7b7e078636..7a5576fa14c9 100644
> >>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-common.h
> >>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-common.h
> >>>>>>>>>>>>>> @@ -300,6 +300,8 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
> >>>>>>>>>>>>>>                        struct v4l2_device *v4l2_dev);
> >>>>>>>>>>>>>>     void rkisp1_isp_unregister(struct rkisp1_device *rkisp1);
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
> >>>>>>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt);
> >>>>>>>>>>>>>>     const struct rkisp1_isp_mbus_info *rkisp1_isp_mbus_info_get(u32 mbus_code);
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>     irqreturn_t rkisp1_isp_isr(struct rkisp1_device *rkisp1);
> >>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-isp.c b/drivers/staging/media/rkisp1/rkisp1-isp.c
> >>>>>>>>>>>>>> index dee8e96f3900..6fdf5ed0b6b1 100644
> >>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-isp.c
> >>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-isp.c
> >>>>>>>>>>>>>> @@ -613,6 +613,10 @@ static int rkisp1_isp_enum_mbus_code(struct v4l2_subdev *sd,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>                if (code->index == pos - 1) {
> >>>>>>>>>>>>>>                        code->code = fmt->mbus_code;
> >>>>>>>>>>>>>> +                    if (fmt->pixel_enc == V4L2_PIXEL_ENC_YUV &&
> >>>>>>>>>>>>>> +                        dir == RKISP1_DIR_SRC)
> >>>>>>>>>>>>>> +                            code->flags =
> >>>>>>>>>>>>>> +                                    V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION;
> >>>>>>>>>>>>>>                        return 0;
> >>>>>>>>>>>>>>                }
> >>>>>>>>>>>>>>        }
> >>>>>>>>>>>>>> @@ -639,12 +643,21 @@ static int rkisp1_isp_init_config(struct v4l2_subdev *sd,
> >>>>>>>>>>>>>>        sink_crop->height = RKISP1_DEFAULT_HEIGHT;
> >>>>>>>>>>>>>>        sink_crop->left = 0;
> >>>>>>>>>>>>>>        sink_crop->top = 0;
> >>>>>>>>>>>>>> +    sink_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >>>>>>>>>>>>>> +    sink_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(sink_fmt->colorspace);
> >>>>>>>>>>>>>> +    sink_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(sink_fmt->colorspace);
> >>>>>>>>>>>>>> +    sink_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>        src_fmt = v4l2_subdev_get_try_format(sd, cfg,
> >>>>>>>>>>>>>>                                             RKISP1_ISP_PAD_SOURCE_VIDEO);
> >>>>>>>>>>>>>>        *src_fmt = *sink_fmt;
> >>>>>>>>>>>>>>        src_fmt->code = RKISP1_DEF_SRC_PAD_FMT;
> >>>>>>>>>>>>>> -    src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >>>>>>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>>>>>>> +    src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
> >>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>        src_crop = v4l2_subdev_get_try_crop(sd, cfg,
> >>>>>>>>>>>>>>                                            RKISP1_ISP_PAD_SOURCE_VIDEO);
> >>>>>>>>>>>>>> @@ -687,10 +700,17 @@ static void rkisp1_isp_set_src_fmt(struct rkisp1_isp *isp,
> >>>>>>>>>>>>>>                isp->src_fmt = mbus_info;
> >>>>>>>>>>>>>>        src_fmt->width  = src_crop->width;
> >>>>>>>>>>>>>>        src_fmt->height = src_crop->height;
> >>>>>>>>>>>>>> -    src_fmt->quantization = format->quantization;
> >>>>>>>>>>>>>> -    /* full range by default */
> >>>>>>>>>>>>>> -    if (!src_fmt->quantization)
> >>>>>>>>>>>>>> +
> >>>>>>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >>>>>>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>>>>>>> +
> >>>>>>>>>>>>>> +    if (mbus_info->pixel_enc == V4L2_PIXEL_ENC_BAYER)
> >>>>>>>>>>>>>>                src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>>> +    else if (format->quantization == V4L2_QUANTIZATION_DEFAULT)
> >>>>>>>>>>>>>> +            src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
> >>>>>>>>>>>>>> +    else
> >>>>>>>>>>>>>> +            src_fmt->quantization = format->quantization;
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>        *format = *src_fmt;
> >>>>>>>>>>>>>>     }
> >>>>>>>>>>>>>> @@ -1068,6 +1088,17 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
> >>>>>>>>>>>>>>        return ret;
> >>>>>>>>>>>>>>     }
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
> >>>>>>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt)
> >>>>>>>>>>>>>> +{
> >>>>>>>>>>>>>> +    struct rkisp1_isp *isp = &rkisp1->isp;
> >>>>>>>>>>>>>> +
> >>>>>>>>>>>>>> +    sd_fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
> >>>>>>>>>>>>>> +    sd_fmt->pad = RKISP1_ISP_PAD_SOURCE_VIDEO;
> >>>>>>>>>>>>>> +
> >>>>>>>>>>>>>> +    return v4l2_subdev_call(&isp->sd, pad, get_fmt, NULL, sd_fmt);
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Do we need to get through the external API to access data that is
> >>>>>>>>>>>>> driver-internal anyway?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> +}
> >>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>     void rkisp1_isp_unregister(struct rkisp1_device *rkisp1)
> >>>>>>>>>>>>>>     {
> >>>>>>>>>>>>>>        struct v4l2_subdev *sd = &rkisp1->isp.sd;
> >>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-resizer.c b/drivers/staging/media/rkisp1/rkisp1-resizer.c
> >>>>>>>>>>>>>> index 7b6b7ddd4169..8705b133de68 100644
> >>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-resizer.c
> >>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-resizer.c
> >>>>>>>>>>>>>> @@ -525,6 +525,7 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
> >>>>>>>>>>>>>>        const struct rkisp1_isp_mbus_info *mbus_info;
> >>>>>>>>>>>>>>        struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
> >>>>>>>>>>>>>>        struct v4l2_rect *sink_crop;
> >>>>>>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>        sink_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SINK, which);
> >>>>>>>>>>>>>>        src_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SRC, which);
> >>>>>>>>>>>>>> @@ -539,8 +540,20 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
> >>>>>>>>>>>>>>        if (which == V4L2_SUBDEV_FORMAT_ACTIVE)
> >>>>>>>>>>>>>>                rsz->pixel_enc = mbus_info->pixel_enc;
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> +    rkisp1_get_isp_src_fmt(rsz->rkisp1, &isp_sd_fmt);
> >>>>>>>>>>>>>> +
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Is this necessary? My understanding was that in the subdev model, it was
> >>>>>>>>>>>>> the userspace responsibility to propagate any configuration changes through
> >>>>>>>>>>>>> the graph.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Also, doing this only here wouldn't fully maintain the
> >>>>>>>>>>>>> consistency of the state. For example, if one sets the ISP subdev format
> >>>>>>>>>>>>> first, then the resizer subdev and then the ISP subdev again, wouldn't the
> >>>>>>>>>>>>> resizer subdev end up with a wrong format?
> >>>>>>>>>>>>
> >>>>>>>>>>>> yes, this is indeed a bug, I am preparing v4 now.
> >>>>>>>>>>>> What I thought to do is adding quantization conversion
> >>>>>>>>>>>> support also on ther resizer and capture entities.
> >>>>>>>>>>>> Then in the 'link_validation' callbacks, there
> >>>>>>>>>>>> is a validation that the quantization fields matches.
> >>>>>>>>>>>
> >>>>>>>>>>> My understanding is that, if we have the following topology
> >>>>>>>>>>>
> >>>>>>>>>>> [ ISP ] -> [ Resizer ] -> [ Video node ]
> >>>>>>>>>>>
> >>>>>>>>>>> then the ISP source pad would have the csc capability, while resizer
> >>>>>>>>>>> and video node would just accept whatever is configured on their sink
> >>>>>>>>>>> pads (no need for csc capability for that) and propagate that to their
> >>>>>>>>>>> outputs, i.e. resizer source pad and video node CAPTURE format.
> >>>>>>>>>>>
> >>>>>>>>>>> Is this what you were going to implement?
> >>>>>>>>> Hi, I sent a v4 where the CSC capability is set on the reiszer and capture as well.
> >>>>>>>>> I can send a v5 implementing it the way you suggest. Currently the doc says that the colorspace fields
> >>>>>>>>> must be set by the driver for capture streams. This implies that userspace can set it
> >>>>>>>>> only if the CSC is supported.
> >>>>>>>>
> >>>>>>>> Why would the userspace have to set it for the capture stream on the
> >>>>>>>> resizer and video nodes? Couldn't the userspace set it to DEFAULT and
> >>>>>>>> then the driver override to whatever it received on the corresponding
> >>>>>>>> sink?
> >>>>>
> >>>>> According to the documentation [1]:
> >>>>>
> >>>>> "the default quantization encoding as defined by the colorspace"
> >>>>>
> >>>>> Then, for instance, the default for V4L2_COLORSPACE_SRGB is limited range [2].
> >>>>>
> >>>>> Unless we change the definition of DEFAULT to let it accept whatever it receives on the corresponding sink.
> >>>>>
> >>>>> Then the way I understand is:
> >>>>>
> >>>>> [sensor]->[isp]->[resizer]->[capture]
> >>>>>
> >>>>> Userspace should set quantization on:
> >>>>> * sensor source pad
> >>>>> * isp sink pad
> >>>>> * resizer sink pad
> >>>>> * capture node
> >>>>>
> >>>>> The remaining source pads would get values propagated from their sink pads.
> >>>>>
> >>>>> [1] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-defs.html#c.v4l2_quantization
> >>>>> [2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-details.html#col-srgb
> >>>>
> >>>> After discussing this offline, we concluded that:
> >>>> - only the source pad of the ISP entity should expose the CSC capability,
> >>>> - the CAPTURE video node should expose only the formats/color spaces
> >>>> compatible with the current setting of the resizer source,
> >>>> - consistency between the resizer source format and CAPTURE format
> >>>> should be implicitly maintained by the driver.
> >>>
> >>> Do you mean it should be validated at streamon time ?
> >>
> >> Nope. Any V4L2 state must be kept valid all the time, as mandated by
> >> the general V4L2 principles. Wrong configuration of the video node
> >> must not be a reason for STREAMON failing.
> >
> > That's a wrong interpretation of V4L2. On a video-node-centric device (a
> > device that doesn't use the subdev userspace API for configuration) this
> > is correct. In a MC-centric device, the video node and its connected
> > subdev are configured separately.
> 
> I don't see this documented anywhere in the specification, maybe
> except a mention of EPIPE or ENOLINK being returned from STREAMON, but
> they explicitly refer to pipeline configuration and not a conflict
> between V4L2 and MC state.

We're suffering from a lack of documentation in this area :-( Niklas has
authored

commit f645e6256bd1b12523b759fcc610861fb21c24c7
Author: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Date:   Tue Apr 21 15:57:38 2020 +0200

    media: v4l2-dev/ioctl: Add V4L2_CAP_IO_MC

which is a first step, now we need to document the behaviour of all
ioctls properly when the MC and !MC cases differ.

> The behavior as you're suggesting would be at least confusing for the
> userspace, because it breaks one of the core principles of V4L2 and
> also there is even no facility to allow the userspace handle this kind
> of behavior, e.g. how to know which memory formats are supported for
> given mbus formats of the related sink.

commit e5b6b07a1b45dd9d19bec1fa1d60750b0fcf2fb0
Author: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Date:   Fri Apr 24 15:43:31 2020 +0200

    media: v4l2: Extend VIDIOC_ENUM_FMT to support MC-centric devices

Been missing for a long time... I realized that when I started using the
API in a real userspace implementation instead of just writing test
tools or very device-specific code that could hardcode all this
knowledge.

> >>> In MC-based
> >>> drivers, entities are isolated from each other from a configuration
> >>> point of view, and the validity of the configuration across the whole
> >>> pipeline is checked at streamon time. I don't think CSC parameters
> >>> should be treated differently than formats in this regard.
> >>
> >> Indeed, the ISP entity and the resizer entity are isolated from each
> >> other and they need to be validated separately. However, the resizer
> >> entity is not isolated from the video node, as there is no
> >> configuration exposed for the video node in the MC API. If MC exposed
> >> an explicit format control for the video interface sink pad, then the
> >> resizer would have to be validated separately indeed, but still that
> >> video interface sink pad would be tied to the rest of the
> >> configuration of the video node (e.g. the CAPTURE format).
> >>
> >>>> Below is the rationale for that:
> >>>> - In the MC model, the video node represents the DMA alone (and thus
> >>>> the reason to have the resizer entities at all) and since the
> >>>> colorspace is not a property of the DMA, it doesn't belong there.
> >>>> - Even if we put that aside, both MP and SP can only output the same
> >>>> colorspace, which would create a dependency between the configuration
> >>>> of the two video nodes, which is not something well defined by V4L2.
> >>>> - A video interface in the MC API doesn't expose control over its
> >>>> input format, which means that the relation between the other side of
> >>>> the link (resizer source) and the video node is implicit and needs to
> >>>> be handled internally in the driver.
> >>>> - V4L2, as opposed to MC, requires that the current state is always
> >>>> consistent and valid. In this case it means that an ioctl on the video
> >>>> node must not be able to alter the state in a way that would make it
> >>>> inconsistent with the state of the matching resizer source.

-- 
Regards,

Laurent Pinchart

Re: [RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad

[Dafna Hirschfeld]

On 28.06.20 23:37, Laurent Pinchart wrote:
> Hi Tomasz,
> 
> On Sun, Jun 28, 2020 at 11:33:30PM +0200, Tomasz Figa wrote:
>> On Sun, Jun 28, 2020 at 10:27 PM Laurent Pinchart wrote:
>>> On Sun, Jun 28, 2020 at 04:59:34PM +0200, Tomasz Figa wrote:
>>>> On Sun, Jun 28, 2020 at 3:15 PM Laurent Pinchart wrote:
>>>>> On Sun, Jun 28, 2020 at 03:10:41PM +0200, Tomasz Figa wrote:
>>>>>> On Fri, Jun 26, 2020 at 2:15 AM Helen Koike <helen.koike@collabora.com> wrote:
>>>>>>> On 6/18/20 4:25 PM, Tomasz Figa wrote:
>>>>>>>> On Thu, Jun 18, 2020 at 08:54:34PM +0200, Dafna Hirschfeld wrote:
>>>>>>>>> On 18.06.20 20:49, Tomasz Figa wrote:
>>>>>>>>>> On Thu, Jun 18, 2020 at 08:26:33PM +0200, Dafna Hirschfeld wrote:
>>>>>>>>>>> On 18.06.20 19:27, Tomasz Figa wrote:
>>>>>>>>>>>> On Wed, Jun 10, 2020 at 2:08 PM Tomasz Figa wrote:
>>>>>>>>>>>>> On Thu, Jun 4, 2020 at 9:11 PM Dafna Hirschfeld wrote:
>>>>>>>>>>>>>> On 04.06.20 19:54, Tomasz Figa wrote:
>>>>>>>>>>>>>>> On Thu, Apr 16, 2020 at 04:56:05PM +0200, Dafna Hirschfeld wrote:
>>>>>>>>>>>>>>>> The isp entity has a hardware support to force full range quantization
>>>>>>>>>>>>>>>> for YUV formats. Use the new API to indicate userspace that
>>>>>>>>>>>>>>>> quantization conversion is supported by adding the flag
>>>>>>>>>>>>>>>> V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION during media code enumeration.
>>>>>>>>>>>>>>>> Then uppon s_fmt on the video source pad, we assign the
>>>>>>>>>>>>>>>> quantization from userspace for YUV formats.
>>>>>>>>>>>>>>>> Also in the capture and resizer entities we retrieve the colorspace
>>>>>>>>>>>>>>>> from the isp entity.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>      drivers/staging/media/rkisp1/rkisp1-capture.c | 28 ++++++-------
>>>>>>>>>>>>>>>>      drivers/staging/media/rkisp1/rkisp1-common.h  |  2 +
>>>>>>>>>>>>>>>>      drivers/staging/media/rkisp1/rkisp1-isp.c     | 39 +++++++++++++++++--
>>>>>>>>>>>>>>>>      drivers/staging/media/rkisp1/rkisp1-resizer.c | 13 +++++++
>>>>>>>>>>>>>>>>      4 files changed, 65 insertions(+), 17 deletions(-)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>>>>>>>>>>>> index fbf62399fe3d..aca0f93bc772 100644
>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>>>>>>>>>>>> @@ -1066,14 +1066,13 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
>>>>>>>>>>>>>>>>                            const struct v4l2_format_info **fmt_info)
>>>>>>>>>>>>>>>>      {
>>>>>>>>>>>>>>>>         const struct rkisp1_capture_config *config = cap->config;
>>>>>>>>>>>>>>>> -    struct rkisp1_capture *other_cap =
>>>>>>>>>>>>>>>> -                    &cap->rkisp1->capture_devs[cap->id ^ 1];
>>>>>>>>>>>>>>>>         const struct rkisp1_capture_fmt_cfg *fmt;
>>>>>>>>>>>>>>>>         const struct v4l2_format_info *info;
>>>>>>>>>>>>>>>>         const unsigned int max_widths[] = { RKISP1_RSZ_MP_SRC_MAX_WIDTH,
>>>>>>>>>>>>>>>>                                             RKISP1_RSZ_SP_SRC_MAX_WIDTH };
>>>>>>>>>>>>>>>>         const unsigned int max_heights[] = { RKISP1_RSZ_MP_SRC_MAX_HEIGHT,
>>>>>>>>>>>>>>>>                                              RKISP1_RSZ_SP_SRC_MAX_HEIGHT};
>>>>>>>>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>         fmt = rkisp1_find_fmt_cfg(cap, pixm->pixelformat);
>>>>>>>>>>>>>>>>         if (!fmt) {
>>>>>>>>>>>>>>>> @@ -1081,24 +1080,27 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
>>>>>>>>>>>>>>>>                 pixm->pixelformat = fmt->fourcc;
>>>>>>>>>>>>>>>>         }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> +    rkisp1_get_isp_src_fmt(cap->rkisp1, &isp_sd_fmt);
>>>>>>>>>>>>>>>> +    pixm->field = isp_sd_fmt.format.field;
>>>>>>>>>>>>>>>> +    pixm->colorspace = isp_sd_fmt.format.colorspace;
>>>>>>>>>>>>>>>> +    pixm->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
>>>>>>>>>>>>>>>> +    pixm->xfer_func = isp_sd_fmt.format.xfer_func;
>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>> +    /*
>>>>>>>>>>>>>>>> +     * isp has a feature to set full range quantization for yuv formats.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> How about "select between limited and full range for YUV formats"?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> +     * so we need to get the format from the isp.
>>>>>>>>>>>>>>>> +     */
>>>>>>>>>>>>>>>> +    pixm->quantization = isp_sd_fmt.format.quantization;
>>>>>>>>>>>>>>>> +    if (!v4l2_is_format_yuv(cap->pix.info))
>>>>>>>>>>>>>>>> +            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>         pixm->width = clamp_t(u32, pixm->width,
>>>>>>>>>>>>>>>>                               RKISP1_RSZ_SRC_MIN_WIDTH, max_widths[cap->id]);
>>>>>>>>>>>>>>>>         pixm->height = clamp_t(u32, pixm->height,
>>>>>>>>>>>>>>>>                                RKISP1_RSZ_SRC_MIN_HEIGHT, max_heights[cap->id]);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -    pixm->field = V4L2_FIELD_NONE;
>>>>>>>>>>>>>>>> -    pixm->colorspace = V4L2_COLORSPACE_DEFAULT;
>>>>>>>>>>>>>>>> -    pixm->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>         info = rkisp1_fill_pixfmt(pixm, cap->id);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -    /* can not change quantization when stream-on */
>>>>>>>>>>>>>>>> -    if (other_cap->is_streaming)
>>>>>>>>>>>>>>>> -            pixm->quantization = other_cap->pix.fmt.quantization;
>>>>>>>>>>>>>>>> -    /* output full range by default, take effect in params */
>>>>>>>>>>>>>>>> -    else if (!pixm->quantization ||
>>>>>>>>>>>>>>>> -             pixm->quantization > V4L2_QUANTIZATION_LIM_RANGE)
>>>>>>>>>>>>>>>> -            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>         if (fmt_cfg)
>>>>>>>>>>>>>>>>                 *fmt_cfg = fmt;
>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-common.h b/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>>>>>>>>>>>> index 2d7b7e078636..7a5576fa14c9 100644
>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>>>>>>>>>>>> @@ -300,6 +300,8 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
>>>>>>>>>>>>>>>>                         struct v4l2_device *v4l2_dev);
>>>>>>>>>>>>>>>>      void rkisp1_isp_unregister(struct rkisp1_device *rkisp1);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
>>>>>>>>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt);
>>>>>>>>>>>>>>>>      const struct rkisp1_isp_mbus_info *rkisp1_isp_mbus_info_get(u32 mbus_code);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>      irqreturn_t rkisp1_isp_isr(struct rkisp1_device *rkisp1);
>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-isp.c b/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>>>>>>>>>>>> index dee8e96f3900..6fdf5ed0b6b1 100644
>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>>>>>>>>>>>> @@ -613,6 +613,10 @@ static int rkisp1_isp_enum_mbus_code(struct v4l2_subdev *sd,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                 if (code->index == pos - 1) {
>>>>>>>>>>>>>>>>                         code->code = fmt->mbus_code;
>>>>>>>>>>>>>>>> +                    if (fmt->pixel_enc == V4L2_PIXEL_ENC_YUV &&
>>>>>>>>>>>>>>>> +                        dir == RKISP1_DIR_SRC)
>>>>>>>>>>>>>>>> +                            code->flags =
>>>>>>>>>>>>>>>> +                                    V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION;
>>>>>>>>>>>>>>>>                         return 0;
>>>>>>>>>>>>>>>>                 }
>>>>>>>>>>>>>>>>         }
>>>>>>>>>>>>>>>> @@ -639,12 +643,21 @@ static int rkisp1_isp_init_config(struct v4l2_subdev *sd,
>>>>>>>>>>>>>>>>         sink_crop->height = RKISP1_DEFAULT_HEIGHT;
>>>>>>>>>>>>>>>>         sink_crop->left = 0;
>>>>>>>>>>>>>>>>         sink_crop->top = 0;
>>>>>>>>>>>>>>>> +    sink_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>>>>>>>>>>>> +    sink_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(sink_fmt->colorspace);
>>>>>>>>>>>>>>>> +    sink_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(sink_fmt->colorspace);
>>>>>>>>>>>>>>>> +    sink_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>         src_fmt = v4l2_subdev_get_try_format(sd, cfg,
>>>>>>>>>>>>>>>>                                              RKISP1_ISP_PAD_SOURCE_VIDEO);
>>>>>>>>>>>>>>>>         *src_fmt = *sink_fmt;
>>>>>>>>>>>>>>>>         src_fmt->code = RKISP1_DEF_SRC_PAD_FMT;
>>>>>>>>>>>>>>>> -    src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>>>>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
>>>>>>>>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
>>>>>>>>>>>>>>>> +    src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>         src_crop = v4l2_subdev_get_try_crop(sd, cfg,
>>>>>>>>>>>>>>>>                                             RKISP1_ISP_PAD_SOURCE_VIDEO);
>>>>>>>>>>>>>>>> @@ -687,10 +700,17 @@ static void rkisp1_isp_set_src_fmt(struct rkisp1_isp *isp,
>>>>>>>>>>>>>>>>                 isp->src_fmt = mbus_info;
>>>>>>>>>>>>>>>>         src_fmt->width  = src_crop->width;
>>>>>>>>>>>>>>>>         src_fmt->height = src_crop->height;
>>>>>>>>>>>>>>>> -    src_fmt->quantization = format->quantization;
>>>>>>>>>>>>>>>> -    /* full range by default */
>>>>>>>>>>>>>>>> -    if (!src_fmt->quantization)
>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>>>>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
>>>>>>>>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>> +    if (mbus_info->pixel_enc == V4L2_PIXEL_ENC_BAYER)
>>>>>>>>>>>>>>>>                 src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>>>> +    else if (format->quantization == V4L2_QUANTIZATION_DEFAULT)
>>>>>>>>>>>>>>>> +            src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
>>>>>>>>>>>>>>>> +    else
>>>>>>>>>>>>>>>> +            src_fmt->quantization = format->quantization;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>         *format = *src_fmt;
>>>>>>>>>>>>>>>>      }
>>>>>>>>>>>>>>>> @@ -1068,6 +1088,17 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
>>>>>>>>>>>>>>>>         return ret;
>>>>>>>>>>>>>>>>      }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
>>>>>>>>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt)
>>>>>>>>>>>>>>>> +{
>>>>>>>>>>>>>>>> +    struct rkisp1_isp *isp = &rkisp1->isp;
>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>> +    sd_fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
>>>>>>>>>>>>>>>> +    sd_fmt->pad = RKISP1_ISP_PAD_SOURCE_VIDEO;
>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>> +    return v4l2_subdev_call(&isp->sd, pad, get_fmt, NULL, sd_fmt);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Do we need to get through the external API to access data that is
>>>>>>>>>>>>>>> driver-internal anyway?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> +}
>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>      void rkisp1_isp_unregister(struct rkisp1_device *rkisp1)
>>>>>>>>>>>>>>>>      {
>>>>>>>>>>>>>>>>         struct v4l2_subdev *sd = &rkisp1->isp.sd;
>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-resizer.c b/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>>>>>>>>>>>> index 7b6b7ddd4169..8705b133de68 100644
>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>>>>>>>>>>>> @@ -525,6 +525,7 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
>>>>>>>>>>>>>>>>         const struct rkisp1_isp_mbus_info *mbus_info;
>>>>>>>>>>>>>>>>         struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
>>>>>>>>>>>>>>>>         struct v4l2_rect *sink_crop;
>>>>>>>>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>         sink_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SINK, which);
>>>>>>>>>>>>>>>>         src_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SRC, which);
>>>>>>>>>>>>>>>> @@ -539,8 +540,20 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
>>>>>>>>>>>>>>>>         if (which == V4L2_SUBDEV_FORMAT_ACTIVE)
>>>>>>>>>>>>>>>>                 rsz->pixel_enc = mbus_info->pixel_enc;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> +    rkisp1_get_isp_src_fmt(rsz->rkisp1, &isp_sd_fmt);
>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Is this necessary? My understanding was that in the subdev model, it was
>>>>>>>>>>>>>>> the userspace responsibility to propagate any configuration changes through
>>>>>>>>>>>>>>> the graph.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Also, doing this only here wouldn't fully maintain the
>>>>>>>>>>>>>>> consistency of the state. For example, if one sets the ISP subdev format
>>>>>>>>>>>>>>> first, then the resizer subdev and then the ISP subdev again, wouldn't the
>>>>>>>>>>>>>>> resizer subdev end up with a wrong format?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> yes, this is indeed a bug, I am preparing v4 now.
>>>>>>>>>>>>>> What I thought to do is adding quantization conversion
>>>>>>>>>>>>>> support also on ther resizer and capture entities.
>>>>>>>>>>>>>> Then in the 'link_validation' callbacks, there
>>>>>>>>>>>>>> is a validation that the quantization fields matches.
>>>>>>>>>>>>>
>>>>>>>>>>>>> My understanding is that, if we have the following topology
>>>>>>>>>>>>>
>>>>>>>>>>>>> [ ISP ] -> [ Resizer ] -> [ Video node ]
>>>>>>>>>>>>>
>>>>>>>>>>>>> then the ISP source pad would have the csc capability, while resizer
>>>>>>>>>>>>> and video node would just accept whatever is configured on their sink
>>>>>>>>>>>>> pads (no need for csc capability for that) and propagate that to their
>>>>>>>>>>>>> outputs, i.e. resizer source pad and video node CAPTURE format.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Is this what you were going to implement?
>>>>>>>>>>> Hi, I sent a v4 where the CSC capability is set on the reiszer and capture as well.
>>>>>>>>>>> I can send a v5 implementing it the way you suggest. Currently the doc says that the colorspace fields
>>>>>>>>>>> must be set by the driver for capture streams. This implies that userspace can set it
>>>>>>>>>>> only if the CSC is supported.
>>>>>>>>>>
>>>>>>>>>> Why would the userspace have to set it for the capture stream on the
>>>>>>>>>> resizer and video nodes? Couldn't the userspace set it to DEFAULT and
>>>>>>>>>> then the driver override to whatever it received on the corresponding
>>>>>>>>>> sink?
>>>>>>>
>>>>>>> According to the documentation [1]:
>>>>>>>
>>>>>>> "the default quantization encoding as defined by the colorspace"
>>>>>>>
>>>>>>> Then, for instance, the default for V4L2_COLORSPACE_SRGB is limited range [2].
>>>>>>>
>>>>>>> Unless we change the definition of DEFAULT to let it accept whatever it receives on the corresponding sink.
>>>>>>>
>>>>>>> Then the way I understand is:
>>>>>>>
>>>>>>> [sensor]->[isp]->[resizer]->[capture]
>>>>>>>
>>>>>>> Userspace should set quantization on:
>>>>>>> * sensor source pad
>>>>>>> * isp sink pad
>>>>>>> * resizer sink pad
>>>>>>> * capture node
>>>>>>>
>>>>>>> The remaining source pads would get values propagated from their sink pads.
>>>>>>>
>>>>>>> [1] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-defs.html#c.v4l2_quantization
>>>>>>> [2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-details.html#col-srgb
>>>>>>
>>>>>> After discussing this offline, we concluded that:
>>>>>> - only the source pad of the ISP entity should expose the CSC capability,
>>>>>> - the CAPTURE video node should expose only the formats/color spaces
>>>>>> compatible with the current setting of the resizer source,
>>>>>> - consistency between the resizer source format and CAPTURE format
>>>>>> should be implicitly maintained by the driver.
>>>>>
>>>>> Do you mean it should be validated at streamon time ?
>>>>
>>>> Nope. Any V4L2 state must be kept valid all the time, as mandated by
>>>> the general V4L2 principles. Wrong configuration of the video node
>>>> must not be a reason for STREAMON failing.
>>>
>>> That's a wrong interpretation of V4L2. On a video-node-centric device (a
>>> device that doesn't use the subdev userspace API for configuration) this
>>> is correct. In a MC-centric device, the video node and its connected
>>> subdev are configured separately.
>>
>> I don't see this documented anywhere in the specification, maybe
>> except a mention of EPIPE or ENOLINK being returned from STREAMON, but
>> they explicitly refer to pipeline configuration and not a conflict
>> between V4L2 and MC state.
> 
> We're suffering from a lack of documentation in this area :-( Niklas has
> authored
> 
> commit f645e6256bd1b12523b759fcc610861fb21c24c7
> Author: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
> Date:   Tue Apr 21 15:57:38 2020 +0200
> 
>      media: v4l2-dev/ioctl: Add V4L2_CAP_IO_MC
> 
> which is a first step, now we need to document the behaviour of all
> ioctls properly when the MC and !MC cases differ.
> 
>> The behavior as you're suggesting would be at least confusing for the
>> userspace, because it breaks one of the core principles of V4L2 and
>> also there is even no facility to allow the userspace handle this kind
>> of behavior, e.g. how to know which memory formats are supported for
>> given mbus formats of the related sink.
> 
> commit e5b6b07a1b45dd9d19bec1fa1d60750b0fcf2fb0
> Author: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Date:   Fri Apr 24 15:43:31 2020 +0200
> 
>      media: v4l2: Extend VIDIOC_ENUM_FMT to support MC-centric devices

Hi,
What if the video device entity wants to support V4L2_CAP_IO_MC and has two
pads registered to the media controller?
It seems to be undefined in the documentation.

Thanks,
Dafna

> 
> Been missing for a long time... I realized that when I started using the
> API in a real userspace implementation instead of just writing test
> tools or very device-specific code that could hardcode all this
> knowledge.
> 
>>>>> In MC-based
>>>>> drivers, entities are isolated from each other from a configuration
>>>>> point of view, and the validity of the configuration across the whole
>>>>> pipeline is checked at streamon time. I don't think CSC parameters
>>>>> should be treated differently than formats in this regard.
>>>>
>>>> Indeed, the ISP entity and the resizer entity are isolated from each
>>>> other and they need to be validated separately. However, the resizer
>>>> entity is not isolated from the video node, as there is no
>>>> configuration exposed for the video node in the MC API. If MC exposed
>>>> an explicit format control for the video interface sink pad, then the
>>>> resizer would have to be validated separately indeed, but still that
>>>> video interface sink pad would be tied to the rest of the
>>>> configuration of the video node (e.g. the CAPTURE format).
>>>>
>>>>>> Below is the rationale for that:
>>>>>> - In the MC model, the video node represents the DMA alone (and thus
>>>>>> the reason to have the resizer entities at all) and since the
>>>>>> colorspace is not a property of the DMA, it doesn't belong there.
>>>>>> - Even if we put that aside, both MP and SP can only output the same
>>>>>> colorspace, which would create a dependency between the configuration
>>>>>> of the two video nodes, which is not something well defined by V4L2.
>>>>>> - A video interface in the MC API doesn't expose control over its
>>>>>> input format, which means that the relation between the other side of
>>>>>> the link (resizer source) and the video node is implicit and needs to
>>>>>> be handled internally in the driver.
>>>>>> - V4L2, as opposed to MC, requires that the current state is always
>>>>>> consistent and valid. In this case it means that an ioctl on the video
>>>>>> node must not be able to alter the state in a way that would make it
>>>>>> inconsistent with the state of the matching resizer source.
> 

Re: [RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad

[Laurent Pinchart]

Hi Dafna,

On Mon, Jun 29, 2020 at 02:08:39PM +0200, Dafna Hirschfeld wrote:
> On 28.06.20 23:37, Laurent Pinchart wrote:
> > On Sun, Jun 28, 2020 at 11:33:30PM +0200, Tomasz Figa wrote:
> >> On Sun, Jun 28, 2020 at 10:27 PM Laurent Pinchart wrote:
> >>> On Sun, Jun 28, 2020 at 04:59:34PM +0200, Tomasz Figa wrote:
> >>>> On Sun, Jun 28, 2020 at 3:15 PM Laurent Pinchart wrote:
> >>>>> On Sun, Jun 28, 2020 at 03:10:41PM +0200, Tomasz Figa wrote:
> >>>>>> On Fri, Jun 26, 2020 at 2:15 AM Helen Koike <helen.koike@collabora.com> wrote:
> >>>>>>> On 6/18/20 4:25 PM, Tomasz Figa wrote:
> >>>>>>>> On Thu, Jun 18, 2020 at 08:54:34PM +0200, Dafna Hirschfeld wrote:
> >>>>>>>>> On 18.06.20 20:49, Tomasz Figa wrote:
> >>>>>>>>>> On Thu, Jun 18, 2020 at 08:26:33PM +0200, Dafna Hirschfeld wrote:
> >>>>>>>>>>> On 18.06.20 19:27, Tomasz Figa wrote:
> >>>>>>>>>>>> On Wed, Jun 10, 2020 at 2:08 PM Tomasz Figa wrote:
> >>>>>>>>>>>>> On Thu, Jun 4, 2020 at 9:11 PM Dafna Hirschfeld wrote:
> >>>>>>>>>>>>>> On 04.06.20 19:54, Tomasz Figa wrote:
> >>>>>>>>>>>>>>> On Thu, Apr 16, 2020 at 04:56:05PM +0200, Dafna Hirschfeld wrote:
> >>>>>>>>>>>>>>>> The isp entity has a hardware support to force full range quantization
> >>>>>>>>>>>>>>>> for YUV formats. Use the new API to indicate userspace that
> >>>>>>>>>>>>>>>> quantization conversion is supported by adding the flag
> >>>>>>>>>>>>>>>> V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION during media code enumeration.
> >>>>>>>>>>>>>>>> Then uppon s_fmt on the video source pad, we assign the
> >>>>>>>>>>>>>>>> quantization from userspace for YUV formats.
> >>>>>>>>>>>>>>>> Also in the capture and resizer entities we retrieve the colorspace
> >>>>>>>>>>>>>>>> from the isp entity.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
> >>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>      drivers/staging/media/rkisp1/rkisp1-capture.c | 28 ++++++-------
> >>>>>>>>>>>>>>>>      drivers/staging/media/rkisp1/rkisp1-common.h  |  2 +
> >>>>>>>>>>>>>>>>      drivers/staging/media/rkisp1/rkisp1-isp.c     | 39 +++++++++++++++++--
> >>>>>>>>>>>>>>>>      drivers/staging/media/rkisp1/rkisp1-resizer.c | 13 +++++++
> >>>>>>>>>>>>>>>>      4 files changed, 65 insertions(+), 17 deletions(-)
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c
> >>>>>>>>>>>>>>>> index fbf62399fe3d..aca0f93bc772 100644
> >>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-capture.c
> >>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-capture.c
> >>>>>>>>>>>>>>>> @@ -1066,14 +1066,13 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
> >>>>>>>>>>>>>>>>                            const struct v4l2_format_info **fmt_info)
> >>>>>>>>>>>>>>>>      {
> >>>>>>>>>>>>>>>>         const struct rkisp1_capture_config *config = cap->config;
> >>>>>>>>>>>>>>>> -    struct rkisp1_capture *other_cap =
> >>>>>>>>>>>>>>>> -                    &cap->rkisp1->capture_devs[cap->id ^ 1];
> >>>>>>>>>>>>>>>>         const struct rkisp1_capture_fmt_cfg *fmt;
> >>>>>>>>>>>>>>>>         const struct v4l2_format_info *info;
> >>>>>>>>>>>>>>>>         const unsigned int max_widths[] = { RKISP1_RSZ_MP_SRC_MAX_WIDTH,
> >>>>>>>>>>>>>>>>                                             RKISP1_RSZ_SP_SRC_MAX_WIDTH };
> >>>>>>>>>>>>>>>>         const unsigned int max_heights[] = { RKISP1_RSZ_MP_SRC_MAX_HEIGHT,
> >>>>>>>>>>>>>>>>                                              RKISP1_RSZ_SP_SRC_MAX_HEIGHT};
> >>>>>>>>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>         fmt = rkisp1_find_fmt_cfg(cap, pixm->pixelformat);
> >>>>>>>>>>>>>>>>         if (!fmt) {
> >>>>>>>>>>>>>>>> @@ -1081,24 +1080,27 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
> >>>>>>>>>>>>>>>>                 pixm->pixelformat = fmt->fourcc;
> >>>>>>>>>>>>>>>>         }
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> +    rkisp1_get_isp_src_fmt(cap->rkisp1, &isp_sd_fmt);
> >>>>>>>>>>>>>>>> +    pixm->field = isp_sd_fmt.format.field;
> >>>>>>>>>>>>>>>> +    pixm->colorspace = isp_sd_fmt.format.colorspace;
> >>>>>>>>>>>>>>>> +    pixm->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
> >>>>>>>>>>>>>>>> +    pixm->xfer_func = isp_sd_fmt.format.xfer_func;
> >>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>> +    /*
> >>>>>>>>>>>>>>>> +     * isp has a feature to set full range quantization for yuv formats.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> How about "select between limited and full range for YUV formats"?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> +     * so we need to get the format from the isp.
> >>>>>>>>>>>>>>>> +     */
> >>>>>>>>>>>>>>>> +    pixm->quantization = isp_sd_fmt.format.quantization;
> >>>>>>>>>>>>>>>> +    if (!v4l2_is_format_yuv(cap->pix.info))
> >>>>>>>>>>>>>>>> +            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>         pixm->width = clamp_t(u32, pixm->width,
> >>>>>>>>>>>>>>>>                               RKISP1_RSZ_SRC_MIN_WIDTH, max_widths[cap->id]);
> >>>>>>>>>>>>>>>>         pixm->height = clamp_t(u32, pixm->height,
> >>>>>>>>>>>>>>>>                                RKISP1_RSZ_SRC_MIN_HEIGHT, max_heights[cap->id]);
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> -    pixm->field = V4L2_FIELD_NONE;
> >>>>>>>>>>>>>>>> -    pixm->colorspace = V4L2_COLORSPACE_DEFAULT;
> >>>>>>>>>>>>>>>> -    pixm->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> >>>>>>>>>>>>>>>> -
> >>>>>>>>>>>>>>>>         info = rkisp1_fill_pixfmt(pixm, cap->id);
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> -    /* can not change quantization when stream-on */
> >>>>>>>>>>>>>>>> -    if (other_cap->is_streaming)
> >>>>>>>>>>>>>>>> -            pixm->quantization = other_cap->pix.fmt.quantization;
> >>>>>>>>>>>>>>>> -    /* output full range by default, take effect in params */
> >>>>>>>>>>>>>>>> -    else if (!pixm->quantization ||
> >>>>>>>>>>>>>>>> -             pixm->quantization > V4L2_QUANTIZATION_LIM_RANGE)
> >>>>>>>>>>>>>>>> -            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>         if (fmt_cfg)
> >>>>>>>>>>>>>>>>                 *fmt_cfg = fmt;
> >>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-common.h b/drivers/staging/media/rkisp1/rkisp1-common.h
> >>>>>>>>>>>>>>>> index 2d7b7e078636..7a5576fa14c9 100644
> >>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-common.h
> >>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-common.h
> >>>>>>>>>>>>>>>> @@ -300,6 +300,8 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
> >>>>>>>>>>>>>>>>                         struct v4l2_device *v4l2_dev);
> >>>>>>>>>>>>>>>>      void rkisp1_isp_unregister(struct rkisp1_device *rkisp1);
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
> >>>>>>>>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt);
> >>>>>>>>>>>>>>>>      const struct rkisp1_isp_mbus_info *rkisp1_isp_mbus_info_get(u32 mbus_code);
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>      irqreturn_t rkisp1_isp_isr(struct rkisp1_device *rkisp1);
> >>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-isp.c b/drivers/staging/media/rkisp1/rkisp1-isp.c
> >>>>>>>>>>>>>>>> index dee8e96f3900..6fdf5ed0b6b1 100644
> >>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-isp.c
> >>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-isp.c
> >>>>>>>>>>>>>>>> @@ -613,6 +613,10 @@ static int rkisp1_isp_enum_mbus_code(struct v4l2_subdev *sd,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>                 if (code->index == pos - 1) {
> >>>>>>>>>>>>>>>>                         code->code = fmt->mbus_code;
> >>>>>>>>>>>>>>>> +                    if (fmt->pixel_enc == V4L2_PIXEL_ENC_YUV &&
> >>>>>>>>>>>>>>>> +                        dir == RKISP1_DIR_SRC)
> >>>>>>>>>>>>>>>> +                            code->flags =
> >>>>>>>>>>>>>>>> +                                    V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION;
> >>>>>>>>>>>>>>>>                         return 0;
> >>>>>>>>>>>>>>>>                 }
> >>>>>>>>>>>>>>>>         }
> >>>>>>>>>>>>>>>> @@ -639,12 +643,21 @@ static int rkisp1_isp_init_config(struct v4l2_subdev *sd,
> >>>>>>>>>>>>>>>>         sink_crop->height = RKISP1_DEFAULT_HEIGHT;
> >>>>>>>>>>>>>>>>         sink_crop->left = 0;
> >>>>>>>>>>>>>>>>         sink_crop->top = 0;
> >>>>>>>>>>>>>>>> +    sink_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >>>>>>>>>>>>>>>> +    sink_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(sink_fmt->colorspace);
> >>>>>>>>>>>>>>>> +    sink_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(sink_fmt->colorspace);
> >>>>>>>>>>>>>>>> +    sink_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>         src_fmt = v4l2_subdev_get_try_format(sd, cfg,
> >>>>>>>>>>>>>>>>                                              RKISP1_ISP_PAD_SOURCE_VIDEO);
> >>>>>>>>>>>>>>>>         *src_fmt = *sink_fmt;
> >>>>>>>>>>>>>>>>         src_fmt->code = RKISP1_DEF_SRC_PAD_FMT;
> >>>>>>>>>>>>>>>> -    src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >>>>>>>>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>>>>>>>>> +    src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
> >>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>         src_crop = v4l2_subdev_get_try_crop(sd, cfg,
> >>>>>>>>>>>>>>>>                                             RKISP1_ISP_PAD_SOURCE_VIDEO);
> >>>>>>>>>>>>>>>> @@ -687,10 +700,17 @@ static void rkisp1_isp_set_src_fmt(struct rkisp1_isp *isp,
> >>>>>>>>>>>>>>>>                 isp->src_fmt = mbus_info;
> >>>>>>>>>>>>>>>>         src_fmt->width  = src_crop->width;
> >>>>>>>>>>>>>>>>         src_fmt->height = src_crop->height;
> >>>>>>>>>>>>>>>> -    src_fmt->quantization = format->quantization;
> >>>>>>>>>>>>>>>> -    /* full range by default */
> >>>>>>>>>>>>>>>> -    if (!src_fmt->quantization)
> >>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >>>>>>>>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>> +    if (mbus_info->pixel_enc == V4L2_PIXEL_ENC_BAYER)
> >>>>>>>>>>>>>>>>                 src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>>>>> +    else if (format->quantization == V4L2_QUANTIZATION_DEFAULT)
> >>>>>>>>>>>>>>>> +            src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
> >>>>>>>>>>>>>>>> +    else
> >>>>>>>>>>>>>>>> +            src_fmt->quantization = format->quantization;
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>         *format = *src_fmt;
> >>>>>>>>>>>>>>>>      }
> >>>>>>>>>>>>>>>> @@ -1068,6 +1088,17 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
> >>>>>>>>>>>>>>>>         return ret;
> >>>>>>>>>>>>>>>>      }
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
> >>>>>>>>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt)
> >>>>>>>>>>>>>>>> +{
> >>>>>>>>>>>>>>>> +    struct rkisp1_isp *isp = &rkisp1->isp;
> >>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>> +    sd_fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
> >>>>>>>>>>>>>>>> +    sd_fmt->pad = RKISP1_ISP_PAD_SOURCE_VIDEO;
> >>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>> +    return v4l2_subdev_call(&isp->sd, pad, get_fmt, NULL, sd_fmt);
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Do we need to get through the external API to access data that is
> >>>>>>>>>>>>>>> driver-internal anyway?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> +}
> >>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>      void rkisp1_isp_unregister(struct rkisp1_device *rkisp1)
> >>>>>>>>>>>>>>>>      {
> >>>>>>>>>>>>>>>>         struct v4l2_subdev *sd = &rkisp1->isp.sd;
> >>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-resizer.c b/drivers/staging/media/rkisp1/rkisp1-resizer.c
> >>>>>>>>>>>>>>>> index 7b6b7ddd4169..8705b133de68 100644
> >>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-resizer.c
> >>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-resizer.c
> >>>>>>>>>>>>>>>> @@ -525,6 +525,7 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
> >>>>>>>>>>>>>>>>         const struct rkisp1_isp_mbus_info *mbus_info;
> >>>>>>>>>>>>>>>>         struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
> >>>>>>>>>>>>>>>>         struct v4l2_rect *sink_crop;
> >>>>>>>>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>         sink_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SINK, which);
> >>>>>>>>>>>>>>>>         src_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SRC, which);
> >>>>>>>>>>>>>>>> @@ -539,8 +540,20 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
> >>>>>>>>>>>>>>>>         if (which == V4L2_SUBDEV_FORMAT_ACTIVE)
> >>>>>>>>>>>>>>>>                 rsz->pixel_enc = mbus_info->pixel_enc;
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> +    rkisp1_get_isp_src_fmt(rsz->rkisp1, &isp_sd_fmt);
> >>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Is this necessary? My understanding was that in the subdev model, it was
> >>>>>>>>>>>>>>> the userspace responsibility to propagate any configuration changes through
> >>>>>>>>>>>>>>> the graph.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Also, doing this only here wouldn't fully maintain the
> >>>>>>>>>>>>>>> consistency of the state. For example, if one sets the ISP subdev format
> >>>>>>>>>>>>>>> first, then the resizer subdev and then the ISP subdev again, wouldn't the
> >>>>>>>>>>>>>>> resizer subdev end up with a wrong format?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> yes, this is indeed a bug, I am preparing v4 now.
> >>>>>>>>>>>>>> What I thought to do is adding quantization conversion
> >>>>>>>>>>>>>> support also on ther resizer and capture entities.
> >>>>>>>>>>>>>> Then in the 'link_validation' callbacks, there
> >>>>>>>>>>>>>> is a validation that the quantization fields matches.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> My understanding is that, if we have the following topology
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> [ ISP ] -> [ Resizer ] -> [ Video node ]
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> then the ISP source pad would have the csc capability, while resizer
> >>>>>>>>>>>>> and video node would just accept whatever is configured on their sink
> >>>>>>>>>>>>> pads (no need for csc capability for that) and propagate that to their
> >>>>>>>>>>>>> outputs, i.e. resizer source pad and video node CAPTURE format.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Is this what you were going to implement?
> >>>>>>>>>>> Hi, I sent a v4 where the CSC capability is set on the reiszer and capture as well.
> >>>>>>>>>>> I can send a v5 implementing it the way you suggest. Currently the doc says that the colorspace fields
> >>>>>>>>>>> must be set by the driver for capture streams. This implies that userspace can set it
> >>>>>>>>>>> only if the CSC is supported.
> >>>>>>>>>>
> >>>>>>>>>> Why would the userspace have to set it for the capture stream on the
> >>>>>>>>>> resizer and video nodes? Couldn't the userspace set it to DEFAULT and
> >>>>>>>>>> then the driver override to whatever it received on the corresponding
> >>>>>>>>>> sink?
> >>>>>>>
> >>>>>>> According to the documentation [1]:
> >>>>>>>
> >>>>>>> "the default quantization encoding as defined by the colorspace"
> >>>>>>>
> >>>>>>> Then, for instance, the default for V4L2_COLORSPACE_SRGB is limited range [2].
> >>>>>>>
> >>>>>>> Unless we change the definition of DEFAULT to let it accept whatever it receives on the corresponding sink.
> >>>>>>>
> >>>>>>> Then the way I understand is:
> >>>>>>>
> >>>>>>> [sensor]->[isp]->[resizer]->[capture]
> >>>>>>>
> >>>>>>> Userspace should set quantization on:
> >>>>>>> * sensor source pad
> >>>>>>> * isp sink pad
> >>>>>>> * resizer sink pad
> >>>>>>> * capture node
> >>>>>>>
> >>>>>>> The remaining source pads would get values propagated from their sink pads.
> >>>>>>>
> >>>>>>> [1] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-defs.html#c.v4l2_quantization
> >>>>>>> [2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-details.html#col-srgb
> >>>>>>
> >>>>>> After discussing this offline, we concluded that:
> >>>>>> - only the source pad of the ISP entity should expose the CSC capability,
> >>>>>> - the CAPTURE video node should expose only the formats/color spaces
> >>>>>> compatible with the current setting of the resizer source,
> >>>>>> - consistency between the resizer source format and CAPTURE format
> >>>>>> should be implicitly maintained by the driver.
> >>>>>
> >>>>> Do you mean it should be validated at streamon time ?
> >>>>
> >>>> Nope. Any V4L2 state must be kept valid all the time, as mandated by
> >>>> the general V4L2 principles. Wrong configuration of the video node
> >>>> must not be a reason for STREAMON failing.
> >>>
> >>> That's a wrong interpretation of V4L2. On a video-node-centric device (a
> >>> device that doesn't use the subdev userspace API for configuration) this
> >>> is correct. In a MC-centric device, the video node and its connected
> >>> subdev are configured separately.
> >>
> >> I don't see this documented anywhere in the specification, maybe
> >> except a mention of EPIPE or ENOLINK being returned from STREAMON, but
> >> they explicitly refer to pipeline configuration and not a conflict
> >> between V4L2 and MC state.
> > 
> > We're suffering from a lack of documentation in this area :-( Niklas has
> > authored
> > 
> > commit f645e6256bd1b12523b759fcc610861fb21c24c7
> > Author: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
> > Date:   Tue Apr 21 15:57:38 2020 +0200
> > 
> >      media: v4l2-dev/ioctl: Add V4L2_CAP_IO_MC
> > 
> > which is a first step, now we need to document the behaviour of all
> > ioctls properly when the MC and !MC cases differ.
> > 
> >> The behavior as you're suggesting would be at least confusing for the
> >> userspace, because it breaks one of the core principles of V4L2 and
> >> also there is even no facility to allow the userspace handle this kind
> >> of behavior, e.g. how to know which memory formats are supported for
> >> given mbus formats of the related sink.
> > 
> > commit e5b6b07a1b45dd9d19bec1fa1d60750b0fcf2fb0
> > Author: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > Date:   Fri Apr 24 15:43:31 2020 +0200
> > 
> >      media: v4l2: Extend VIDIOC_ENUM_FMT to support MC-centric devices
> 
> Hi,
> What if the video device entity wants to support V4L2_CAP_IO_MC and has two
> pads registered to the media controller?
> It seems to be undefined in the documentation.

What kind of video device would that be ?

> > Been missing for a long time... I realized that when I started using the
> > API in a real userspace implementation instead of just writing test
> > tools or very device-specific code that could hardcode all this
> > knowledge.
> > 
> >>>>> In MC-based
> >>>>> drivers, entities are isolated from each other from a configuration
> >>>>> point of view, and the validity of the configuration across the whole
> >>>>> pipeline is checked at streamon time. I don't think CSC parameters
> >>>>> should be treated differently than formats in this regard.
> >>>>
> >>>> Indeed, the ISP entity and the resizer entity are isolated from each
> >>>> other and they need to be validated separately. However, the resizer
> >>>> entity is not isolated from the video node, as there is no
> >>>> configuration exposed for the video node in the MC API. If MC exposed
> >>>> an explicit format control for the video interface sink pad, then the
> >>>> resizer would have to be validated separately indeed, but still that
> >>>> video interface sink pad would be tied to the rest of the
> >>>> configuration of the video node (e.g. the CAPTURE format).
> >>>>
> >>>>>> Below is the rationale for that:
> >>>>>> - In the MC model, the video node represents the DMA alone (and thus
> >>>>>> the reason to have the resizer entities at all) and since the
> >>>>>> colorspace is not a property of the DMA, it doesn't belong there.
> >>>>>> - Even if we put that aside, both MP and SP can only output the same
> >>>>>> colorspace, which would create a dependency between the configuration
> >>>>>> of the two video nodes, which is not something well defined by V4L2.
> >>>>>> - A video interface in the MC API doesn't expose control over its
> >>>>>> input format, which means that the relation between the other side of
> >>>>>> the link (resizer source) and the video node is implicit and needs to
> >>>>>> be handled internally in the driver.
> >>>>>> - V4L2, as opposed to MC, requires that the current state is always
> >>>>>> consistent and valid. In this case it means that an ioctl on the video
> >>>>>> node must not be able to alter the state in a way that would make it
> >>>>>> inconsistent with the state of the matching resizer source.

-- 
Regards,

Laurent Pinchart

Re: [RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad

[Dafna Hirschfeld]

On 29.06.20 14:21, Laurent Pinchart wrote:
> Hi Dafna,
> 
> On Mon, Jun 29, 2020 at 02:08:39PM +0200, Dafna Hirschfeld wrote:
>> On 28.06.20 23:37, Laurent Pinchart wrote:
>>> On Sun, Jun 28, 2020 at 11:33:30PM +0200, Tomasz Figa wrote:
>>>> On Sun, Jun 28, 2020 at 10:27 PM Laurent Pinchart wrote:
>>>>> On Sun, Jun 28, 2020 at 04:59:34PM +0200, Tomasz Figa wrote:
>>>>>> On Sun, Jun 28, 2020 at 3:15 PM Laurent Pinchart wrote:
>>>>>>> On Sun, Jun 28, 2020 at 03:10:41PM +0200, Tomasz Figa wrote:
>>>>>>>> On Fri, Jun 26, 2020 at 2:15 AM Helen Koike <helen.koike@collabora.com> wrote:
>>>>>>>>> On 6/18/20 4:25 PM, Tomasz Figa wrote:
>>>>>>>>>> On Thu, Jun 18, 2020 at 08:54:34PM +0200, Dafna Hirschfeld wrote:
>>>>>>>>>>> On 18.06.20 20:49, Tomasz Figa wrote:
>>>>>>>>>>>> On Thu, Jun 18, 2020 at 08:26:33PM +0200, Dafna Hirschfeld wrote:
>>>>>>>>>>>>> On 18.06.20 19:27, Tomasz Figa wrote:
>>>>>>>>>>>>>> On Wed, Jun 10, 2020 at 2:08 PM Tomasz Figa wrote:
>>>>>>>>>>>>>>> On Thu, Jun 4, 2020 at 9:11 PM Dafna Hirschfeld wrote:
>>>>>>>>>>>>>>>> On 04.06.20 19:54, Tomasz Figa wrote:
>>>>>>>>>>>>>>>>> On Thu, Apr 16, 2020 at 04:56:05PM +0200, Dafna Hirschfeld wrote:
>>>>>>>>>>>>>>>>>> The isp entity has a hardware support to force full range quantization
>>>>>>>>>>>>>>>>>> for YUV formats. Use the new API to indicate userspace that
>>>>>>>>>>>>>>>>>> quantization conversion is supported by adding the flag
>>>>>>>>>>>>>>>>>> V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION during media code enumeration.
>>>>>>>>>>>>>>>>>> Then uppon s_fmt on the video source pad, we assign the
>>>>>>>>>>>>>>>>>> quantization from userspace for YUV formats.
>>>>>>>>>>>>>>>>>> Also in the capture and resizer entities we retrieve the colorspace
>>>>>>>>>>>>>>>>>> from the isp entity.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>       drivers/staging/media/rkisp1/rkisp1-capture.c | 28 ++++++-------
>>>>>>>>>>>>>>>>>>       drivers/staging/media/rkisp1/rkisp1-common.h  |  2 +
>>>>>>>>>>>>>>>>>>       drivers/staging/media/rkisp1/rkisp1-isp.c     | 39 +++++++++++++++++--
>>>>>>>>>>>>>>>>>>       drivers/staging/media/rkisp1/rkisp1-resizer.c | 13 +++++++
>>>>>>>>>>>>>>>>>>       4 files changed, 65 insertions(+), 17 deletions(-)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>>>>>>>>>>>>>> index fbf62399fe3d..aca0f93bc772 100644
>>>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>>>>>>>>>>>>>> @@ -1066,14 +1066,13 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
>>>>>>>>>>>>>>>>>>                             const struct v4l2_format_info **fmt_info)
>>>>>>>>>>>>>>>>>>       {
>>>>>>>>>>>>>>>>>>          const struct rkisp1_capture_config *config = cap->config;
>>>>>>>>>>>>>>>>>> -    struct rkisp1_capture *other_cap =
>>>>>>>>>>>>>>>>>> -                    &cap->rkisp1->capture_devs[cap->id ^ 1];
>>>>>>>>>>>>>>>>>>          const struct rkisp1_capture_fmt_cfg *fmt;
>>>>>>>>>>>>>>>>>>          const struct v4l2_format_info *info;
>>>>>>>>>>>>>>>>>>          const unsigned int max_widths[] = { RKISP1_RSZ_MP_SRC_MAX_WIDTH,
>>>>>>>>>>>>>>>>>>                                              RKISP1_RSZ_SP_SRC_MAX_WIDTH };
>>>>>>>>>>>>>>>>>>          const unsigned int max_heights[] = { RKISP1_RSZ_MP_SRC_MAX_HEIGHT,
>>>>>>>>>>>>>>>>>>                                               RKISP1_RSZ_SP_SRC_MAX_HEIGHT};
>>>>>>>>>>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>          fmt = rkisp1_find_fmt_cfg(cap, pixm->pixelformat);
>>>>>>>>>>>>>>>>>>          if (!fmt) {
>>>>>>>>>>>>>>>>>> @@ -1081,24 +1080,27 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
>>>>>>>>>>>>>>>>>>                  pixm->pixelformat = fmt->fourcc;
>>>>>>>>>>>>>>>>>>          }
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> +    rkisp1_get_isp_src_fmt(cap->rkisp1, &isp_sd_fmt);
>>>>>>>>>>>>>>>>>> +    pixm->field = isp_sd_fmt.format.field;
>>>>>>>>>>>>>>>>>> +    pixm->colorspace = isp_sd_fmt.format.colorspace;
>>>>>>>>>>>>>>>>>> +    pixm->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
>>>>>>>>>>>>>>>>>> +    pixm->xfer_func = isp_sd_fmt.format.xfer_func;
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>> +    /*
>>>>>>>>>>>>>>>>>> +     * isp has a feature to set full range quantization for yuv formats.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> How about "select between limited and full range for YUV formats"?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> +     * so we need to get the format from the isp.
>>>>>>>>>>>>>>>>>> +     */
>>>>>>>>>>>>>>>>>> +    pixm->quantization = isp_sd_fmt.format.quantization;
>>>>>>>>>>>>>>>>>> +    if (!v4l2_is_format_yuv(cap->pix.info))
>>>>>>>>>>>>>>>>>> +            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>          pixm->width = clamp_t(u32, pixm->width,
>>>>>>>>>>>>>>>>>>                                RKISP1_RSZ_SRC_MIN_WIDTH, max_widths[cap->id]);
>>>>>>>>>>>>>>>>>>          pixm->height = clamp_t(u32, pixm->height,
>>>>>>>>>>>>>>>>>>                                 RKISP1_RSZ_SRC_MIN_HEIGHT, max_heights[cap->id]);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -    pixm->field = V4L2_FIELD_NONE;
>>>>>>>>>>>>>>>>>> -    pixm->colorspace = V4L2_COLORSPACE_DEFAULT;
>>>>>>>>>>>>>>>>>> -    pixm->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>>>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>>>          info = rkisp1_fill_pixfmt(pixm, cap->id);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -    /* can not change quantization when stream-on */
>>>>>>>>>>>>>>>>>> -    if (other_cap->is_streaming)
>>>>>>>>>>>>>>>>>> -            pixm->quantization = other_cap->pix.fmt.quantization;
>>>>>>>>>>>>>>>>>> -    /* output full range by default, take effect in params */
>>>>>>>>>>>>>>>>>> -    else if (!pixm->quantization ||
>>>>>>>>>>>>>>>>>> -             pixm->quantization > V4L2_QUANTIZATION_LIM_RANGE)
>>>>>>>>>>>>>>>>>> -            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>          if (fmt_cfg)
>>>>>>>>>>>>>>>>>>                  *fmt_cfg = fmt;
>>>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-common.h b/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>>>>>>>>>>>>>> index 2d7b7e078636..7a5576fa14c9 100644
>>>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>>>>>>>>>>>>>> @@ -300,6 +300,8 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
>>>>>>>>>>>>>>>>>>                          struct v4l2_device *v4l2_dev);
>>>>>>>>>>>>>>>>>>       void rkisp1_isp_unregister(struct rkisp1_device *rkisp1);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
>>>>>>>>>>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt);
>>>>>>>>>>>>>>>>>>       const struct rkisp1_isp_mbus_info *rkisp1_isp_mbus_info_get(u32 mbus_code);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>       irqreturn_t rkisp1_isp_isr(struct rkisp1_device *rkisp1);
>>>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-isp.c b/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>>>>>>>>>>>>>> index dee8e96f3900..6fdf5ed0b6b1 100644
>>>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>>>>>>>>>>>>>> @@ -613,6 +613,10 @@ static int rkisp1_isp_enum_mbus_code(struct v4l2_subdev *sd,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                  if (code->index == pos - 1) {
>>>>>>>>>>>>>>>>>>                          code->code = fmt->mbus_code;
>>>>>>>>>>>>>>>>>> +                    if (fmt->pixel_enc == V4L2_PIXEL_ENC_YUV &&
>>>>>>>>>>>>>>>>>> +                        dir == RKISP1_DIR_SRC)
>>>>>>>>>>>>>>>>>> +                            code->flags =
>>>>>>>>>>>>>>>>>> +                                    V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION;
>>>>>>>>>>>>>>>>>>                          return 0;
>>>>>>>>>>>>>>>>>>                  }
>>>>>>>>>>>>>>>>>>          }
>>>>>>>>>>>>>>>>>> @@ -639,12 +643,21 @@ static int rkisp1_isp_init_config(struct v4l2_subdev *sd,
>>>>>>>>>>>>>>>>>>          sink_crop->height = RKISP1_DEFAULT_HEIGHT;
>>>>>>>>>>>>>>>>>>          sink_crop->left = 0;
>>>>>>>>>>>>>>>>>>          sink_crop->top = 0;
>>>>>>>>>>>>>>>>>> +    sink_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>>>>>>>>>>>>>> +    sink_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(sink_fmt->colorspace);
>>>>>>>>>>>>>>>>>> +    sink_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(sink_fmt->colorspace);
>>>>>>>>>>>>>>>>>> +    sink_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>          src_fmt = v4l2_subdev_get_try_format(sd, cfg,
>>>>>>>>>>>>>>>>>>                                               RKISP1_ISP_PAD_SOURCE_VIDEO);
>>>>>>>>>>>>>>>>>>          *src_fmt = *sink_fmt;
>>>>>>>>>>>>>>>>>>          src_fmt->code = RKISP1_DEF_SRC_PAD_FMT;
>>>>>>>>>>>>>>>>>> -    src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>>>>>>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
>>>>>>>>>>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
>>>>>>>>>>>>>>>>>> +    src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>          src_crop = v4l2_subdev_get_try_crop(sd, cfg,
>>>>>>>>>>>>>>>>>>                                              RKISP1_ISP_PAD_SOURCE_VIDEO);
>>>>>>>>>>>>>>>>>> @@ -687,10 +700,17 @@ static void rkisp1_isp_set_src_fmt(struct rkisp1_isp *isp,
>>>>>>>>>>>>>>>>>>                  isp->src_fmt = mbus_info;
>>>>>>>>>>>>>>>>>>          src_fmt->width  = src_crop->width;
>>>>>>>>>>>>>>>>>>          src_fmt->height = src_crop->height;
>>>>>>>>>>>>>>>>>> -    src_fmt->quantization = format->quantization;
>>>>>>>>>>>>>>>>>> -    /* full range by default */
>>>>>>>>>>>>>>>>>> -    if (!src_fmt->quantization)
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>>>>>>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
>>>>>>>>>>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>> +    if (mbus_info->pixel_enc == V4L2_PIXEL_ENC_BAYER)
>>>>>>>>>>>>>>>>>>                  src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>>>>>>>>>> +    else if (format->quantization == V4L2_QUANTIZATION_DEFAULT)
>>>>>>>>>>>>>>>>>> +            src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
>>>>>>>>>>>>>>>>>> +    else
>>>>>>>>>>>>>>>>>> +            src_fmt->quantization = format->quantization;
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>          *format = *src_fmt;
>>>>>>>>>>>>>>>>>>       }
>>>>>>>>>>>>>>>>>> @@ -1068,6 +1088,17 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
>>>>>>>>>>>>>>>>>>          return ret;
>>>>>>>>>>>>>>>>>>       }
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
>>>>>>>>>>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt)
>>>>>>>>>>>>>>>>>> +{
>>>>>>>>>>>>>>>>>> +    struct rkisp1_isp *isp = &rkisp1->isp;
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>> +    sd_fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
>>>>>>>>>>>>>>>>>> +    sd_fmt->pad = RKISP1_ISP_PAD_SOURCE_VIDEO;
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>> +    return v4l2_subdev_call(&isp->sd, pad, get_fmt, NULL, sd_fmt);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Do we need to get through the external API to access data that is
>>>>>>>>>>>>>>>>> driver-internal anyway?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> +}
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>       void rkisp1_isp_unregister(struct rkisp1_device *rkisp1)
>>>>>>>>>>>>>>>>>>       {
>>>>>>>>>>>>>>>>>>          struct v4l2_subdev *sd = &rkisp1->isp.sd;
>>>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-resizer.c b/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>>>>>>>>>>>>>> index 7b6b7ddd4169..8705b133de68 100644
>>>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>>>>>>>>>>>>>> @@ -525,6 +525,7 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
>>>>>>>>>>>>>>>>>>          const struct rkisp1_isp_mbus_info *mbus_info;
>>>>>>>>>>>>>>>>>>          struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
>>>>>>>>>>>>>>>>>>          struct v4l2_rect *sink_crop;
>>>>>>>>>>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>          sink_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SINK, which);
>>>>>>>>>>>>>>>>>>          src_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SRC, which);
>>>>>>>>>>>>>>>>>> @@ -539,8 +540,20 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
>>>>>>>>>>>>>>>>>>          if (which == V4L2_SUBDEV_FORMAT_ACTIVE)
>>>>>>>>>>>>>>>>>>                  rsz->pixel_enc = mbus_info->pixel_enc;
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> +    rkisp1_get_isp_src_fmt(rsz->rkisp1, &isp_sd_fmt);
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Is this necessary? My understanding was that in the subdev model, it was
>>>>>>>>>>>>>>>>> the userspace responsibility to propagate any configuration changes through
>>>>>>>>>>>>>>>>> the graph.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Also, doing this only here wouldn't fully maintain the
>>>>>>>>>>>>>>>>> consistency of the state. For example, if one sets the ISP subdev format
>>>>>>>>>>>>>>>>> first, then the resizer subdev and then the ISP subdev again, wouldn't the
>>>>>>>>>>>>>>>>> resizer subdev end up with a wrong format?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> yes, this is indeed a bug, I am preparing v4 now.
>>>>>>>>>>>>>>>> What I thought to do is adding quantization conversion
>>>>>>>>>>>>>>>> support also on ther resizer and capture entities.
>>>>>>>>>>>>>>>> Then in the 'link_validation' callbacks, there
>>>>>>>>>>>>>>>> is a validation that the quantization fields matches.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> My understanding is that, if we have the following topology
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [ ISP ] -> [ Resizer ] -> [ Video node ]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> then the ISP source pad would have the csc capability, while resizer
>>>>>>>>>>>>>>> and video node would just accept whatever is configured on their sink
>>>>>>>>>>>>>>> pads (no need for csc capability for that) and propagate that to their
>>>>>>>>>>>>>>> outputs, i.e. resizer source pad and video node CAPTURE format.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Is this what you were going to implement?
>>>>>>>>>>>>> Hi, I sent a v4 where the CSC capability is set on the reiszer and capture as well.
>>>>>>>>>>>>> I can send a v5 implementing it the way you suggest. Currently the doc says that the colorspace fields
>>>>>>>>>>>>> must be set by the driver for capture streams. This implies that userspace can set it
>>>>>>>>>>>>> only if the CSC is supported.
>>>>>>>>>>>>
>>>>>>>>>>>> Why would the userspace have to set it for the capture stream on the
>>>>>>>>>>>> resizer and video nodes? Couldn't the userspace set it to DEFAULT and
>>>>>>>>>>>> then the driver override to whatever it received on the corresponding
>>>>>>>>>>>> sink?
>>>>>>>>>
>>>>>>>>> According to the documentation [1]:
>>>>>>>>>
>>>>>>>>> "the default quantization encoding as defined by the colorspace"
>>>>>>>>>
>>>>>>>>> Then, for instance, the default for V4L2_COLORSPACE_SRGB is limited range [2].
>>>>>>>>>
>>>>>>>>> Unless we change the definition of DEFAULT to let it accept whatever it receives on the corresponding sink.
>>>>>>>>>
>>>>>>>>> Then the way I understand is:
>>>>>>>>>
>>>>>>>>> [sensor]->[isp]->[resizer]->[capture]
>>>>>>>>>
>>>>>>>>> Userspace should set quantization on:
>>>>>>>>> * sensor source pad
>>>>>>>>> * isp sink pad
>>>>>>>>> * resizer sink pad
>>>>>>>>> * capture node
>>>>>>>>>
>>>>>>>>> The remaining source pads would get values propagated from their sink pads.
>>>>>>>>>
>>>>>>>>> [1] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-defs.html#c.v4l2_quantization
>>>>>>>>> [2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-details.html#col-srgb
>>>>>>>>
>>>>>>>> After discussing this offline, we concluded that:
>>>>>>>> - only the source pad of the ISP entity should expose the CSC capability,
>>>>>>>> - the CAPTURE video node should expose only the formats/color spaces
>>>>>>>> compatible with the current setting of the resizer source,
>>>>>>>> - consistency between the resizer source format and CAPTURE format
>>>>>>>> should be implicitly maintained by the driver.
>>>>>>>
>>>>>>> Do you mean it should be validated at streamon time ?
>>>>>>
>>>>>> Nope. Any V4L2 state must be kept valid all the time, as mandated by
>>>>>> the general V4L2 principles. Wrong configuration of the video node
>>>>>> must not be a reason for STREAMON failing.
>>>>>
>>>>> That's a wrong interpretation of V4L2. On a video-node-centric device (a
>>>>> device that doesn't use the subdev userspace API for configuration) this
>>>>> is correct. In a MC-centric device, the video node and its connected
>>>>> subdev are configured separately.
>>>>
>>>> I don't see this documented anywhere in the specification, maybe
>>>> except a mention of EPIPE or ENOLINK being returned from STREAMON, but
>>>> they explicitly refer to pipeline configuration and not a conflict
>>>> between V4L2 and MC state.
>>>
>>> We're suffering from a lack of documentation in this area :-( Niklas has
>>> authored
>>>
>>> commit f645e6256bd1b12523b759fcc610861fb21c24c7
>>> Author: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
>>> Date:   Tue Apr 21 15:57:38 2020 +0200
>>>
>>>       media: v4l2-dev/ioctl: Add V4L2_CAP_IO_MC
>>>
>>> which is a first step, now we need to document the behaviour of all
>>> ioctls properly when the MC and !MC cases differ.
>>>
>>>> The behavior as you're suggesting would be at least confusing for the
>>>> userspace, because it breaks one of the core principles of V4L2 and
>>>> also there is even no facility to allow the userspace handle this kind
>>>> of behavior, e.g. how to know which memory formats are supported for
>>>> given mbus formats of the related sink.
>>>
>>> commit e5b6b07a1b45dd9d19bec1fa1d60750b0fcf2fb0
>>> Author: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
>>> Date:   Fri Apr 24 15:43:31 2020 +0200
>>>
>>>       media: v4l2: Extend VIDIOC_ENUM_FMT to support MC-centric devices
>>
>> Hi,
>> What if the video device entity wants to support V4L2_CAP_IO_MC and has two
>> pads registered to the media controller?
>> It seems to be undefined in the documentation.
> 
> What kind of video device would that be ?

I don't know, it is more a theoretical question.




> 
>>> Been missing for a long time... I realized that when I started using the
>>> API in a real userspace implementation instead of just writing test
>>> tools or very device-specific code that could hardcode all this
>>> knowledge.
>>>
>>>>>>> In MC-based
>>>>>>> drivers, entities are isolated from each other from a configuration
>>>>>>> point of view, and the validity of the configuration across the whole
>>>>>>> pipeline is checked at streamon time. I don't think CSC parameters
>>>>>>> should be treated differently than formats in this regard.
>>>>>>
>>>>>> Indeed, the ISP entity and the resizer entity are isolated from each
>>>>>> other and they need to be validated separately. However, the resizer
>>>>>> entity is not isolated from the video node, as there is no
>>>>>> configuration exposed for the video node in the MC API. If MC exposed
>>>>>> an explicit format control for the video interface sink pad, then the
>>>>>> resizer would have to be validated separately indeed, but still that
>>>>>> video interface sink pad would be tied to the rest of the
>>>>>> configuration of the video node (e.g. the CAPTURE format).
>>>>>>
>>>>>>>> Below is the rationale for that:
>>>>>>>> - In the MC model, the video node represents the DMA alone (and thus
>>>>>>>> the reason to have the resizer entities at all) and since the
>>>>>>>> colorspace is not a property of the DMA, it doesn't belong there.
>>>>>>>> - Even if we put that aside, both MP and SP can only output the same
>>>>>>>> colorspace, which would create a dependency between the configuration
>>>>>>>> of the two video nodes, which is not something well defined by V4L2.
>>>>>>>> - A video interface in the MC API doesn't expose control over its
>>>>>>>> input format, which means that the relation between the other side of
>>>>>>>> the link (resizer source) and the video node is implicit and needs to
>>>>>>>> be handled internally in the driver.
>>>>>>>> - V4L2, as opposed to MC, requires that the current state is always
>>>>>>>> consistent and valid. In this case it means that an ioctl on the video
>>>>>>>> node must not be able to alter the state in a way that would make it
>>>>>>>> inconsistent with the state of the matching resizer source.
> 

Re: [RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad

[Laurent Pinchart]

Hi Dafna,

On Mon, Jun 29, 2020 at 02:29:39PM +0200, Dafna Hirschfeld wrote:
> On 29.06.20 14:21, Laurent Pinchart wrote:
> > On Mon, Jun 29, 2020 at 02:08:39PM +0200, Dafna Hirschfeld wrote:
> >> On 28.06.20 23:37, Laurent Pinchart wrote:
> >>> On Sun, Jun 28, 2020 at 11:33:30PM +0200, Tomasz Figa wrote:
> >>>> On Sun, Jun 28, 2020 at 10:27 PM Laurent Pinchart wrote:
> >>>>> On Sun, Jun 28, 2020 at 04:59:34PM +0200, Tomasz Figa wrote:
> >>>>>> On Sun, Jun 28, 2020 at 3:15 PM Laurent Pinchart wrote:
> >>>>>>> On Sun, Jun 28, 2020 at 03:10:41PM +0200, Tomasz Figa wrote:
> >>>>>>>> On Fri, Jun 26, 2020 at 2:15 AM Helen Koike <helen.koike@collabora.com> wrote:
> >>>>>>>>> On 6/18/20 4:25 PM, Tomasz Figa wrote:
> >>>>>>>>>> On Thu, Jun 18, 2020 at 08:54:34PM +0200, Dafna Hirschfeld wrote:
> >>>>>>>>>>> On 18.06.20 20:49, Tomasz Figa wrote:
> >>>>>>>>>>>> On Thu, Jun 18, 2020 at 08:26:33PM +0200, Dafna Hirschfeld wrote:
> >>>>>>>>>>>>> On 18.06.20 19:27, Tomasz Figa wrote:
> >>>>>>>>>>>>>> On Wed, Jun 10, 2020 at 2:08 PM Tomasz Figa wrote:
> >>>>>>>>>>>>>>> On Thu, Jun 4, 2020 at 9:11 PM Dafna Hirschfeld wrote:
> >>>>>>>>>>>>>>>> On 04.06.20 19:54, Tomasz Figa wrote:
> >>>>>>>>>>>>>>>>> On Thu, Apr 16, 2020 at 04:56:05PM +0200, Dafna Hirschfeld wrote:
> >>>>>>>>>>>>>>>>>> The isp entity has a hardware support to force full range quantization
> >>>>>>>>>>>>>>>>>> for YUV formats. Use the new API to indicate userspace that
> >>>>>>>>>>>>>>>>>> quantization conversion is supported by adding the flag
> >>>>>>>>>>>>>>>>>> V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION during media code enumeration.
> >>>>>>>>>>>>>>>>>> Then uppon s_fmt on the video source pad, we assign the
> >>>>>>>>>>>>>>>>>> quantization from userspace for YUV formats.
> >>>>>>>>>>>>>>>>>> Also in the capture and resizer entities we retrieve the colorspace
> >>>>>>>>>>>>>>>>>> from the isp entity.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
> >>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>       drivers/staging/media/rkisp1/rkisp1-capture.c | 28 ++++++-------
> >>>>>>>>>>>>>>>>>>       drivers/staging/media/rkisp1/rkisp1-common.h  |  2 +
> >>>>>>>>>>>>>>>>>>       drivers/staging/media/rkisp1/rkisp1-isp.c     | 39 +++++++++++++++++--
> >>>>>>>>>>>>>>>>>>       drivers/staging/media/rkisp1/rkisp1-resizer.c | 13 +++++++
> >>>>>>>>>>>>>>>>>>       4 files changed, 65 insertions(+), 17 deletions(-)
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c
> >>>>>>>>>>>>>>>>>> index fbf62399fe3d..aca0f93bc772 100644
> >>>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-capture.c
> >>>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-capture.c
> >>>>>>>>>>>>>>>>>> @@ -1066,14 +1066,13 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
> >>>>>>>>>>>>>>>>>>                             const struct v4l2_format_info **fmt_info)
> >>>>>>>>>>>>>>>>>>       {
> >>>>>>>>>>>>>>>>>>          const struct rkisp1_capture_config *config = cap->config;
> >>>>>>>>>>>>>>>>>> -    struct rkisp1_capture *other_cap =
> >>>>>>>>>>>>>>>>>> -                    &cap->rkisp1->capture_devs[cap->id ^ 1];
> >>>>>>>>>>>>>>>>>>          const struct rkisp1_capture_fmt_cfg *fmt;
> >>>>>>>>>>>>>>>>>>          const struct v4l2_format_info *info;
> >>>>>>>>>>>>>>>>>>          const unsigned int max_widths[] = { RKISP1_RSZ_MP_SRC_MAX_WIDTH,
> >>>>>>>>>>>>>>>>>>                                              RKISP1_RSZ_SP_SRC_MAX_WIDTH };
> >>>>>>>>>>>>>>>>>>          const unsigned int max_heights[] = { RKISP1_RSZ_MP_SRC_MAX_HEIGHT,
> >>>>>>>>>>>>>>>>>>                                               RKISP1_RSZ_SP_SRC_MAX_HEIGHT};
> >>>>>>>>>>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>          fmt = rkisp1_find_fmt_cfg(cap, pixm->pixelformat);
> >>>>>>>>>>>>>>>>>>          if (!fmt) {
> >>>>>>>>>>>>>>>>>> @@ -1081,24 +1080,27 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
> >>>>>>>>>>>>>>>>>>                  pixm->pixelformat = fmt->fourcc;
> >>>>>>>>>>>>>>>>>>          }
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> +    rkisp1_get_isp_src_fmt(cap->rkisp1, &isp_sd_fmt);
> >>>>>>>>>>>>>>>>>> +    pixm->field = isp_sd_fmt.format.field;
> >>>>>>>>>>>>>>>>>> +    pixm->colorspace = isp_sd_fmt.format.colorspace;
> >>>>>>>>>>>>>>>>>> +    pixm->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
> >>>>>>>>>>>>>>>>>> +    pixm->xfer_func = isp_sd_fmt.format.xfer_func;
> >>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>> +    /*
> >>>>>>>>>>>>>>>>>> +     * isp has a feature to set full range quantization for yuv formats.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> How about "select between limited and full range for YUV formats"?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> +     * so we need to get the format from the isp.
> >>>>>>>>>>>>>>>>>> +     */
> >>>>>>>>>>>>>>>>>> +    pixm->quantization = isp_sd_fmt.format.quantization;
> >>>>>>>>>>>>>>>>>> +    if (!v4l2_is_format_yuv(cap->pix.info))
> >>>>>>>>>>>>>>>>>> +            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>>          pixm->width = clamp_t(u32, pixm->width,
> >>>>>>>>>>>>>>>>>>                                RKISP1_RSZ_SRC_MIN_WIDTH, max_widths[cap->id]);
> >>>>>>>>>>>>>>>>>>          pixm->height = clamp_t(u32, pixm->height,
> >>>>>>>>>>>>>>>>>>                                 RKISP1_RSZ_SRC_MIN_HEIGHT, max_heights[cap->id]);
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> -    pixm->field = V4L2_FIELD_NONE;
> >>>>>>>>>>>>>>>>>> -    pixm->colorspace = V4L2_COLORSPACE_DEFAULT;
> >>>>>>>>>>>>>>>>>> -    pixm->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> >>>>>>>>>>>>>>>>>> -
> >>>>>>>>>>>>>>>>>>          info = rkisp1_fill_pixfmt(pixm, cap->id);
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> -    /* can not change quantization when stream-on */
> >>>>>>>>>>>>>>>>>> -    if (other_cap->is_streaming)
> >>>>>>>>>>>>>>>>>> -            pixm->quantization = other_cap->pix.fmt.quantization;
> >>>>>>>>>>>>>>>>>> -    /* output full range by default, take effect in params */
> >>>>>>>>>>>>>>>>>> -    else if (!pixm->quantization ||
> >>>>>>>>>>>>>>>>>> -             pixm->quantization > V4L2_QUANTIZATION_LIM_RANGE)
> >>>>>>>>>>>>>>>>>> -            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>          if (fmt_cfg)
> >>>>>>>>>>>>>>>>>>                  *fmt_cfg = fmt;
> >>>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-common.h b/drivers/staging/media/rkisp1/rkisp1-common.h
> >>>>>>>>>>>>>>>>>> index 2d7b7e078636..7a5576fa14c9 100644
> >>>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-common.h
> >>>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-common.h
> >>>>>>>>>>>>>>>>>> @@ -300,6 +300,8 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
> >>>>>>>>>>>>>>>>>>                          struct v4l2_device *v4l2_dev);
> >>>>>>>>>>>>>>>>>>       void rkisp1_isp_unregister(struct rkisp1_device *rkisp1);
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
> >>>>>>>>>>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt);
> >>>>>>>>>>>>>>>>>>       const struct rkisp1_isp_mbus_info *rkisp1_isp_mbus_info_get(u32 mbus_code);
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>       irqreturn_t rkisp1_isp_isr(struct rkisp1_device *rkisp1);
> >>>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-isp.c b/drivers/staging/media/rkisp1/rkisp1-isp.c
> >>>>>>>>>>>>>>>>>> index dee8e96f3900..6fdf5ed0b6b1 100644
> >>>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-isp.c
> >>>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-isp.c
> >>>>>>>>>>>>>>>>>> @@ -613,6 +613,10 @@ static int rkisp1_isp_enum_mbus_code(struct v4l2_subdev *sd,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>                  if (code->index == pos - 1) {
> >>>>>>>>>>>>>>>>>>                          code->code = fmt->mbus_code;
> >>>>>>>>>>>>>>>>>> +                    if (fmt->pixel_enc == V4L2_PIXEL_ENC_YUV &&
> >>>>>>>>>>>>>>>>>> +                        dir == RKISP1_DIR_SRC)
> >>>>>>>>>>>>>>>>>> +                            code->flags =
> >>>>>>>>>>>>>>>>>> +                                    V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION;
> >>>>>>>>>>>>>>>>>>                          return 0;
> >>>>>>>>>>>>>>>>>>                  }
> >>>>>>>>>>>>>>>>>>          }
> >>>>>>>>>>>>>>>>>> @@ -639,12 +643,21 @@ static int rkisp1_isp_init_config(struct v4l2_subdev *sd,
> >>>>>>>>>>>>>>>>>>          sink_crop->height = RKISP1_DEFAULT_HEIGHT;
> >>>>>>>>>>>>>>>>>>          sink_crop->left = 0;
> >>>>>>>>>>>>>>>>>>          sink_crop->top = 0;
> >>>>>>>>>>>>>>>>>> +    sink_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >>>>>>>>>>>>>>>>>> +    sink_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(sink_fmt->colorspace);
> >>>>>>>>>>>>>>>>>> +    sink_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(sink_fmt->colorspace);
> >>>>>>>>>>>>>>>>>> +    sink_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>          src_fmt = v4l2_subdev_get_try_format(sd, cfg,
> >>>>>>>>>>>>>>>>>>                                               RKISP1_ISP_PAD_SOURCE_VIDEO);
> >>>>>>>>>>>>>>>>>>          *src_fmt = *sink_fmt;
> >>>>>>>>>>>>>>>>>>          src_fmt->code = RKISP1_DEF_SRC_PAD_FMT;
> >>>>>>>>>>>>>>>>>> -    src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >>>>>>>>>>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>>>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>>>>>>>>>>> +    src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
> >>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>          src_crop = v4l2_subdev_get_try_crop(sd, cfg,
> >>>>>>>>>>>>>>>>>>                                              RKISP1_ISP_PAD_SOURCE_VIDEO);
> >>>>>>>>>>>>>>>>>> @@ -687,10 +700,17 @@ static void rkisp1_isp_set_src_fmt(struct rkisp1_isp *isp,
> >>>>>>>>>>>>>>>>>>                  isp->src_fmt = mbus_info;
> >>>>>>>>>>>>>>>>>>          src_fmt->width  = src_crop->width;
> >>>>>>>>>>>>>>>>>>          src_fmt->height = src_crop->height;
> >>>>>>>>>>>>>>>>>> -    src_fmt->quantization = format->quantization;
> >>>>>>>>>>>>>>>>>> -    /* full range by default */
> >>>>>>>>>>>>>>>>>> -    if (!src_fmt->quantization)
> >>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
> >>>>>>>>>>>>>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>>>>>>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
> >>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>> +    if (mbus_info->pixel_enc == V4L2_PIXEL_ENC_BAYER)
> >>>>>>>>>>>>>>>>>>                  src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
> >>>>>>>>>>>>>>>>>> +    else if (format->quantization == V4L2_QUANTIZATION_DEFAULT)
> >>>>>>>>>>>>>>>>>> +            src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
> >>>>>>>>>>>>>>>>>> +    else
> >>>>>>>>>>>>>>>>>> +            src_fmt->quantization = format->quantization;
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>          *format = *src_fmt;
> >>>>>>>>>>>>>>>>>>       }
> >>>>>>>>>>>>>>>>>> @@ -1068,6 +1088,17 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
> >>>>>>>>>>>>>>>>>>          return ret;
> >>>>>>>>>>>>>>>>>>       }
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
> >>>>>>>>>>>>>>>>>> +                       struct v4l2_subdev_format *sd_fmt)
> >>>>>>>>>>>>>>>>>> +{
> >>>>>>>>>>>>>>>>>> +    struct rkisp1_isp *isp = &rkisp1->isp;
> >>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>> +    sd_fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
> >>>>>>>>>>>>>>>>>> +    sd_fmt->pad = RKISP1_ISP_PAD_SOURCE_VIDEO;
> >>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>> +    return v4l2_subdev_call(&isp->sd, pad, get_fmt, NULL, sd_fmt);
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Do we need to get through the external API to access data that is
> >>>>>>>>>>>>>>>>> driver-internal anyway?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> +}
> >>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>>       void rkisp1_isp_unregister(struct rkisp1_device *rkisp1)
> >>>>>>>>>>>>>>>>>>       {
> >>>>>>>>>>>>>>>>>>          struct v4l2_subdev *sd = &rkisp1->isp.sd;
> >>>>>>>>>>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-resizer.c b/drivers/staging/media/rkisp1/rkisp1-resizer.c
> >>>>>>>>>>>>>>>>>> index 7b6b7ddd4169..8705b133de68 100644
> >>>>>>>>>>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-resizer.c
> >>>>>>>>>>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-resizer.c
> >>>>>>>>>>>>>>>>>> @@ -525,6 +525,7 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
> >>>>>>>>>>>>>>>>>>          const struct rkisp1_isp_mbus_info *mbus_info;
> >>>>>>>>>>>>>>>>>>          struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
> >>>>>>>>>>>>>>>>>>          struct v4l2_rect *sink_crop;
> >>>>>>>>>>>>>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>          sink_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SINK, which);
> >>>>>>>>>>>>>>>>>>          src_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SRC, which);
> >>>>>>>>>>>>>>>>>> @@ -539,8 +540,20 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
> >>>>>>>>>>>>>>>>>>          if (which == V4L2_SUBDEV_FORMAT_ACTIVE)
> >>>>>>>>>>>>>>>>>>                  rsz->pixel_enc = mbus_info->pixel_enc;
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> +    rkisp1_get_isp_src_fmt(rsz->rkisp1, &isp_sd_fmt);
> >>>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Is this necessary? My understanding was that in the subdev model, it was
> >>>>>>>>>>>>>>>>> the userspace responsibility to propagate any configuration changes through
> >>>>>>>>>>>>>>>>> the graph.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Also, doing this only here wouldn't fully maintain the
> >>>>>>>>>>>>>>>>> consistency of the state. For example, if one sets the ISP subdev format
> >>>>>>>>>>>>>>>>> first, then the resizer subdev and then the ISP subdev again, wouldn't the
> >>>>>>>>>>>>>>>>> resizer subdev end up with a wrong format?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> yes, this is indeed a bug, I am preparing v4 now.
> >>>>>>>>>>>>>>>> What I thought to do is adding quantization conversion
> >>>>>>>>>>>>>>>> support also on ther resizer and capture entities.
> >>>>>>>>>>>>>>>> Then in the 'link_validation' callbacks, there
> >>>>>>>>>>>>>>>> is a validation that the quantization fields matches.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> My understanding is that, if we have the following topology
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> [ ISP ] -> [ Resizer ] -> [ Video node ]
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> then the ISP source pad would have the csc capability, while resizer
> >>>>>>>>>>>>>>> and video node would just accept whatever is configured on their sink
> >>>>>>>>>>>>>>> pads (no need for csc capability for that) and propagate that to their
> >>>>>>>>>>>>>>> outputs, i.e. resizer source pad and video node CAPTURE format.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Is this what you were going to implement?
> >>>>>>>>>>>>> Hi, I sent a v4 where the CSC capability is set on the reiszer and capture as well.
> >>>>>>>>>>>>> I can send a v5 implementing it the way you suggest. Currently the doc says that the colorspace fields
> >>>>>>>>>>>>> must be set by the driver for capture streams. This implies that userspace can set it
> >>>>>>>>>>>>> only if the CSC is supported.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Why would the userspace have to set it for the capture stream on the
> >>>>>>>>>>>> resizer and video nodes? Couldn't the userspace set it to DEFAULT and
> >>>>>>>>>>>> then the driver override to whatever it received on the corresponding
> >>>>>>>>>>>> sink?
> >>>>>>>>>
> >>>>>>>>> According to the documentation [1]:
> >>>>>>>>>
> >>>>>>>>> "the default quantization encoding as defined by the colorspace"
> >>>>>>>>>
> >>>>>>>>> Then, for instance, the default for V4L2_COLORSPACE_SRGB is limited range [2].
> >>>>>>>>>
> >>>>>>>>> Unless we change the definition of DEFAULT to let it accept whatever it receives on the corresponding sink.
> >>>>>>>>>
> >>>>>>>>> Then the way I understand is:
> >>>>>>>>>
> >>>>>>>>> [sensor]->[isp]->[resizer]->[capture]
> >>>>>>>>>
> >>>>>>>>> Userspace should set quantization on:
> >>>>>>>>> * sensor source pad
> >>>>>>>>> * isp sink pad
> >>>>>>>>> * resizer sink pad
> >>>>>>>>> * capture node
> >>>>>>>>>
> >>>>>>>>> The remaining source pads would get values propagated from their sink pads.
> >>>>>>>>>
> >>>>>>>>> [1] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-defs.html#c.v4l2_quantization
> >>>>>>>>> [2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-details.html#col-srgb
> >>>>>>>>
> >>>>>>>> After discussing this offline, we concluded that:
> >>>>>>>> - only the source pad of the ISP entity should expose the CSC capability,
> >>>>>>>> - the CAPTURE video node should expose only the formats/color spaces
> >>>>>>>> compatible with the current setting of the resizer source,
> >>>>>>>> - consistency between the resizer source format and CAPTURE format
> >>>>>>>> should be implicitly maintained by the driver.
> >>>>>>>
> >>>>>>> Do you mean it should be validated at streamon time ?
> >>>>>>
> >>>>>> Nope. Any V4L2 state must be kept valid all the time, as mandated by
> >>>>>> the general V4L2 principles. Wrong configuration of the video node
> >>>>>> must not be a reason for STREAMON failing.
> >>>>>
> >>>>> That's a wrong interpretation of V4L2. On a video-node-centric device (a
> >>>>> device that doesn't use the subdev userspace API for configuration) this
> >>>>> is correct. In a MC-centric device, the video node and its connected
> >>>>> subdev are configured separately.
> >>>>
> >>>> I don't see this documented anywhere in the specification, maybe
> >>>> except a mention of EPIPE or ENOLINK being returned from STREAMON, but
> >>>> they explicitly refer to pipeline configuration and not a conflict
> >>>> between V4L2 and MC state.
> >>>
> >>> We're suffering from a lack of documentation in this area :-( Niklas has
> >>> authored
> >>>
> >>> commit f645e6256bd1b12523b759fcc610861fb21c24c7
> >>> Author: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
> >>> Date:   Tue Apr 21 15:57:38 2020 +0200
> >>>
> >>>       media: v4l2-dev/ioctl: Add V4L2_CAP_IO_MC
> >>>
> >>> which is a first step, now we need to document the behaviour of all
> >>> ioctls properly when the MC and !MC cases differ.
> >>>
> >>>> The behavior as you're suggesting would be at least confusing for the
> >>>> userspace, because it breaks one of the core principles of V4L2 and
> >>>> also there is even no facility to allow the userspace handle this kind
> >>>> of behavior, e.g. how to know which memory formats are supported for
> >>>> given mbus formats of the related sink.
> >>>
> >>> commit e5b6b07a1b45dd9d19bec1fa1d60750b0fcf2fb0
> >>> Author: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> >>> Date:   Fri Apr 24 15:43:31 2020 +0200
> >>>
> >>>       media: v4l2: Extend VIDIOC_ENUM_FMT to support MC-centric devices
> >>
> >> Hi,
> >> What if the video device entity wants to support V4L2_CAP_IO_MC and has two
> >> pads registered to the media controller?
> >> It seems to be undefined in the documentation.
> > 
> > What kind of video device would that be ?
> 
> I don't know, it is more a theoretical question.

Video nodes are supposed to have a single pad in an MC graph. I think we
need an exception to that rule for mem-to-mem devices though, and then
we should update the documentation to support that explicitly. For now,
and for the majority of devices, we should be fine with a single pad.

> >>> Been missing for a long time... I realized that when I started using the
> >>> API in a real userspace implementation instead of just writing test
> >>> tools or very device-specific code that could hardcode all this
> >>> knowledge.
> >>>
> >>>>>>> In MC-based
> >>>>>>> drivers, entities are isolated from each other from a configuration
> >>>>>>> point of view, and the validity of the configuration across the whole
> >>>>>>> pipeline is checked at streamon time. I don't think CSC parameters
> >>>>>>> should be treated differently than formats in this regard.
> >>>>>>
> >>>>>> Indeed, the ISP entity and the resizer entity are isolated from each
> >>>>>> other and they need to be validated separately. However, the resizer
> >>>>>> entity is not isolated from the video node, as there is no
> >>>>>> configuration exposed for the video node in the MC API. If MC exposed
> >>>>>> an explicit format control for the video interface sink pad, then the
> >>>>>> resizer would have to be validated separately indeed, but still that
> >>>>>> video interface sink pad would be tied to the rest of the
> >>>>>> configuration of the video node (e.g. the CAPTURE format).
> >>>>>>
> >>>>>>>> Below is the rationale for that:
> >>>>>>>> - In the MC model, the video node represents the DMA alone (and thus
> >>>>>>>> the reason to have the resizer entities at all) and since the
> >>>>>>>> colorspace is not a property of the DMA, it doesn't belong there.
> >>>>>>>> - Even if we put that aside, both MP and SP can only output the same
> >>>>>>>> colorspace, which would create a dependency between the configuration
> >>>>>>>> of the two video nodes, which is not something well defined by V4L2.
> >>>>>>>> - A video interface in the MC API doesn't expose control over its
> >>>>>>>> input format, which means that the relation between the other side of
> >>>>>>>> the link (resizer source) and the video node is implicit and needs to
> >>>>>>>> be handled internally in the driver.
> >>>>>>>> - V4L2, as opposed to MC, requires that the current state is always
> >>>>>>>> consistent and valid. In this case it means that an ioctl on the video
> >>>>>>>> node must not be able to alter the state in a way that would make it
> >>>>>>>> inconsistent with the state of the matching resizer source.

-- 
Regards,

Laurent Pinchart

Re: [RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad

[Helen Koike]

On 6/18/20 3:26 PM, Dafna Hirschfeld wrote:
> 
> 
> On 18.06.20 19:27, Tomasz Figa wrote:
>> On Wed, Jun 10, 2020 at 2:08 PM Tomasz Figa <tfiga@chromium.org> wrote:
>>>
>>> On Thu, Jun 4, 2020 at 9:11 PM Dafna Hirschfeld
>>> <dafna.hirschfeld@collabora.com> wrote:
>>>>
>>>> Hi
>>>>
>>>> On 04.06.20 19:54, Tomasz Figa wrote:
>>>>> On Thu, Apr 16, 2020 at 04:56:05PM +0200, Dafna Hirschfeld wrote:
>>>>>> The isp entity has a hardware support to force full range quantization
>>>>>> for YUV formats. Use the new API to indicate userspace that
>>>>>> quantization conversion is supported by adding the flag
>>>>>> V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION during media code enumeration.
>>>>>> Then uppon s_fmt on the video source pad, we assign the
>>>>>> quantization from userspace for YUV formats.
>>>>>> Also in the capture and resizer entities we retrieve the colorspace
>>>>>> from the isp entity.
>>>>>>
>>>>>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
>>>>>> ---
>>>>>>    drivers/staging/media/rkisp1/rkisp1-capture.c | 28 ++++++-------
>>>>>>    drivers/staging/media/rkisp1/rkisp1-common.h  |  2 +
>>>>>>    drivers/staging/media/rkisp1/rkisp1-isp.c     | 39 +++++++++++++++++--
>>>>>>    drivers/staging/media/rkisp1/rkisp1-resizer.c | 13 +++++++
>>>>>>    4 files changed, 65 insertions(+), 17 deletions(-)
>>>>>>
>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>> index fbf62399fe3d..aca0f93bc772 100644
>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>> @@ -1066,14 +1066,13 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
>>>>>>                          const struct v4l2_format_info **fmt_info)
>>>>>>    {
>>>>>>       const struct rkisp1_capture_config *config = cap->config;
>>>>>> -    struct rkisp1_capture *other_cap =
>>>>>> -                    &cap->rkisp1->capture_devs[cap->id ^ 1];
>>>>>>       const struct rkisp1_capture_fmt_cfg *fmt;
>>>>>>       const struct v4l2_format_info *info;
>>>>>>       const unsigned int max_widths[] = { RKISP1_RSZ_MP_SRC_MAX_WIDTH,
>>>>>>                                           RKISP1_RSZ_SP_SRC_MAX_WIDTH };
>>>>>>       const unsigned int max_heights[] = { RKISP1_RSZ_MP_SRC_MAX_HEIGHT,
>>>>>>                                            RKISP1_RSZ_SP_SRC_MAX_HEIGHT};
>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
>>>>>>
>>>>>>       fmt = rkisp1_find_fmt_cfg(cap, pixm->pixelformat);
>>>>>>       if (!fmt) {
>>>>>> @@ -1081,24 +1080,27 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
>>>>>>               pixm->pixelformat = fmt->fourcc;
>>>>>>       }
>>>>>>
>>>>>> +    rkisp1_get_isp_src_fmt(cap->rkisp1, &isp_sd_fmt);
>>>>>> +    pixm->field = isp_sd_fmt.format.field;
>>>>>> +    pixm->colorspace = isp_sd_fmt.format.colorspace;
>>>>>> +    pixm->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
>>>>>> +    pixm->xfer_func = isp_sd_fmt.format.xfer_func;
>>>>>> +
>>>>>> +    /*
>>>>>> +     * isp has a feature to set full range quantization for yuv formats.
>>>>>
>>>>> How about "select between limited and full range for YUV formats"?
>>>>>
>>>>>> +     * so we need to get the format from the isp.
>>>>>> +     */
>>>>>> +    pixm->quantization = isp_sd_fmt.format.quantization;
>>>>>> +    if (!v4l2_is_format_yuv(cap->pix.info))
>>>>>> +            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>> +
>>>>>>       pixm->width = clamp_t(u32, pixm->width,
>>>>>>                             RKISP1_RSZ_SRC_MIN_WIDTH, max_widths[cap->id]);
>>>>>>       pixm->height = clamp_t(u32, pixm->height,
>>>>>>                              RKISP1_RSZ_SRC_MIN_HEIGHT, max_heights[cap->id]);
>>>>>>
>>>>>> -    pixm->field = V4L2_FIELD_NONE;
>>>>>> -    pixm->colorspace = V4L2_COLORSPACE_DEFAULT;
>>>>>> -    pixm->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>>>>>> -
>>>>>>       info = rkisp1_fill_pixfmt(pixm, cap->id);
>>>>>>
>>>>>> -    /* can not change quantization when stream-on */
>>>>>> -    if (other_cap->is_streaming)
>>>>>> -            pixm->quantization = other_cap->pix.fmt.quantization;
>>>>>> -    /* output full range by default, take effect in params */
>>>>>> -    else if (!pixm->quantization ||
>>>>>> -             pixm->quantization > V4L2_QUANTIZATION_LIM_RANGE)
>>>>>> -            pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>
>>>>>>       if (fmt_cfg)
>>>>>>               *fmt_cfg = fmt;
>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-common.h b/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>> index 2d7b7e078636..7a5576fa14c9 100644
>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>> @@ -300,6 +300,8 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
>>>>>>                       struct v4l2_device *v4l2_dev);
>>>>>>    void rkisp1_isp_unregister(struct rkisp1_device *rkisp1);
>>>>>>
>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
>>>>>> +                       struct v4l2_subdev_format *sd_fmt);
>>>>>>    const struct rkisp1_isp_mbus_info *rkisp1_isp_mbus_info_get(u32 mbus_code);
>>>>>>
>>>>>>    irqreturn_t rkisp1_isp_isr(struct rkisp1_device *rkisp1);
>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-isp.c b/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>> index dee8e96f3900..6fdf5ed0b6b1 100644
>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>> @@ -613,6 +613,10 @@ static int rkisp1_isp_enum_mbus_code(struct v4l2_subdev *sd,
>>>>>>
>>>>>>               if (code->index == pos - 1) {
>>>>>>                       code->code = fmt->mbus_code;
>>>>>> +                    if (fmt->pixel_enc == V4L2_PIXEL_ENC_YUV &&
>>>>>> +                        dir == RKISP1_DIR_SRC)
>>>>>> +                            code->flags =
>>>>>> +                                    V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION;
>>>>>>                       return 0;
>>>>>>               }
>>>>>>       }
>>>>>> @@ -639,12 +643,21 @@ static int rkisp1_isp_init_config(struct v4l2_subdev *sd,
>>>>>>       sink_crop->height = RKISP1_DEFAULT_HEIGHT;
>>>>>>       sink_crop->left = 0;
>>>>>>       sink_crop->top = 0;
>>>>>> +    sink_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>> +    sink_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(sink_fmt->colorspace);
>>>>>> +    sink_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(sink_fmt->colorspace);
>>>>>> +    sink_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>> +
>>>>>>
>>>>>>       src_fmt = v4l2_subdev_get_try_format(sd, cfg,
>>>>>>                                            RKISP1_ISP_PAD_SOURCE_VIDEO);
>>>>>>       *src_fmt = *sink_fmt;
>>>>>>       src_fmt->code = RKISP1_DEF_SRC_PAD_FMT;
>>>>>> -    src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
>>>>>> +    src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
>>>>>> +
>>>>>>
>>>>>>       src_crop = v4l2_subdev_get_try_crop(sd, cfg,
>>>>>>                                           RKISP1_ISP_PAD_SOURCE_VIDEO);
>>>>>> @@ -687,10 +700,17 @@ static void rkisp1_isp_set_src_fmt(struct rkisp1_isp *isp,
>>>>>>               isp->src_fmt = mbus_info;
>>>>>>       src_fmt->width  = src_crop->width;
>>>>>>       src_fmt->height = src_crop->height;
>>>>>> -    src_fmt->quantization = format->quantization;
>>>>>> -    /* full range by default */
>>>>>> -    if (!src_fmt->quantization)
>>>>>> +
>>>>>> +    src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>> +    src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
>>>>>> +    src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
>>>>>> +
>>>>>> +    if (mbus_info->pixel_enc == V4L2_PIXEL_ENC_BAYER)
>>>>>>               src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>> +    else if (format->quantization == V4L2_QUANTIZATION_DEFAULT)
>>>>>> +            src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
>>>>>> +    else
>>>>>> +            src_fmt->quantization = format->quantization;
>>>>>>
>>>>>>       *format = *src_fmt;
>>>>>>    }
>>>>>> @@ -1068,6 +1088,17 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
>>>>>>       return ret;
>>>>>>    }
>>>>>>
>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
>>>>>> +                       struct v4l2_subdev_format *sd_fmt)
>>>>>> +{
>>>>>> +    struct rkisp1_isp *isp = &rkisp1->isp;
>>>>>> +
>>>>>> +    sd_fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
>>>>>> +    sd_fmt->pad = RKISP1_ISP_PAD_SOURCE_VIDEO;
>>>>>> +
>>>>>> +    return v4l2_subdev_call(&isp->sd, pad, get_fmt, NULL, sd_fmt);
>>>>>
>>>>> Do we need to get through the external API to access data that is
>>>>> driver-internal anyway?
>>>>>
>>>>>> +}
>>>>>> +
>>>>>>    void rkisp1_isp_unregister(struct rkisp1_device *rkisp1)
>>>>>>    {
>>>>>>       struct v4l2_subdev *sd = &rkisp1->isp.sd;
>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-resizer.c b/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>> index 7b6b7ddd4169..8705b133de68 100644
>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>> @@ -525,6 +525,7 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
>>>>>>       const struct rkisp1_isp_mbus_info *mbus_info;
>>>>>>       struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
>>>>>>       struct v4l2_rect *sink_crop;
>>>>>> +    struct v4l2_subdev_format isp_sd_fmt;
>>>>>>
>>>>>>       sink_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SINK, which);
>>>>>>       src_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SRC, which);
>>>>>> @@ -539,8 +540,20 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
>>>>>>       if (which == V4L2_SUBDEV_FORMAT_ACTIVE)
>>>>>>               rsz->pixel_enc = mbus_info->pixel_enc;
>>>>>>
>>>>>> +    rkisp1_get_isp_src_fmt(rsz->rkisp1, &isp_sd_fmt);
>>>>>> +
>>>>>
>>>>> Is this necessary? My understanding was that in the subdev model, it was
>>>>> the userspace responsibility to propagate any configuration changes through
>>>>> the graph.
>>>>>
>>>>> Also, doing this only here wouldn't fully maintain the
>>>>> consistency of the state. For example, if one sets the ISP subdev format
>>>>> first, then the resizer subdev and then the ISP subdev again, wouldn't the
>>>>> resizer subdev end up with a wrong format?
>>>>
>>>> yes, this is indeed a bug, I am preparing v4 now.
>>>> What I thought to do is adding quantization conversion
>>>> support also on ther resizer and capture entities.
>>>> Then in the 'link_validation' callbacks, there
>>>> is a validation that the quantization fields matches.
>>>
>>> My understanding is that, if we have the following topology
>>>
>>> [ ISP ] -> [ Resizer ] -> [ Video node ]
>>>
>>> then the ISP source pad would have the csc capability, while resizer
>>> and video node would just accept whatever is configured on their sink
>>> pads (no need for csc capability for that) and propagate that to their
>>> outputs, i.e. resizer source pad and video node CAPTURE format.
>>>
>>> Is this what you were going to implement?
> Hi, I sent a v4 where the CSC capability is set on the reiszer and capture as well.
> I can send a v5 implementing it the way you suggest. Currently the doc says that the colorspace fields
> must be set by the driver for capture streams. This implies that userspace can set it
> only if the CSC is supported.
> 
> Another option would be to remove the CSC API for subdevices and to decide that the quantiztion is a feature
> of the capture device.

If we chose to configure in a single point, I would choose configuring in the capture.

But then we are propagating configurations between entities, I'm not sure if this is the v4l2 way,
since we don't propagate pixelformats or image resolutions.

Regards,
Helen

> 
> Thanks,
> Dafna
>>
>> Gentle ping.
>>
>> Best regards,
>> Tomasz
>>

Re: [RFC v3 1/2] v4l2: add support for colorspace conversion for video capture

[Hans Verkuil]

On 16/04/2020 16:56, Dafna Hirschfeld wrote:
> From: Philipp Zabel <p.zabel@pengutronix.de>
> 
> For video capture it is the driver that reports the colorspace,
> Y'CbCr/HSV encoding, quantization range and transfer function
> used by the video, and there is no way to request something
> different, even though many HDTV receivers have some sort of
> colorspace conversion capabilities.
> 
> For output video this feature already exists since the application
> specifies this information for the video format it will send out, and
> the transmitter will enable any available CSC if a format conversion has
> to be performed in order to match the capabilities of the sink.
> 
> For video capture we propose adding new pix_format flag:
> V4L2_PIX_FMT_FLAG_HAS_CSC. The flag is set by the application,

That should be V4L2_PIX_FMT_FLAG_REQUEST_CSC, since the application actively
requests the use of the CSC.

> the driver will interpret the ycbcr_enc/hsv_enc, and quantization fields
> as the requested colorspace information and will attempt to
> do the conversion it supports.
> 
> Drivers set the flags
> V4L2_FMT_FLAG_CSC_YCBCR_ENC,
> V4L2_FMT_FLAG_CSC_HSV_ENC,
> V4L2_FMT_FLAG_CSC_HSV_QUANTIZATION,

That last define should be 'V4L2_FMT_FLAG_CSC_QUANTIZATION' without 'HSV_'.

> in the flags field of the struct v4l2_fmtdesc during enumeration to
> indicate that they support colorspace conversion for the respective field.
> Currently the conversion of the fields colorspace, xfer_func is not
> supported since there are no drivers that support it.
> 
> The same API is added for the subdevices. With the flag V4L2_MBUS_FRAMEFMT_HAS_CSC

Should also be REQUEST_CSC.

> set by the application in the VIDIOC_SUBDEV_S_FMT ioctl and the flags
> V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC, V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION
> set by the driver in the VIDIOC_SUBDEV_ENUM_MBUS_CODE ioctl.
> 
> For subdevices, new 'flags' fields were added to the structs
> v4l2_subdev_mbus_code_enum, v4l2_mbus_framefmt which are borrowed from the 'reserved' field
> 
> Drivers do not have to actually look at the flags: if the flags are not
> set, then the colorspace, ycbcr_enc and quantization fields are set to
> the default values by the core, i.e. just pass on the received format
> without conversion.
> 
> Signed-off-by: Hans Verkuil <Hans Verkuil@cisco.com>
> Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
> ---
> This is v3 of the RFC suggested originaly by Hans Verkuil:
> 
>  https://patchwork.linuxtv.org/patch/28847/
> 
> And then a v2 from Philipp Zabel:
> 
>  https://patchwork.kernel.org/project/linux-media/list/?series=15483
> 
> changes in v3:
> I added the API to subdevices as well and added fixes according to
> comments from Hans.
> I also added a usecase for the new API for the rkisp1 driver.
> 
> changes in v2 (reported by Philipp Zabel):
>  - convert to rst
>  - split V4L2_PIX_FMT_FLAG_REQUEST_CSC into four separate flags for
>    colorspace, ycbcr_enc/hsv_enc, quantization, and xfer_func
>  - let driver set flags to indicate supported features
> 
> [1] https://patchwork.linuxtv.org/patch/28847/
> 
>  .../media/v4l/pixfmt-reserved.rst             |  6 +++
>  .../media/v4l/pixfmt-v4l2-mplane.rst          | 16 ++-----
>  .../userspace-api/media/v4l/pixfmt-v4l2.rst   | 36 +++++++++++++---
>  .../media/v4l/subdev-formats.rst              | 42 +++++++++++++++++--
>  .../media/v4l/vidioc-enum-fmt.rst             | 18 ++++++++
>  .../v4l/vidioc-subdev-enum-mbus-code.rst      | 23 ++++++++++
>  drivers/media/v4l2-core/v4l2-ioctl.c          | 19 ++++++++-
>  drivers/media/v4l2-core/v4l2-subdev.c         |  7 ++++
>  include/uapi/linux/v4l2-mediabus.h            |  5 ++-
>  include/uapi/linux/v4l2-subdev.h              |  5 ++-
>  include/uapi/linux/videodev2.h                |  4 ++
>  11 files changed, 158 insertions(+), 23 deletions(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst b/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst
> index 59b9e7238f90..fa8dada69f8c 100644
> --- a/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst
> +++ b/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst

Why on earth did the documentation of these pixel format flags end up in
pixfmt-reserved.rst?

Can you make a patch that moves this to pixfmt-v4l2.rst?

> @@ -280,3 +280,9 @@ please make a proposal on the linux-media mailing list.
>  	by RGBA values (128, 192, 255, 128), the same pixel described with
>  	premultiplied colors would be described by RGBA values (64, 96,
>  	128, 128)
> +    * - ``V4L2_PIX_FMT_FLAG_HAS_CSC``
> +      - 0x00000002
> +      - Set by the application. It is only used for capture and is
> +	ignored for output streams. If set, then request the driver to do
> +	colorspace conversion from the received colorspace, only conversions
> +	of Ycbcr encoding, HSV encoding and quantization are supported.

I'd replace the part "If set..." with this:

----------
If set, then request the driver to do colorspace conversion from the received
colorspace to the requested colorspace values. If a colorimetry field
(``colorspace``, ``xfer_func``, ``ycncr_enc`` or ``quantization``) is set to 0,
then that colorimetry setting will remain unchanged from what was received.
So to just change the quantization only the ``quantization`` field shall be set
to a non-zero value (``V4L2_QUANTIZATION_FULL_RANGE`` or
``V4L2_QUANTIZATION_LIM_RANGE``) and all other colorimetry fields shall be set to 0.

To check which conversions are supported by the hardware for the current pixel
format, see :ref:`fmtdesc-flags`.
----------


> diff --git a/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst b/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst
> index 444b4082684c..66f3365d7b72 100644
> --- a/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst
> +++ b/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst
> @@ -105,29 +105,21 @@ describing all planes of that format.
>      * - __u8
>        - ``ycbcr_enc``
>        - Y'CbCr encoding, from enum :c:type:`v4l2_ycbcr_encoding`.
> -        This information supplements the ``colorspace`` and must be set by
> -	the driver for capture streams and by the application for output
> -	streams, see :ref:`colorspaces`.
> +	See struct :c:type:`v4l2_pix_format`.
>      * - __u8
>        - ``hsv_enc``
>        - HSV encoding, from enum :c:type:`v4l2_hsv_encoding`.
> -        This information supplements the ``colorspace`` and must be set by
> -	the driver for capture streams and by the application for output
> -	streams, see :ref:`colorspaces`.
> +	See struct :c:type:`v4l2_pix_format`.
>      * - }
>        -
>      * - __u8
>        - ``quantization``
>        - Quantization range, from enum :c:type:`v4l2_quantization`.
> -        This information supplements the ``colorspace`` and must be set by
> -	the driver for capture streams and by the application for output
> -	streams, see :ref:`colorspaces`.
> +	See struct :c:type:`v4l2_pix_format`.
>      * - __u8
>        - ``xfer_func``
>        - Transfer function, from enum :c:type:`v4l2_xfer_func`.
> -        This information supplements the ``colorspace`` and must be set by
> -	the driver for capture streams and by the application for output
> -	streams, see :ref:`colorspaces`.
> +	See struct :c:type:`v4l2_pix_format`.
>      * - __u8
>        - ``reserved[7]``
>        - Reserved for future extensions. Should be zeroed by drivers and
> diff --git a/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst b/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst
> index 759420a872d6..ce57718cd66b 100644
> --- a/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst
> +++ b/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst
> @@ -110,8 +110,8 @@ Single-planar format structure
>        - ``colorspace``
>        - Image colorspace, from enum :c:type:`v4l2_colorspace`.
>          This information supplements the ``pixelformat`` and must be set
> -	by the driver for capture streams and by the application for
> -	output streams, see :ref:`colorspaces`.
> +	by the driver for capture streams and by the application for output
> +	streams, see :ref:`colorspace`.

This doesn't appear to change anything.

>      * - __u32
>        - ``priv``
>        - This field indicates whether the remaining fields of the
> @@ -148,13 +148,31 @@ Single-planar format structure
>        - Y'CbCr encoding, from enum :c:type:`v4l2_ycbcr_encoding`.
>          This information supplements the ``colorspace`` and must be set by
>  	the driver for capture streams and by the application for output
> -	streams, see :ref:`colorspaces`.
> +	streams, see :ref:`colorspaces`. If the application sets the
> +	flag ``V4L2_PIX_FMT_FLAG_HAS_CSC`` then the application can set
> +	this field for a capture stream to request a specific Y'CbCr encoding
> +	for the captured image data. The driver will attempt to do the
> +	conversion to the specified Y'CbCr encoding or return the encoding it
> +	will use if it can't do the conversion. This field is ignored for
> +	non-Y'CbCr pixelformats. The driver indicates that ycbcr_enc conversion
> +	is supported by setting the flag V4L2_FMT_FLAG_CSC_YCBCR_ENC on the
> +	on the corresponding struct :c:type:`v4l2_fmtdesc` during enumeration.

on the on the -> in the

> +	See :ref:`fmtdesc-flags`
>      * - __u32
>        - ``hsv_enc``
>        - HSV encoding, from enum :c:type:`v4l2_hsv_encoding`.
>          This information supplements the ``colorspace`` and must be set by
>  	the driver for capture streams and by the application for output
> -	streams, see :ref:`colorspaces`.
> +	streams, see :ref:`colorspaces`. If the application sets the flag
> +	``V4L2_PIX_FMT_FLAG_HAS_CSC`` then the application can set this
> +	field for a capture stream to request a specific HSV encoding for the
> +	captured image data. The driver will attempt to do the conversion to
> +	the specified HSV encoding or return the encoding it will use if it
> +	can't do the conversion. This field is ignored for non-HSV
> +	pixelformats. The driver indicates that hsv_enc conversion
> +	is supported by setting the flag V4L2_FMT_FLAG_CSC_HSV_ENC on the
> +	on the corresponding struct :c:type:`v4l2_fmtdesc` during enumeration.

Ditto.

> +	See :ref:`fmtdesc-flags`
>      * - }
>        -
>      * - __u32
> @@ -162,7 +180,15 @@ Single-planar format structure
>        - Quantization range, from enum :c:type:`v4l2_quantization`.
>          This information supplements the ``colorspace`` and must be set by
>  	the driver for capture streams and by the application for output
> -	streams, see :ref:`colorspaces`.
> +	streams, see :ref:`colorspaces`. If the application sets the flag
> +	``V4L2_PIX_FMT_FLAG_HAS_CSC`` then the application can set
> +	this field for a capture stream to request a specific quantization
> +	range for the captured image data. The driver will attempt to do the
> +	conversion to the specified quantization range or return the
> +	quantization it will use if it can't do the conversion.  The driver
> +	indicates that quantization conversion is supported by setting the flag
> +	V4L2_FMT_FLAG_CSC_QUANTIZATION on the on the corresponding struct

Ditto

> +	:c:type:`v4l2_fmtdesc` during enumeration. See :ref:`fmtdesc-flags`
>      * - __u32
>        - ``xfer_func``
>        - Transfer function, from enum :c:type:`v4l2_xfer_func`.
> diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> index 9a4d61b0d76f..f1d4ca29a3e8 100644
> --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
> +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> @@ -49,13 +49,33 @@ Media Bus Formats
>        - Y'CbCr encoding, from enum :c:type:`v4l2_ycbcr_encoding`.
>          This information supplements the ``colorspace`` and must be set by
>  	the driver for capture streams and by the application for output
> -	streams, see :ref:`colorspaces`.
> +	streams, see :ref:`colorspaces`. If the application sets the
> +	flag ``V4L2_MBUS_FRAMEFMT_HAS_CSC`` the the application can set
> +	this field for a capture stream to request a specific Y'CbCr encoding
> +	for the mbus data. The driver will attempt to do the

mbus -> media bus

> +	conversion to the specified Y'CbCr encoding or return the encoding it
> +	will use if it can't do the conversion. This field is ignored for
> +	non-Y'CbCr pixelformats. The driver indicates that ycbcr_enc conversion

This is a media bus format, not a pixelformat.

> +	is supported by setting the flag V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC on

on -> in

> +	the corresponding struct c:type:`v4l2_subdev_mbus_code_enum` during
> +	enumeration. See :ref:`vidioc-subdev-enum-mbus-code.rst <v4l2-subdev-mbus-code-flags>`
> +
>      * - __u16
>        - ``quantization``
>        - Quantization range, from enum :c:type:`v4l2_quantization`.
>          This information supplements the ``colorspace`` and must be set by
>  	the driver for capture streams and by the application for output
> -	streams, see :ref:`colorspaces`.
> +	streams, see :ref:`colorspaces`. If the application sets the
> +	flag ``V4L2_MBUS_FRAMEFMT_HAS_CSC`` the the application can set
> +	this field for a capture stream to request a specific quantization
> +	encoding for the mbus data. The driver will attempt to do the

mbus -> media bus

> +	conversion to the specified quantization or return the quantization it
> +	will use if it can't do the conversion. The driver indicates that
> +	quantization conversion is supported by setting the flag
> +	V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION on the corresponding struct

on -> in

> +	c:type:`v4l2_subdev_mbus_code_enum` during enumeration.
> +	See :ref:`vidioc-subdev-enum-mbus-code.rst <v4l2-subdev-mbus-code-flags>`
> +
>      * - __u16
>        - ``xfer_func``
>        - Transfer function, from enum :c:type:`v4l2_xfer_func`.
> @@ -63,10 +83,26 @@ Media Bus Formats
>  	the driver for capture streams and by the application for output
>  	streams, see :ref:`colorspaces`.
>      * - __u16
> -      - ``reserved``\ [11]
> +      - ``flags``
> +      - flags see:  :ref:v4l2-mbus-framefmt-flags

flags see: -> See:

> +    * - __u16
> +      - ``reserved``\ [10]
>        - Reserved for future extensions. Applications and drivers must set
>  	the array to zero.
>  
> +.. _v4l2-mbus-framefmt-flags:
> +
> +.. flat-table:: v4l2_mbus_framefmt Flags
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths:       3 1 4
> +
> +    * - ``V4L2_MBUS_FRAMEFMT_HAS_CSC``
> +      - 0x0001
> +      - Set by the application. It is only used for capture and is
> +	ignored for output streams. If set, then request the subdevice to do
> +	colorspace conversion from the received colorspace, only conversions
> +	of Ycbcr encoding, and quantization are supported.

Replace by a similar text as for V4L2_PIX_FMT_FLAG_HAS_CSC.

>  
>  
>  .. _v4l2-mbus-pixelcode:
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> index 7e3142e11d77..125f074543af 100644
> --- a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> +++ b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> @@ -145,6 +145,24 @@ formats in preference order, where preferred formats are returned before
>  	parameters are detected. This flag can only be used in combination
>  	with the ``V4L2_FMT_FLAG_COMPRESSED`` flag, since this applies to
>  	compressed formats only. It is also only applies to stateful codecs.
> +    * - ``V4L2_FMT_FLAG_CSC_YCBCR_ENC``
> +      - 0x0010
> +      - The driver allows the application to try to change the default
> +	ycbcr encoding. This flag is relevant only for capture devices.
> +	The application can ask to configure the ycbcr_enc of the capture device
> +	when calling the :c:func:`VIDIOC_S_FMT` ioctl.

add: ...ioctl with ``V4L2_FMT_FLAG_REQUEST_CSC`` set. (this should probably be
a reference to the REQUEST_CSC flag documentation).

> +    * - ``V4L2_FMT_FLAG_CSC_HSV_ENC``
> +      - 0x0010
> +      - The driver allows the application to try to change the default
> +	hsv encoding. This flag is relevant only for capture devices.
> +	The application can ask to configure the hsv_enc of the capture device
> +	when calling the :c:func:`VIDIOC_S_FMT` ioctl.

Ditto. Also: hsv -> HSV.

> +    * - ``V4L2_FMT_FLAG_CSC_QUANTIZATION``
> +      - 0x0020
> +      - The driver allows the application to try to change the default
> +	quantization. This flag is relevant only for capture devices.
> +	The application can ask to configure the quantization of the capture device
> +	when calling the :c:func:`VIDIOC_S_FMT` ioctl.

Ditto.

>  
>  
>  Return Value
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
> index 35b8607203a4..4ad87cb74f57 100644
> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
> @@ -78,11 +78,34 @@ information about the try formats.
>        - ``which``
>        - Media bus format codes to be enumerated, from enum
>  	:ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
> +    * - __u32
> +      - ``flags``
> +      - see :ref:`v4l2-subdev-mbus-code-flags`

see -> See

>      * - __u32
>        - ``reserved``\ [8]
>        - Reserved for future extensions. Applications and drivers must set
>  	the array to zero.
>  
> +.. _v4l2-subdev-mbus-code-flags:
> +
> +
> +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{7.7cm}|
> +
> +.. flat-table:: flags in struct v4l2_subdev_mbus_code_enum
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths:       1 1 2
> +
> +    * - V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC
> +      - 0x00000001
> +      - The driver supports setting the ycbcr encoding by the application
> +	when calling the VIDIOC_SUBDEV_S_FMT ioctl. see :ref:`v4l2-mbus-format`

see -> See

> +	on how to do this.
> +    * - V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION
> +      - 0x00000002
> +      - The driver supports setting the quantization by the application
> +	when calling the VIDIOC_SUBDEV_S_FMT ioctl. see :ref:`v4l2-mbus-format`

see -> See

> +	on how to do this.
>  
>  Return Value
>  ============
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index b2ef8e60ea7d..3c7ffb6b15cb 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -1029,6 +1029,15 @@ static void v4l_sanitize_format(struct v4l2_format *fmt)
>  		fmt->fmt.pix_mp.num_planes = min_t(u32, fmt->fmt.pix_mp.num_planes,
>  					       VIDEO_MAX_PLANES);
>  
> +	if (fmt->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE) {
> +		if (fmt->fmt.pix_mp.flags & V4L2_PIX_FMT_FLAG_HAS_CSC)
> +			return;
> +		fmt->fmt.pix_mp.colorspace = V4L2_COLORSPACE_DEFAULT;
> +		fmt->fmt.pix_mp.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> +		fmt->fmt.pix_mp.quantization = V4L2_QUANTIZATION_DEFAULT;
> +		fmt->fmt.pix_mp.xfer_func = V4L2_XFER_FUNC_DEFAULT;
> +	}
> +
>  	/*
>  	 * The v4l2_pix_format structure has been extended with fields that were
>  	 * not previously required to be set to zero by applications. The priv
> @@ -1043,8 +1052,16 @@ static void v4l_sanitize_format(struct v4l2_format *fmt)
>  	    fmt->type != V4L2_BUF_TYPE_VIDEO_OUTPUT)
>  		return;
>  
> -	if (fmt->fmt.pix.priv == V4L2_PIX_FMT_PRIV_MAGIC)
> +	if (fmt->fmt.pix.priv == V4L2_PIX_FMT_PRIV_MAGIC) {
> +		if (fmt->type != V4L2_BUF_TYPE_VIDEO_CAPTURE ||
> +		    fmt->fmt.pix.flags & V4L2_PIX_FMT_FLAG_HAS_CSC)
> +			return;
> +		fmt->fmt.pix.colorspace = V4L2_COLORSPACE_DEFAULT;
> +		fmt->fmt.pix.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> +		fmt->fmt.pix.quantization = V4L2_QUANTIZATION_DEFAULT;
> +		fmt->fmt.pix.xfer_func = V4L2_XFER_FUNC_DEFAULT;
>  		return;
> +	}
>  
>  	fmt->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>  
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index a376b351135f..51e009936aad 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -477,6 +477,13 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  	case VIDIOC_SUBDEV_S_FMT: {
>  		struct v4l2_subdev_format *format = arg;
>  
> +		if (!(format->format.flags & V4L2_MBUS_FRAMEFMT_HAS_CSC)) {
> +			format->format.colorspace = V4L2_COLORSPACE_DEFAULT;
> +			format->format.xfer_func = V4L2_XFER_FUNC_DEFAULT;
> +			format->format.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> +			format->format.quantization = V4L2_QUANTIZATION_DEFAULT;
> +		}
> +
>  		memset(format->reserved, 0, sizeof(format->reserved));
>  		memset(format->format.reserved, 0, sizeof(format->format.reserved));
>  		return v4l2_subdev_call(sd, pad, set_fmt, subdev_fh->pad, format);
> diff --git a/include/uapi/linux/v4l2-mediabus.h b/include/uapi/linux/v4l2-mediabus.h
> index 123a231001a8..89ff0ad6a631 100644
> --- a/include/uapi/linux/v4l2-mediabus.h
> +++ b/include/uapi/linux/v4l2-mediabus.h
> @@ -16,6 +16,8 @@
>  #include <linux/types.h>
>  #include <linux/videodev2.h>
>  
> +#define V4L2_MBUS_FRAMEFMT_HAS_CSC	0x0001
> +
>  /**
>   * struct v4l2_mbus_framefmt - frame format on the media bus
>   * @width:	image width
> @@ -36,7 +38,8 @@ struct v4l2_mbus_framefmt {
>  	__u16			ycbcr_enc;
>  	__u16			quantization;
>  	__u16			xfer_func;
> -	__u16			reserved[11];
> +	__u16			flags;
> +	__u16			reserved[10];
>  };
>  
>  #ifndef __KERNEL__
> diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
> index 03970ce30741..4410b26a7158 100644
> --- a/include/uapi/linux/v4l2-subdev.h
> +++ b/include/uapi/linux/v4l2-subdev.h
> @@ -65,6 +65,8 @@ struct v4l2_subdev_crop {
>  	__u32 reserved[8];
>  };
>  
> +#define V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC	0x00000001

I think it is better to add an HSV_ENC alias here as well.

It's for readability only, since talking about YCBCR_ENC when the mediabus format
is a HSV format is just weird.

> +#define V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION	0x00000002
>  /**
>   * struct v4l2_subdev_mbus_code_enum - Media bus format enumeration
>   * @pad: pad number, as reported by the media API
> @@ -77,7 +79,8 @@ struct v4l2_subdev_mbus_code_enum {
>  	__u32 index;
>  	__u32 code;
>  	__u32 which;
> -	__u32 reserved[8];
> +	__u32 flags;
> +	__u32 reserved[7];
>  };
>  
>  /**
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index 9817b7e2c968..adc9dd1080b8 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -772,6 +772,7 @@ struct v4l2_pix_format {
>  
>  /* Flags */
>  #define V4L2_PIX_FMT_FLAG_PREMUL_ALPHA	0x00000001
> +#define V4L2_PIX_FMT_FLAG_HAS_CSC	0x00000002
>  
>  /*
>   *	F O R M A T   E N U M E R A T I O N
> @@ -789,6 +790,9 @@ struct v4l2_fmtdesc {
>  #define V4L2_FMT_FLAG_EMULATED			0x0002
>  #define V4L2_FMT_FLAG_CONTINUOUS_BYTESTREAM	0x0004
>  #define V4L2_FMT_FLAG_DYN_RESOLUTION		0x0008
> +#define V4L2_FMT_FLAG_CSC_YCBCR_ENC		0x0010
> +#define V4L2_FMT_FLAG_CSC_HSV_ENC		0x0010
> +#define V4L2_FMT_FLAG_CSC_QUANTIZATION		0x0020
>  
>  	/* Frame Size and frame rate enumeration */
>  /*
> 

I think this is quite good, it's mainly doc improvements and renaming HAS_CSC to REQUEST_CSC.

That said, I do want to see this supported in at least vivid (vimc would be nice
too!). And v4l2-ctl needs a patch to support this.

A compliance test for v4l2-compliance would be nice too, but it is not a prerequisite.
I'm not entirely sure what would be the best way to test this. Perhaps a vivid-specific
test might be easiest (i.e., don't test it for all drivers, but just for vivid).

Regards,

	Hans

Re: [RFC v3 1/2] v4l2: add support for colorspace conversion for video capture

[Dafna Hirschfeld]

Hi

On 07.05.20 15:00, Hans Verkuil wrote:
> On 16/04/2020 16:56, Dafna Hirschfeld wrote:
>> From: Philipp Zabel <p.zabel@pengutronix.de>
>>
>> For video capture it is the driver that reports the colorspace,
>> Y'CbCr/HSV encoding, quantization range and transfer function
>> used by the video, and there is no way to request something
>> different, even though many HDTV receivers have some sort of
>> colorspace conversion capabilities.
>>
>> For output video this feature already exists since the application
>> specifies this information for the video format it will send out, and
>> the transmitter will enable any available CSC if a format conversion has
>> to be performed in order to match the capabilities of the sink.
>>
>> For video capture we propose adding new pix_format flag:
>> V4L2_PIX_FMT_FLAG_HAS_CSC. The flag is set by the application,
> 
> That should be V4L2_PIX_FMT_FLAG_REQUEST_CSC, since the application actively
> requests the use of the CSC.
Maybe change to V4L2_PIX_FMT_FLAG_ASK_FOR_CSC so people won't think
it is related to the request API ?
> 
>> the driver will interpret the ycbcr_enc/hsv_enc, and quantization fields
>> as the requested colorspace information and will attempt to
>> do the conversion it supports.
>>
>> Drivers set the flags
>> V4L2_FMT_FLAG_CSC_YCBCR_ENC,
>> V4L2_FMT_FLAG_CSC_HSV_ENC,
>> V4L2_FMT_FLAG_CSC_HSV_QUANTIZATION,
> 
> That last define should be 'V4L2_FMT_FLAG_CSC_QUANTIZATION' without 'HSV_'.
> 
>> in the flags field of the struct v4l2_fmtdesc during enumeration to
>> indicate that they support colorspace conversion for the respective field.
>> Currently the conversion of the fields colorspace, xfer_func is not
>> supported since there are no drivers that support it.
>>
>> The same API is added for the subdevices. With the flag V4L2_MBUS_FRAMEFMT_HAS_CSC
> 
> Should also be REQUEST_CSC.
> 
>> set by the application in the VIDIOC_SUBDEV_S_FMT ioctl and the flags
>> V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC, V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION
>> set by the driver in the VIDIOC_SUBDEV_ENUM_MBUS_CODE ioctl.
>>
>> For subdevices, new 'flags' fields were added to the structs
>> v4l2_subdev_mbus_code_enum, v4l2_mbus_framefmt which are borrowed from the 'reserved' field
>>
>> Drivers do not have to actually look at the flags: if the flags are not
>> set, then the colorspace, ycbcr_enc and quantization fields are set to
>> the default values by the core, i.e. just pass on the received format
>> without conversion.
>>
>> Signed-off-by: Hans Verkuil <Hans Verkuil@cisco.com>
>> Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
>> ---
>> This is v3 of the RFC suggested originaly by Hans Verkuil:
>>
>>   https://patchwork.linuxtv.org/patch/28847/
>>
>> And then a v2 from Philipp Zabel:
>>
>>   https://patchwork.kernel.org/project/linux-media/list/?series=15483
>>
>> changes in v3:
>> I added the API to subdevices as well and added fixes according to
>> comments from Hans.
>> I also added a usecase for the new API for the rkisp1 driver.
>>
>> changes in v2 (reported by Philipp Zabel):
>>   - convert to rst
>>   - split V4L2_PIX_FMT_FLAG_REQUEST_CSC into four separate flags for
>>     colorspace, ycbcr_enc/hsv_enc, quantization, and xfer_func
>>   - let driver set flags to indicate supported features
>>
>> [1] https://patchwork.linuxtv.org/patch/28847/
>>
>>   .../media/v4l/pixfmt-reserved.rst             |  6 +++
>>   .../media/v4l/pixfmt-v4l2-mplane.rst          | 16 ++-----
>>   .../userspace-api/media/v4l/pixfmt-v4l2.rst   | 36 +++++++++++++---
>>   .../media/v4l/subdev-formats.rst              | 42 +++++++++++++++++--
>>   .../media/v4l/vidioc-enum-fmt.rst             | 18 ++++++++
>>   .../v4l/vidioc-subdev-enum-mbus-code.rst      | 23 ++++++++++
>>   drivers/media/v4l2-core/v4l2-ioctl.c          | 19 ++++++++-
>>   drivers/media/v4l2-core/v4l2-subdev.c         |  7 ++++
>>   include/uapi/linux/v4l2-mediabus.h            |  5 ++-
>>   include/uapi/linux/v4l2-subdev.h              |  5 ++-
>>   include/uapi/linux/videodev2.h                |  4 ++
>>   11 files changed, 158 insertions(+), 23 deletions(-)
>>
>> diff --git a/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst b/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst
>> index 59b9e7238f90..fa8dada69f8c 100644
>> --- a/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst
>> +++ b/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst
> 
> Why on earth did the documentation of these pixel format flags end up in
> pixfmt-reserved.rst?
> 
> Can you make a patch that moves this to pixfmt-v4l2.rst?
> 
>> @@ -280,3 +280,9 @@ please make a proposal on the linux-media mailing list.
>>   	by RGBA values (128, 192, 255, 128), the same pixel described with
>>   	premultiplied colors would be described by RGBA values (64, 96,
>>   	128, 128)
>> +    * - ``V4L2_PIX_FMT_FLAG_HAS_CSC``
>> +      - 0x00000002
>> +      - Set by the application. It is only used for capture and is
>> +	ignored for output streams. If set, then request the driver to do
>> +	colorspace conversion from the received colorspace, only conversions
>> +	of Ycbcr encoding, HSV encoding and quantization are supported.
> 
> I'd replace the part "If set..." with this:
> 
> ----------
> If set, then request the driver to do colorspace conversion from the received
> colorspace to the requested colorspace values. If a colorimetry field
> (``colorspace``, ``xfer_func``, ``ycncr_enc`` or ``quantization``) is set to 0,
> then that colorimetry setting will remain unchanged from what was received.
> So to just change the quantization only the ``quantization`` field shall be set
> to a non-zero value (``V4L2_QUANTIZATION_FULL_RANGE`` or
> ``V4L2_QUANTIZATION_LIM_RANGE``) and all other colorimetry fields shall be set to 0.
> 
> To check which conversions are supported by the hardware for the current pixel
> format, see :ref:`fmtdesc-flags`.
> ----------
> 
> 
>> diff --git a/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst b/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst
>> index 444b4082684c..66f3365d7b72 100644
>> --- a/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst
>> +++ b/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst
>> @@ -105,29 +105,21 @@ describing all planes of that format.
>>       * - __u8
>>         - ``ycbcr_enc``
>>         - Y'CbCr encoding, from enum :c:type:`v4l2_ycbcr_encoding`.
>> -        This information supplements the ``colorspace`` and must be set by
>> -	the driver for capture streams and by the application for output
>> -	streams, see :ref:`colorspaces`.
>> +	See struct :c:type:`v4l2_pix_format`.
>>       * - __u8
>>         - ``hsv_enc``
>>         - HSV encoding, from enum :c:type:`v4l2_hsv_encoding`.
>> -        This information supplements the ``colorspace`` and must be set by
>> -	the driver for capture streams and by the application for output
>> -	streams, see :ref:`colorspaces`.
>> +	See struct :c:type:`v4l2_pix_format`.
>>       * - }
>>         -
>>       * - __u8
>>         - ``quantization``
>>         - Quantization range, from enum :c:type:`v4l2_quantization`.
>> -        This information supplements the ``colorspace`` and must be set by
>> -	the driver for capture streams and by the application for output
>> -	streams, see :ref:`colorspaces`.
>> +	See struct :c:type:`v4l2_pix_format`.
>>       * - __u8
>>         - ``xfer_func``
>>         - Transfer function, from enum :c:type:`v4l2_xfer_func`.
>> -        This information supplements the ``colorspace`` and must be set by
>> -	the driver for capture streams and by the application for output
>> -	streams, see :ref:`colorspaces`.
>> +	See struct :c:type:`v4l2_pix_format`.
>>       * - __u8
>>         - ``reserved[7]``
>>         - Reserved for future extensions. Should be zeroed by drivers and
>> diff --git a/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst b/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst
>> index 759420a872d6..ce57718cd66b 100644
>> --- a/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst
>> +++ b/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst
>> @@ -110,8 +110,8 @@ Single-planar format structure
>>         - ``colorspace``
>>         - Image colorspace, from enum :c:type:`v4l2_colorspace`.
>>           This information supplements the ``pixelformat`` and must be set
>> -	by the driver for capture streams and by the application for
>> -	output streams, see :ref:`colorspaces`.
>> +	by the driver for capture streams and by the application for output
>> +	streams, see :ref:`colorspace`.
> 
> This doesn't appear to change anything.
> 
>>       * - __u32
>>         - ``priv``
>>         - This field indicates whether the remaining fields of the
>> @@ -148,13 +148,31 @@ Single-planar format structure
>>         - Y'CbCr encoding, from enum :c:type:`v4l2_ycbcr_encoding`.
>>           This information supplements the ``colorspace`` and must be set by
>>   	the driver for capture streams and by the application for output
>> -	streams, see :ref:`colorspaces`.
>> +	streams, see :ref:`colorspaces`. If the application sets the
>> +	flag ``V4L2_PIX_FMT_FLAG_HAS_CSC`` then the application can set
>> +	this field for a capture stream to request a specific Y'CbCr encoding
>> +	for the captured image data. The driver will attempt to do the
>> +	conversion to the specified Y'CbCr encoding or return the encoding it
>> +	will use if it can't do the conversion. This field is ignored for
>> +	non-Y'CbCr pixelformats. The driver indicates that ycbcr_enc conversion
>> +	is supported by setting the flag V4L2_FMT_FLAG_CSC_YCBCR_ENC on the
>> +	on the corresponding struct :c:type:`v4l2_fmtdesc` during enumeration.
> 
> on the on the -> in the
> 
>> +	See :ref:`fmtdesc-flags`
>>       * - __u32
>>         - ``hsv_enc``
>>         - HSV encoding, from enum :c:type:`v4l2_hsv_encoding`.
>>           This information supplements the ``colorspace`` and must be set by
>>   	the driver for capture streams and by the application for output
>> -	streams, see :ref:`colorspaces`.
>> +	streams, see :ref:`colorspaces`. If the application sets the flag
>> +	``V4L2_PIX_FMT_FLAG_HAS_CSC`` then the application can set this
>> +	field for a capture stream to request a specific HSV encoding for the
>> +	captured image data. The driver will attempt to do the conversion to
>> +	the specified HSV encoding or return the encoding it will use if it
>> +	can't do the conversion. This field is ignored for non-HSV
>> +	pixelformats. The driver indicates that hsv_enc conversion
>> +	is supported by setting the flag V4L2_FMT_FLAG_CSC_HSV_ENC on the
>> +	on the corresponding struct :c:type:`v4l2_fmtdesc` during enumeration.
> 
> Ditto.
> 
>> +	See :ref:`fmtdesc-flags`
>>       * - }
>>         -
>>       * - __u32
>> @@ -162,7 +180,15 @@ Single-planar format structure
>>         - Quantization range, from enum :c:type:`v4l2_quantization`.
>>           This information supplements the ``colorspace`` and must be set by
>>   	the driver for capture streams and by the application for output
>> -	streams, see :ref:`colorspaces`.
>> +	streams, see :ref:`colorspaces`. If the application sets the flag
>> +	``V4L2_PIX_FMT_FLAG_HAS_CSC`` then the application can set
>> +	this field for a capture stream to request a specific quantization
>> +	range for the captured image data. The driver will attempt to do the
>> +	conversion to the specified quantization range or return the
>> +	quantization it will use if it can't do the conversion.  The driver
>> +	indicates that quantization conversion is supported by setting the flag
>> +	V4L2_FMT_FLAG_CSC_QUANTIZATION on the on the corresponding struct
> 
> Ditto
> 
>> +	:c:type:`v4l2_fmtdesc` during enumeration. See :ref:`fmtdesc-flags`
>>       * - __u32
>>         - ``xfer_func``
>>         - Transfer function, from enum :c:type:`v4l2_xfer_func`.
>> diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
>> index 9a4d61b0d76f..f1d4ca29a3e8 100644
>> --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
>> +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
>> @@ -49,13 +49,33 @@ Media Bus Formats
>>         - Y'CbCr encoding, from enum :c:type:`v4l2_ycbcr_encoding`.
>>           This information supplements the ``colorspace`` and must be set by
>>   	the driver for capture streams and by the application for output
>> -	streams, see :ref:`colorspaces`.
>> +	streams, see :ref:`colorspaces`. If the application sets the
>> +	flag ``V4L2_MBUS_FRAMEFMT_HAS_CSC`` the the application can set
>> +	this field for a capture stream to request a specific Y'CbCr encoding
>> +	for the mbus data. The driver will attempt to do the
> 
> mbus -> media bus
> 
>> +	conversion to the specified Y'CbCr encoding or return the encoding it
>> +	will use if it can't do the conversion. This field is ignored for
>> +	non-Y'CbCr pixelformats. The driver indicates that ycbcr_enc conversion
> 
> This is a media bus format, not a pixelformat.
> 
>> +	is supported by setting the flag V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC on
> 
> on -> in
> 
>> +	the corresponding struct c:type:`v4l2_subdev_mbus_code_enum` during
>> +	enumeration. See :ref:`vidioc-subdev-enum-mbus-code.rst <v4l2-subdev-mbus-code-flags>`
>> +
>>       * - __u16
>>         - ``quantization``
>>         - Quantization range, from enum :c:type:`v4l2_quantization`.
>>           This information supplements the ``colorspace`` and must be set by
>>   	the driver for capture streams and by the application for output
>> -	streams, see :ref:`colorspaces`.
>> +	streams, see :ref:`colorspaces`. If the application sets the
>> +	flag ``V4L2_MBUS_FRAMEFMT_HAS_CSC`` the the application can set
>> +	this field for a capture stream to request a specific quantization
>> +	encoding for the mbus data. The driver will attempt to do the
> 
> mbus -> media bus
> 
>> +	conversion to the specified quantization or return the quantization it
>> +	will use if it can't do the conversion. The driver indicates that
>> +	quantization conversion is supported by setting the flag
>> +	V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION on the corresponding struct
> 
> on -> in
> 
>> +	c:type:`v4l2_subdev_mbus_code_enum` during enumeration.
>> +	See :ref:`vidioc-subdev-enum-mbus-code.rst <v4l2-subdev-mbus-code-flags>`
>> +
>>       * - __u16
>>         - ``xfer_func``
>>         - Transfer function, from enum :c:type:`v4l2_xfer_func`.
>> @@ -63,10 +83,26 @@ Media Bus Formats
>>   	the driver for capture streams and by the application for output
>>   	streams, see :ref:`colorspaces`.
>>       * - __u16
>> -      - ``reserved``\ [11]
>> +      - ``flags``
>> +      - flags see:  :ref:v4l2-mbus-framefmt-flags
> 
> flags see: -> See:
> 
>> +    * - __u16
>> +      - ``reserved``\ [10]
>>         - Reserved for future extensions. Applications and drivers must set
>>   	the array to zero.
>>   
>> +.. _v4l2-mbus-framefmt-flags:
>> +
>> +.. flat-table:: v4l2_mbus_framefmt Flags
>> +    :header-rows:  0
>> +    :stub-columns: 0
>> +    :widths:       3 1 4
>> +
>> +    * - ``V4L2_MBUS_FRAMEFMT_HAS_CSC``
>> +      - 0x0001
>> +      - Set by the application. It is only used for capture and is
>> +	ignored for output streams. If set, then request the subdevice to do
>> +	colorspace conversion from the received colorspace, only conversions
>> +	of Ycbcr encoding, and quantization are supported.
> 
> Replace by a similar text as for V4L2_PIX_FMT_FLAG_HAS_CSC.
> 
>>   
>>   
>>   .. _v4l2-mbus-pixelcode:
>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
>> index 7e3142e11d77..125f074543af 100644
>> --- a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
>> +++ b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
>> @@ -145,6 +145,24 @@ formats in preference order, where preferred formats are returned before
>>   	parameters are detected. This flag can only be used in combination
>>   	with the ``V4L2_FMT_FLAG_COMPRESSED`` flag, since this applies to
>>   	compressed formats only. It is also only applies to stateful codecs.
>> +    * - ``V4L2_FMT_FLAG_CSC_YCBCR_ENC``
>> +      - 0x0010
>> +      - The driver allows the application to try to change the default
>> +	ycbcr encoding. This flag is relevant only for capture devices.
>> +	The application can ask to configure the ycbcr_enc of the capture device
>> +	when calling the :c:func:`VIDIOC_S_FMT` ioctl.
> 
> add: ...ioctl with ``V4L2_FMT_FLAG_REQUEST_CSC`` set. (this should probably be
> a reference to the REQUEST_CSC flag documentation).
> 
>> +    * - ``V4L2_FMT_FLAG_CSC_HSV_ENC``
>> +      - 0x0010
>> +      - The driver allows the application to try to change the default
>> +	hsv encoding. This flag is relevant only for capture devices.
>> +	The application can ask to configure the hsv_enc of the capture device
>> +	when calling the :c:func:`VIDIOC_S_FMT` ioctl.
> 
> Ditto. Also: hsv -> HSV.
> 
>> +    * - ``V4L2_FMT_FLAG_CSC_QUANTIZATION``
>> +      - 0x0020
>> +      - The driver allows the application to try to change the default
>> +	quantization. This flag is relevant only for capture devices.
>> +	The application can ask to configure the quantization of the capture device
>> +	when calling the :c:func:`VIDIOC_S_FMT` ioctl.
> 
> Ditto.
> 
>>   
>>   
>>   Return Value
>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
>> index 35b8607203a4..4ad87cb74f57 100644
>> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
>> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
>> @@ -78,11 +78,34 @@ information about the try formats.
>>         - ``which``
>>         - Media bus format codes to be enumerated, from enum
>>   	:ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
>> +    * - __u32
>> +      - ``flags``
>> +      - see :ref:`v4l2-subdev-mbus-code-flags`
> 
> see -> See
> 
>>       * - __u32
>>         - ``reserved``\ [8]
>>         - Reserved for future extensions. Applications and drivers must set
>>   	the array to zero.
>>   
>> +.. _v4l2-subdev-mbus-code-flags:
>> +
>> +
>> +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{7.7cm}|
>> +
>> +.. flat-table:: flags in struct v4l2_subdev_mbus_code_enum
>> +    :header-rows:  0
>> +    :stub-columns: 0
>> +    :widths:       1 1 2
>> +
>> +    * - V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC
>> +      - 0x00000001
>> +      - The driver supports setting the ycbcr encoding by the application
>> +	when calling the VIDIOC_SUBDEV_S_FMT ioctl. see :ref:`v4l2-mbus-format`
> 
> see -> See
> 
>> +	on how to do this.
>> +    * - V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION
>> +      - 0x00000002
>> +      - The driver supports setting the quantization by the application
>> +	when calling the VIDIOC_SUBDEV_S_FMT ioctl. see :ref:`v4l2-mbus-format`
> 
> see -> See
> 
>> +	on how to do this.
>>   
>>   Return Value
>>   ============
>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
>> index b2ef8e60ea7d..3c7ffb6b15cb 100644
>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
>> @@ -1029,6 +1029,15 @@ static void v4l_sanitize_format(struct v4l2_format *fmt)
>>   		fmt->fmt.pix_mp.num_planes = min_t(u32, fmt->fmt.pix_mp.num_planes,
>>   					       VIDEO_MAX_PLANES);
>>   
>> +	if (fmt->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE) {
>> +		if (fmt->fmt.pix_mp.flags & V4L2_PIX_FMT_FLAG_HAS_CSC)
>> +			return;
>> +		fmt->fmt.pix_mp.colorspace = V4L2_COLORSPACE_DEFAULT;
>> +		fmt->fmt.pix_mp.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>> +		fmt->fmt.pix_mp.quantization = V4L2_QUANTIZATION_DEFAULT;
>> +		fmt->fmt.pix_mp.xfer_func = V4L2_XFER_FUNC_DEFAULT;
>> +	}
>> +
>>   	/*
>>   	 * The v4l2_pix_format structure has been extended with fields that were
>>   	 * not previously required to be set to zero by applications. The priv
>> @@ -1043,8 +1052,16 @@ static void v4l_sanitize_format(struct v4l2_format *fmt)
>>   	    fmt->type != V4L2_BUF_TYPE_VIDEO_OUTPUT)
>>   		return;
>>   
>> -	if (fmt->fmt.pix.priv == V4L2_PIX_FMT_PRIV_MAGIC)
>> +	if (fmt->fmt.pix.priv == V4L2_PIX_FMT_PRIV_MAGIC) {
>> +		if (fmt->type != V4L2_BUF_TYPE_VIDEO_CAPTURE ||
>> +		    fmt->fmt.pix.flags & V4L2_PIX_FMT_FLAG_HAS_CSC)
>> +			return;
>> +		fmt->fmt.pix.colorspace = V4L2_COLORSPACE_DEFAULT;
>> +		fmt->fmt.pix.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>> +		fmt->fmt.pix.quantization = V4L2_QUANTIZATION_DEFAULT;
>> +		fmt->fmt.pix.xfer_func = V4L2_XFER_FUNC_DEFAULT;
>>   		return;
>> +	}
>>   
>>   	fmt->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>>   
>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>> index a376b351135f..51e009936aad 100644
>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>> @@ -477,6 +477,13 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>>   	case VIDIOC_SUBDEV_S_FMT: {
>>   		struct v4l2_subdev_format *format = arg;
>>   
>> +		if (!(format->format.flags & V4L2_MBUS_FRAMEFMT_HAS_CSC)) {
>> +			format->format.colorspace = V4L2_COLORSPACE_DEFAULT;
>> +			format->format.xfer_func = V4L2_XFER_FUNC_DEFAULT;
>> +			format->format.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>> +			format->format.quantization = V4L2_QUANTIZATION_DEFAULT;
>> +		}
>> +
>>   		memset(format->reserved, 0, sizeof(format->reserved));
>>   		memset(format->format.reserved, 0, sizeof(format->format.reserved));
>>   		return v4l2_subdev_call(sd, pad, set_fmt, subdev_fh->pad, format);
>> diff --git a/include/uapi/linux/v4l2-mediabus.h b/include/uapi/linux/v4l2-mediabus.h
>> index 123a231001a8..89ff0ad6a631 100644
>> --- a/include/uapi/linux/v4l2-mediabus.h
>> +++ b/include/uapi/linux/v4l2-mediabus.h
>> @@ -16,6 +16,8 @@
>>   #include <linux/types.h>
>>   #include <linux/videodev2.h>
>>   
>> +#define V4L2_MBUS_FRAMEFMT_HAS_CSC	0x0001
>> +
>>   /**
>>    * struct v4l2_mbus_framefmt - frame format on the media bus
>>    * @width:	image width
>> @@ -36,7 +38,8 @@ struct v4l2_mbus_framefmt {
>>   	__u16			ycbcr_enc;
>>   	__u16			quantization;
>>   	__u16			xfer_func;
>> -	__u16			reserved[11];
>> +	__u16			flags;
>> +	__u16			reserved[10];
>>   };
>>   
>>   #ifndef __KERNEL__
>> diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
>> index 03970ce30741..4410b26a7158 100644
>> --- a/include/uapi/linux/v4l2-subdev.h
>> +++ b/include/uapi/linux/v4l2-subdev.h
>> @@ -65,6 +65,8 @@ struct v4l2_subdev_crop {
>>   	__u32 reserved[8];
>>   };
>>   
>> +#define V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC	0x00000001
> 
> I think it is better to add an HSV_ENC alias here as well.
> 
> It's for readability only, since talking about YCBCR_ENC when the mediabus format
> is a HSV format is just weird.

I see that in 'v4l2_pix_format' the {ycbcr_enc, hsv_enc} are a union
but in 'v4l2_mbus_framefmt' there is only the field ycbcr_enc.
I wonder why is that, since I see there is one HSV media bus format
V4L2_MBUS_FROM_MEDIA_BUS_FMT(AHSV8888_1X32).

So I guess this is why I didn't add a V4L2_SUBDEV_MBUS_CODE_CSC_HSV_ENC define.
I also didn't add this flag to the docs. Should I?

Thanks,
Dafna

> 
>> +#define V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION	0x00000002
>>   /**
>>    * struct v4l2_subdev_mbus_code_enum - Media bus format enumeration
>>    * @pad: pad number, as reported by the media API
>> @@ -77,7 +79,8 @@ struct v4l2_subdev_mbus_code_enum {
>>   	__u32 index;
>>   	__u32 code;
>>   	__u32 which;
>> -	__u32 reserved[8];
>> +	__u32 flags;
>> +	__u32 reserved[7];
>>   };
>>   
>>   /**
>> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
>> index 9817b7e2c968..adc9dd1080b8 100644
>> --- a/include/uapi/linux/videodev2.h
>> +++ b/include/uapi/linux/videodev2.h
>> @@ -772,6 +772,7 @@ struct v4l2_pix_format {
>>   
>>   /* Flags */
>>   #define V4L2_PIX_FMT_FLAG_PREMUL_ALPHA	0x00000001
>> +#define V4L2_PIX_FMT_FLAG_HAS_CSC	0x00000002
>>   
>>   /*
>>    *	F O R M A T   E N U M E R A T I O N
>> @@ -789,6 +790,9 @@ struct v4l2_fmtdesc {
>>   #define V4L2_FMT_FLAG_EMULATED			0x0002
>>   #define V4L2_FMT_FLAG_CONTINUOUS_BYTESTREAM	0x0004
>>   #define V4L2_FMT_FLAG_DYN_RESOLUTION		0x0008
>> +#define V4L2_FMT_FLAG_CSC_YCBCR_ENC		0x0010
>> +#define V4L2_FMT_FLAG_CSC_HSV_ENC		0x0010
>> +#define V4L2_FMT_FLAG_CSC_QUANTIZATION		0x0020
>>   
>>   	/* Frame Size and frame rate enumeration */
>>   /*
>>
> 
> I think this is quite good, it's mainly doc improvements and renaming HAS_CSC to REQUEST_CSC.
> 
> That said, I do want to see this supported in at least vivid (vimc would be nice
> too!). And v4l2-ctl needs a patch to support this.
> 
> A compliance test for v4l2-compliance would be nice too, but it is not a prerequisite.
> I'm not entirely sure what would be the best way to test this. Perhaps a vivid-specific
> test might be easiest (i.e., don't test it for all drivers, but just for vivid).
> 
> Regards,
> 
> 	Hans
> 

Re: [RFC v3 1/2] v4l2: add support for colorspace conversion for video capture

[Hans Verkuil]

On 25/05/2020 15:56, Dafna Hirschfeld wrote:
> Hi
> 
> On 07.05.20 15:00, Hans Verkuil wrote:
>> On 16/04/2020 16:56, Dafna Hirschfeld wrote:
>>> From: Philipp Zabel <p.zabel@pengutronix.de>
>>>
>>> For video capture it is the driver that reports the colorspace,
>>> Y'CbCr/HSV encoding, quantization range and transfer function
>>> used by the video, and there is no way to request something
>>> different, even though many HDTV receivers have some sort of
>>> colorspace conversion capabilities.
>>>
>>> For output video this feature already exists since the application
>>> specifies this information for the video format it will send out, and
>>> the transmitter will enable any available CSC if a format conversion has
>>> to be performed in order to match the capabilities of the sink.
>>>
>>> For video capture we propose adding new pix_format flag:
>>> V4L2_PIX_FMT_FLAG_HAS_CSC. The flag is set by the application,
>>
>> That should be V4L2_PIX_FMT_FLAG_REQUEST_CSC, since the application actively
>> requests the use of the CSC.
> Maybe change to V4L2_PIX_FMT_FLAG_ASK_FOR_CSC so people won't think
> it is related to the request API ?

How about _SET_CSC? I think that's even better than REQUEST_CSC.

>>
>>> the driver will interpret the ycbcr_enc/hsv_enc, and quantization fields
>>> as the requested colorspace information and will attempt to
>>> do the conversion it supports.
>>>
>>> Drivers set the flags
>>> V4L2_FMT_FLAG_CSC_YCBCR_ENC,
>>> V4L2_FMT_FLAG_CSC_HSV_ENC,
>>> V4L2_FMT_FLAG_CSC_HSV_QUANTIZATION,
>>
>> That last define should be 'V4L2_FMT_FLAG_CSC_QUANTIZATION' without 'HSV_'.
>>
>>> in the flags field of the struct v4l2_fmtdesc during enumeration to
>>> indicate that they support colorspace conversion for the respective field.
>>> Currently the conversion of the fields colorspace, xfer_func is not
>>> supported since there are no drivers that support it.
>>>
>>> The same API is added for the subdevices. With the flag V4L2_MBUS_FRAMEFMT_HAS_CSC
>>
>> Should also be REQUEST_CSC.
>>
>>> set by the application in the VIDIOC_SUBDEV_S_FMT ioctl and the flags
>>> V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC, V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION
>>> set by the driver in the VIDIOC_SUBDEV_ENUM_MBUS_CODE ioctl.
>>>
>>> For subdevices, new 'flags' fields were added to the structs
>>> v4l2_subdev_mbus_code_enum, v4l2_mbus_framefmt which are borrowed from the 'reserved' field
>>>
>>> Drivers do not have to actually look at the flags: if the flags are not
>>> set, then the colorspace, ycbcr_enc and quantization fields are set to
>>> the default values by the core, i.e. just pass on the received format
>>> without conversion.
>>>
>>> Signed-off-by: Hans Verkuil <Hans Verkuil@cisco.com>
>>> Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
>>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
>>> ---
>>> This is v3 of the RFC suggested originaly by Hans Verkuil:
>>>
>>>   https://patchwork.linuxtv.org/patch/28847/
>>>
>>> And then a v2 from Philipp Zabel:
>>>
>>>   https://patchwork.kernel.org/project/linux-media/list/?series=15483
>>>
>>> changes in v3:
>>> I added the API to subdevices as well and added fixes according to
>>> comments from Hans.
>>> I also added a usecase for the new API for the rkisp1 driver.
>>>
>>> changes in v2 (reported by Philipp Zabel):
>>>   - convert to rst
>>>   - split V4L2_PIX_FMT_FLAG_REQUEST_CSC into four separate flags for
>>>     colorspace, ycbcr_enc/hsv_enc, quantization, and xfer_func
>>>   - let driver set flags to indicate supported features
>>>
>>> [1] https://patchwork.linuxtv.org/patch/28847/
>>>
>>>   .../media/v4l/pixfmt-reserved.rst             |  6 +++
>>>   .../media/v4l/pixfmt-v4l2-mplane.rst          | 16 ++-----
>>>   .../userspace-api/media/v4l/pixfmt-v4l2.rst   | 36 +++++++++++++---
>>>   .../media/v4l/subdev-formats.rst              | 42 +++++++++++++++++--
>>>   .../media/v4l/vidioc-enum-fmt.rst             | 18 ++++++++
>>>   .../v4l/vidioc-subdev-enum-mbus-code.rst      | 23 ++++++++++
>>>   drivers/media/v4l2-core/v4l2-ioctl.c          | 19 ++++++++-
>>>   drivers/media/v4l2-core/v4l2-subdev.c         |  7 ++++
>>>   include/uapi/linux/v4l2-mediabus.h            |  5 ++-
>>>   include/uapi/linux/v4l2-subdev.h              |  5 ++-
>>>   include/uapi/linux/videodev2.h                |  4 ++
>>>   11 files changed, 158 insertions(+), 23 deletions(-)
>>>
>>> diff --git a/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst b/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst
>>> index 59b9e7238f90..fa8dada69f8c 100644
>>> --- a/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst
>>> +++ b/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst
>>
>> Why on earth did the documentation of these pixel format flags end up in
>> pixfmt-reserved.rst?
>>
>> Can you make a patch that moves this to pixfmt-v4l2.rst?
>>
>>> @@ -280,3 +280,9 @@ please make a proposal on the linux-media mailing list.
>>>   	by RGBA values (128, 192, 255, 128), the same pixel described with
>>>   	premultiplied colors would be described by RGBA values (64, 96,
>>>   	128, 128)
>>> +    * - ``V4L2_PIX_FMT_FLAG_HAS_CSC``
>>> +      - 0x00000002
>>> +      - Set by the application. It is only used for capture and is
>>> +	ignored for output streams. If set, then request the driver to do
>>> +	colorspace conversion from the received colorspace, only conversions
>>> +	of Ycbcr encoding, HSV encoding and quantization are supported.
>>
>> I'd replace the part "If set..." with this:
>>
>> ----------
>> If set, then request the driver to do colorspace conversion from the received
>> colorspace to the requested colorspace values. If a colorimetry field
>> (``colorspace``, ``xfer_func``, ``ycncr_enc`` or ``quantization``) is set to 0,
>> then that colorimetry setting will remain unchanged from what was received.
>> So to just change the quantization only the ``quantization`` field shall be set
>> to a non-zero value (``V4L2_QUANTIZATION_FULL_RANGE`` or
>> ``V4L2_QUANTIZATION_LIM_RANGE``) and all other colorimetry fields shall be set to 0.
>>
>> To check which conversions are supported by the hardware for the current pixel
>> format, see :ref:`fmtdesc-flags`.
>> ----------
>>
>>
>>> diff --git a/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst b/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst
>>> index 444b4082684c..66f3365d7b72 100644
>>> --- a/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst
>>> +++ b/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst
>>> @@ -105,29 +105,21 @@ describing all planes of that format.
>>>       * - __u8
>>>         - ``ycbcr_enc``
>>>         - Y'CbCr encoding, from enum :c:type:`v4l2_ycbcr_encoding`.
>>> -        This information supplements the ``colorspace`` and must be set by
>>> -	the driver for capture streams and by the application for output
>>> -	streams, see :ref:`colorspaces`.
>>> +	See struct :c:type:`v4l2_pix_format`.
>>>       * - __u8
>>>         - ``hsv_enc``
>>>         - HSV encoding, from enum :c:type:`v4l2_hsv_encoding`.
>>> -        This information supplements the ``colorspace`` and must be set by
>>> -	the driver for capture streams and by the application for output
>>> -	streams, see :ref:`colorspaces`.
>>> +	See struct :c:type:`v4l2_pix_format`.
>>>       * - }
>>>         -
>>>       * - __u8
>>>         - ``quantization``
>>>         - Quantization range, from enum :c:type:`v4l2_quantization`.
>>> -        This information supplements the ``colorspace`` and must be set by
>>> -	the driver for capture streams and by the application for output
>>> -	streams, see :ref:`colorspaces`.
>>> +	See struct :c:type:`v4l2_pix_format`.
>>>       * - __u8
>>>         - ``xfer_func``
>>>         - Transfer function, from enum :c:type:`v4l2_xfer_func`.
>>> -        This information supplements the ``colorspace`` and must be set by
>>> -	the driver for capture streams and by the application for output
>>> -	streams, see :ref:`colorspaces`.
>>> +	See struct :c:type:`v4l2_pix_format`.
>>>       * - __u8
>>>         - ``reserved[7]``
>>>         - Reserved for future extensions. Should be zeroed by drivers and
>>> diff --git a/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst b/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst
>>> index 759420a872d6..ce57718cd66b 100644
>>> --- a/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst
>>> +++ b/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst
>>> @@ -110,8 +110,8 @@ Single-planar format structure
>>>         - ``colorspace``
>>>         - Image colorspace, from enum :c:type:`v4l2_colorspace`.
>>>           This information supplements the ``pixelformat`` and must be set
>>> -	by the driver for capture streams and by the application for
>>> -	output streams, see :ref:`colorspaces`.
>>> +	by the driver for capture streams and by the application for output
>>> +	streams, see :ref:`colorspace`.
>>
>> This doesn't appear to change anything.
>>
>>>       * - __u32
>>>         - ``priv``
>>>         - This field indicates whether the remaining fields of the
>>> @@ -148,13 +148,31 @@ Single-planar format structure
>>>         - Y'CbCr encoding, from enum :c:type:`v4l2_ycbcr_encoding`.
>>>           This information supplements the ``colorspace`` and must be set by
>>>   	the driver for capture streams and by the application for output
>>> -	streams, see :ref:`colorspaces`.
>>> +	streams, see :ref:`colorspaces`. If the application sets the
>>> +	flag ``V4L2_PIX_FMT_FLAG_HAS_CSC`` then the application can set
>>> +	this field for a capture stream to request a specific Y'CbCr encoding
>>> +	for the captured image data. The driver will attempt to do the
>>> +	conversion to the specified Y'CbCr encoding or return the encoding it
>>> +	will use if it can't do the conversion. This field is ignored for
>>> +	non-Y'CbCr pixelformats. The driver indicates that ycbcr_enc conversion
>>> +	is supported by setting the flag V4L2_FMT_FLAG_CSC_YCBCR_ENC on the
>>> +	on the corresponding struct :c:type:`v4l2_fmtdesc` during enumeration.
>>
>> on the on the -> in the
>>
>>> +	See :ref:`fmtdesc-flags`
>>>       * - __u32
>>>         - ``hsv_enc``
>>>         - HSV encoding, from enum :c:type:`v4l2_hsv_encoding`.
>>>           This information supplements the ``colorspace`` and must be set by
>>>   	the driver for capture streams and by the application for output
>>> -	streams, see :ref:`colorspaces`.
>>> +	streams, see :ref:`colorspaces`. If the application sets the flag
>>> +	``V4L2_PIX_FMT_FLAG_HAS_CSC`` then the application can set this
>>> +	field for a capture stream to request a specific HSV encoding for the
>>> +	captured image data. The driver will attempt to do the conversion to
>>> +	the specified HSV encoding or return the encoding it will use if it
>>> +	can't do the conversion. This field is ignored for non-HSV
>>> +	pixelformats. The driver indicates that hsv_enc conversion
>>> +	is supported by setting the flag V4L2_FMT_FLAG_CSC_HSV_ENC on the
>>> +	on the corresponding struct :c:type:`v4l2_fmtdesc` during enumeration.
>>
>> Ditto.
>>
>>> +	See :ref:`fmtdesc-flags`
>>>       * - }
>>>         -
>>>       * - __u32
>>> @@ -162,7 +180,15 @@ Single-planar format structure
>>>         - Quantization range, from enum :c:type:`v4l2_quantization`.
>>>           This information supplements the ``colorspace`` and must be set by
>>>   	the driver for capture streams and by the application for output
>>> -	streams, see :ref:`colorspaces`.
>>> +	streams, see :ref:`colorspaces`. If the application sets the flag
>>> +	``V4L2_PIX_FMT_FLAG_HAS_CSC`` then the application can set
>>> +	this field for a capture stream to request a specific quantization
>>> +	range for the captured image data. The driver will attempt to do the
>>> +	conversion to the specified quantization range or return the
>>> +	quantization it will use if it can't do the conversion.  The driver
>>> +	indicates that quantization conversion is supported by setting the flag
>>> +	V4L2_FMT_FLAG_CSC_QUANTIZATION on the on the corresponding struct
>>
>> Ditto
>>
>>> +	:c:type:`v4l2_fmtdesc` during enumeration. See :ref:`fmtdesc-flags`
>>>       * - __u32
>>>         - ``xfer_func``
>>>         - Transfer function, from enum :c:type:`v4l2_xfer_func`.
>>> diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
>>> index 9a4d61b0d76f..f1d4ca29a3e8 100644
>>> --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
>>> +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
>>> @@ -49,13 +49,33 @@ Media Bus Formats
>>>         - Y'CbCr encoding, from enum :c:type:`v4l2_ycbcr_encoding`.
>>>           This information supplements the ``colorspace`` and must be set by
>>>   	the driver for capture streams and by the application for output
>>> -	streams, see :ref:`colorspaces`.
>>> +	streams, see :ref:`colorspaces`. If the application sets the
>>> +	flag ``V4L2_MBUS_FRAMEFMT_HAS_CSC`` the the application can set
>>> +	this field for a capture stream to request a specific Y'CbCr encoding
>>> +	for the mbus data. The driver will attempt to do the
>>
>> mbus -> media bus
>>
>>> +	conversion to the specified Y'CbCr encoding or return the encoding it
>>> +	will use if it can't do the conversion. This field is ignored for
>>> +	non-Y'CbCr pixelformats. The driver indicates that ycbcr_enc conversion
>>
>> This is a media bus format, not a pixelformat.
>>
>>> +	is supported by setting the flag V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC on
>>
>> on -> in
>>
>>> +	the corresponding struct c:type:`v4l2_subdev_mbus_code_enum` during
>>> +	enumeration. See :ref:`vidioc-subdev-enum-mbus-code.rst <v4l2-subdev-mbus-code-flags>`
>>> +
>>>       * - __u16
>>>         - ``quantization``
>>>         - Quantization range, from enum :c:type:`v4l2_quantization`.
>>>           This information supplements the ``colorspace`` and must be set by
>>>   	the driver for capture streams and by the application for output
>>> -	streams, see :ref:`colorspaces`.
>>> +	streams, see :ref:`colorspaces`. If the application sets the
>>> +	flag ``V4L2_MBUS_FRAMEFMT_HAS_CSC`` the the application can set
>>> +	this field for a capture stream to request a specific quantization
>>> +	encoding for the mbus data. The driver will attempt to do the
>>
>> mbus -> media bus
>>
>>> +	conversion to the specified quantization or return the quantization it
>>> +	will use if it can't do the conversion. The driver indicates that
>>> +	quantization conversion is supported by setting the flag
>>> +	V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION on the corresponding struct
>>
>> on -> in
>>
>>> +	c:type:`v4l2_subdev_mbus_code_enum` during enumeration.
>>> +	See :ref:`vidioc-subdev-enum-mbus-code.rst <v4l2-subdev-mbus-code-flags>`
>>> +
>>>       * - __u16
>>>         - ``xfer_func``
>>>         - Transfer function, from enum :c:type:`v4l2_xfer_func`.
>>> @@ -63,10 +83,26 @@ Media Bus Formats
>>>   	the driver for capture streams and by the application for output
>>>   	streams, see :ref:`colorspaces`.
>>>       * - __u16
>>> -      - ``reserved``\ [11]
>>> +      - ``flags``
>>> +      - flags see:  :ref:v4l2-mbus-framefmt-flags
>>
>> flags see: -> See:
>>
>>> +    * - __u16
>>> +      - ``reserved``\ [10]
>>>         - Reserved for future extensions. Applications and drivers must set
>>>   	the array to zero.
>>>   
>>> +.. _v4l2-mbus-framefmt-flags:
>>> +
>>> +.. flat-table:: v4l2_mbus_framefmt Flags
>>> +    :header-rows:  0
>>> +    :stub-columns: 0
>>> +    :widths:       3 1 4
>>> +
>>> +    * - ``V4L2_MBUS_FRAMEFMT_HAS_CSC``
>>> +      - 0x0001
>>> +      - Set by the application. It is only used for capture and is
>>> +	ignored for output streams. If set, then request the subdevice to do
>>> +	colorspace conversion from the received colorspace, only conversions
>>> +	of Ycbcr encoding, and quantization are supported.
>>
>> Replace by a similar text as for V4L2_PIX_FMT_FLAG_HAS_CSC.
>>
>>>   
>>>   
>>>   .. _v4l2-mbus-pixelcode:
>>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
>>> index 7e3142e11d77..125f074543af 100644
>>> --- a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
>>> +++ b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
>>> @@ -145,6 +145,24 @@ formats in preference order, where preferred formats are returned before
>>>   	parameters are detected. This flag can only be used in combination
>>>   	with the ``V4L2_FMT_FLAG_COMPRESSED`` flag, since this applies to
>>>   	compressed formats only. It is also only applies to stateful codecs.
>>> +    * - ``V4L2_FMT_FLAG_CSC_YCBCR_ENC``
>>> +      - 0x0010
>>> +      - The driver allows the application to try to change the default
>>> +	ycbcr encoding. This flag is relevant only for capture devices.
>>> +	The application can ask to configure the ycbcr_enc of the capture device
>>> +	when calling the :c:func:`VIDIOC_S_FMT` ioctl.
>>
>> add: ...ioctl with ``V4L2_FMT_FLAG_REQUEST_CSC`` set. (this should probably be
>> a reference to the REQUEST_CSC flag documentation).
>>
>>> +    * - ``V4L2_FMT_FLAG_CSC_HSV_ENC``
>>> +      - 0x0010
>>> +      - The driver allows the application to try to change the default
>>> +	hsv encoding. This flag is relevant only for capture devices.
>>> +	The application can ask to configure the hsv_enc of the capture device
>>> +	when calling the :c:func:`VIDIOC_S_FMT` ioctl.
>>
>> Ditto. Also: hsv -> HSV.
>>
>>> +    * - ``V4L2_FMT_FLAG_CSC_QUANTIZATION``
>>> +      - 0x0020
>>> +      - The driver allows the application to try to change the default
>>> +	quantization. This flag is relevant only for capture devices.
>>> +	The application can ask to configure the quantization of the capture device
>>> +	when calling the :c:func:`VIDIOC_S_FMT` ioctl.
>>
>> Ditto.
>>
>>>   
>>>   
>>>   Return Value
>>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
>>> index 35b8607203a4..4ad87cb74f57 100644
>>> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
>>> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
>>> @@ -78,11 +78,34 @@ information about the try formats.
>>>         - ``which``
>>>         - Media bus format codes to be enumerated, from enum
>>>   	:ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
>>> +    * - __u32
>>> +      - ``flags``
>>> +      - see :ref:`v4l2-subdev-mbus-code-flags`
>>
>> see -> See
>>
>>>       * - __u32
>>>         - ``reserved``\ [8]
>>>         - Reserved for future extensions. Applications and drivers must set
>>>   	the array to zero.
>>>   
>>> +.. _v4l2-subdev-mbus-code-flags:
>>> +
>>> +
>>> +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{7.7cm}|
>>> +
>>> +.. flat-table:: flags in struct v4l2_subdev_mbus_code_enum
>>> +    :header-rows:  0
>>> +    :stub-columns: 0
>>> +    :widths:       1 1 2
>>> +
>>> +    * - V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC
>>> +      - 0x00000001
>>> +      - The driver supports setting the ycbcr encoding by the application
>>> +	when calling the VIDIOC_SUBDEV_S_FMT ioctl. see :ref:`v4l2-mbus-format`
>>
>> see -> See
>>
>>> +	on how to do this.
>>> +    * - V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION
>>> +      - 0x00000002
>>> +      - The driver supports setting the quantization by the application
>>> +	when calling the VIDIOC_SUBDEV_S_FMT ioctl. see :ref:`v4l2-mbus-format`
>>
>> see -> See
>>
>>> +	on how to do this.
>>>   
>>>   Return Value
>>>   ============
>>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
>>> index b2ef8e60ea7d..3c7ffb6b15cb 100644
>>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
>>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
>>> @@ -1029,6 +1029,15 @@ static void v4l_sanitize_format(struct v4l2_format *fmt)
>>>   		fmt->fmt.pix_mp.num_planes = min_t(u32, fmt->fmt.pix_mp.num_planes,
>>>   					       VIDEO_MAX_PLANES);
>>>   
>>> +	if (fmt->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE) {
>>> +		if (fmt->fmt.pix_mp.flags & V4L2_PIX_FMT_FLAG_HAS_CSC)
>>> +			return;
>>> +		fmt->fmt.pix_mp.colorspace = V4L2_COLORSPACE_DEFAULT;
>>> +		fmt->fmt.pix_mp.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>>> +		fmt->fmt.pix_mp.quantization = V4L2_QUANTIZATION_DEFAULT;
>>> +		fmt->fmt.pix_mp.xfer_func = V4L2_XFER_FUNC_DEFAULT;
>>> +	}
>>> +
>>>   	/*
>>>   	 * The v4l2_pix_format structure has been extended with fields that were
>>>   	 * not previously required to be set to zero by applications. The priv
>>> @@ -1043,8 +1052,16 @@ static void v4l_sanitize_format(struct v4l2_format *fmt)
>>>   	    fmt->type != V4L2_BUF_TYPE_VIDEO_OUTPUT)
>>>   		return;
>>>   
>>> -	if (fmt->fmt.pix.priv == V4L2_PIX_FMT_PRIV_MAGIC)
>>> +	if (fmt->fmt.pix.priv == V4L2_PIX_FMT_PRIV_MAGIC) {
>>> +		if (fmt->type != V4L2_BUF_TYPE_VIDEO_CAPTURE ||
>>> +		    fmt->fmt.pix.flags & V4L2_PIX_FMT_FLAG_HAS_CSC)
>>> +			return;
>>> +		fmt->fmt.pix.colorspace = V4L2_COLORSPACE_DEFAULT;
>>> +		fmt->fmt.pix.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>>> +		fmt->fmt.pix.quantization = V4L2_QUANTIZATION_DEFAULT;
>>> +		fmt->fmt.pix.xfer_func = V4L2_XFER_FUNC_DEFAULT;
>>>   		return;
>>> +	}
>>>   
>>>   	fmt->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>>>   
>>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>>> index a376b351135f..51e009936aad 100644
>>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>>> @@ -477,6 +477,13 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>>>   	case VIDIOC_SUBDEV_S_FMT: {
>>>   		struct v4l2_subdev_format *format = arg;
>>>   
>>> +		if (!(format->format.flags & V4L2_MBUS_FRAMEFMT_HAS_CSC)) {
>>> +			format->format.colorspace = V4L2_COLORSPACE_DEFAULT;
>>> +			format->format.xfer_func = V4L2_XFER_FUNC_DEFAULT;
>>> +			format->format.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>>> +			format->format.quantization = V4L2_QUANTIZATION_DEFAULT;
>>> +		}
>>> +
>>>   		memset(format->reserved, 0, sizeof(format->reserved));
>>>   		memset(format->format.reserved, 0, sizeof(format->format.reserved));
>>>   		return v4l2_subdev_call(sd, pad, set_fmt, subdev_fh->pad, format);
>>> diff --git a/include/uapi/linux/v4l2-mediabus.h b/include/uapi/linux/v4l2-mediabus.h
>>> index 123a231001a8..89ff0ad6a631 100644
>>> --- a/include/uapi/linux/v4l2-mediabus.h
>>> +++ b/include/uapi/linux/v4l2-mediabus.h
>>> @@ -16,6 +16,8 @@
>>>   #include <linux/types.h>
>>>   #include <linux/videodev2.h>
>>>   
>>> +#define V4L2_MBUS_FRAMEFMT_HAS_CSC	0x0001
>>> +
>>>   /**
>>>    * struct v4l2_mbus_framefmt - frame format on the media bus
>>>    * @width:	image width
>>> @@ -36,7 +38,8 @@ struct v4l2_mbus_framefmt {
>>>   	__u16			ycbcr_enc;
>>>   	__u16			quantization;
>>>   	__u16			xfer_func;
>>> -	__u16			reserved[11];
>>> +	__u16			flags;
>>> +	__u16			reserved[10];
>>>   };
>>>   
>>>   #ifndef __KERNEL__
>>> diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
>>> index 03970ce30741..4410b26a7158 100644
>>> --- a/include/uapi/linux/v4l2-subdev.h
>>> +++ b/include/uapi/linux/v4l2-subdev.h
>>> @@ -65,6 +65,8 @@ struct v4l2_subdev_crop {
>>>   	__u32 reserved[8];
>>>   };
>>>   
>>> +#define V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC	0x00000001
>>
>> I think it is better to add an HSV_ENC alias here as well.
>>
>> It's for readability only, since talking about YCBCR_ENC when the mediabus format
>> is a HSV format is just weird.
> 
> I see that in 'v4l2_pix_format' the {ycbcr_enc, hsv_enc} are a union
> but in 'v4l2_mbus_framefmt' there is only the field ycbcr_enc.
> I wonder why is that, since I see there is one HSV media bus format
> V4L2_MBUS_FROM_MEDIA_BUS_FMT(AHSV8888_1X32).
> 
> So I guess this is why I didn't add a V4L2_SUBDEV_MBUS_CODE_CSC_HSV_ENC define.
> I also didn't add this flag to the docs. Should I?

I think this can be done in a separate (final) patch where a union is added for
v4l2_mbus_framefmt together with a V4L2_SUBDEV_MBUS_CODE_CSC_HSV_ENC define.

These are effectively aliases to make code that works with HSV easier to read,
so they do not change functionality.

Regards,

	Hans

> 
> Thanks,
> Dafna
> 
>>
>>> +#define V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION	0x00000002
>>>   /**
>>>    * struct v4l2_subdev_mbus_code_enum - Media bus format enumeration
>>>    * @pad: pad number, as reported by the media API
>>> @@ -77,7 +79,8 @@ struct v4l2_subdev_mbus_code_enum {
>>>   	__u32 index;
>>>   	__u32 code;
>>>   	__u32 which;
>>> -	__u32 reserved[8];
>>> +	__u32 flags;
>>> +	__u32 reserved[7];
>>>   };
>>>   
>>>   /**
>>> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
>>> index 9817b7e2c968..adc9dd1080b8 100644
>>> --- a/include/uapi/linux/videodev2.h
>>> +++ b/include/uapi/linux/videodev2.h
>>> @@ -772,6 +772,7 @@ struct v4l2_pix_format {
>>>   
>>>   /* Flags */
>>>   #define V4L2_PIX_FMT_FLAG_PREMUL_ALPHA	0x00000001
>>> +#define V4L2_PIX_FMT_FLAG_HAS_CSC	0x00000002
>>>   
>>>   /*
>>>    *	F O R M A T   E N U M E R A T I O N
>>> @@ -789,6 +790,9 @@ struct v4l2_fmtdesc {
>>>   #define V4L2_FMT_FLAG_EMULATED			0x0002
>>>   #define V4L2_FMT_FLAG_CONTINUOUS_BYTESTREAM	0x0004
>>>   #define V4L2_FMT_FLAG_DYN_RESOLUTION		0x0008
>>> +#define V4L2_FMT_FLAG_CSC_YCBCR_ENC		0x0010
>>> +#define V4L2_FMT_FLAG_CSC_HSV_ENC		0x0010
>>> +#define V4L2_FMT_FLAG_CSC_QUANTIZATION		0x0020
>>>   
>>>   	/* Frame Size and frame rate enumeration */
>>>   /*
>>>
>>
>> I think this is quite good, it's mainly doc improvements and renaming HAS_CSC to REQUEST_CSC.
>>
>> That said, I do want to see this supported in at least vivid (vimc would be nice
>> too!). And v4l2-ctl needs a patch to support this.
>>
>> A compliance test for v4l2-compliance would be nice too, but it is not a prerequisite.
>> I'm not entirely sure what would be the best way to test this. Perhaps a vivid-specific
>> test might be easiest (i.e., don't test it for all drivers, but just for vivid).
>>
>> Regards,
>>
>> 	Hans
>>

Re: [RFC v3 1/2] v4l2: add support for colorspace conversion for video capture

[Dafna Hirschfeld]

On 25.05.20 16:25, Hans Verkuil wrote:
> On 25/05/2020 15:56, Dafna Hirschfeld wrote:
>> Hi
>>
>> On 07.05.20 15:00, Hans Verkuil wrote:
>>> On 16/04/2020 16:56, Dafna Hirschfeld wrote:
>>>> From: Philipp Zabel <p.zabel@pengutronix.de>
>>>>
>>>> For video capture it is the driver that reports the colorspace,
>>>> Y'CbCr/HSV encoding, quantization range and transfer function
>>>> used by the video, and there is no way to request something
>>>> different, even though many HDTV receivers have some sort of
>>>> colorspace conversion capabilities.
>>>>
>>>> For output video this feature already exists since the application
>>>> specifies this information for the video format it will send out, and
>>>> the transmitter will enable any available CSC if a format conversion has
>>>> to be performed in order to match the capabilities of the sink.
>>>>
>>>> For video capture we propose adding new pix_format flag:
>>>> V4L2_PIX_FMT_FLAG_HAS_CSC. The flag is set by the application,
>>>
>>> That should be V4L2_PIX_FMT_FLAG_REQUEST_CSC, since the application actively
>>> requests the use of the CSC.
>> Maybe change to V4L2_PIX_FMT_FLAG_ASK_FOR_CSC so people won't think
>> it is related to the request API ?
> 
> How about _SET_CSC? I think that's even better than REQUEST_CSC.
> 
>>>
>>>> the driver will interpret the ycbcr_enc/hsv_enc, and quantization fields
>>>> as the requested colorspace information and will attempt to
>>>> do the conversion it supports.
>>>>
>>>> Drivers set the flags
>>>> V4L2_FMT_FLAG_CSC_YCBCR_ENC,
>>>> V4L2_FMT_FLAG_CSC_HSV_ENC,
>>>> V4L2_FMT_FLAG_CSC_HSV_QUANTIZATION,
>>>
>>> That last define should be 'V4L2_FMT_FLAG_CSC_QUANTIZATION' without 'HSV_'.
>>>
>>>> in the flags field of the struct v4l2_fmtdesc during enumeration to
>>>> indicate that they support colorspace conversion for the respective field.
>>>> Currently the conversion of the fields colorspace, xfer_func is not
>>>> supported since there are no drivers that support it.
>>>>
>>>> The same API is added for the subdevices. With the flag V4L2_MBUS_FRAMEFMT_HAS_CSC
>>>
>>> Should also be REQUEST_CSC.
>>>
>>>> set by the application in the VIDIOC_SUBDEV_S_FMT ioctl and the flags
>>>> V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC, V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION
>>>> set by the driver in the VIDIOC_SUBDEV_ENUM_MBUS_CODE ioctl.
>>>>
>>>> For subdevices, new 'flags' fields were added to the structs
>>>> v4l2_subdev_mbus_code_enum, v4l2_mbus_framefmt which are borrowed from the 'reserved' field
>>>>
>>>> Drivers do not have to actually look at the flags: if the flags are not
>>>> set, then the colorspace, ycbcr_enc and quantization fields are set to
>>>> the default values by the core, i.e. just pass on the received format
>>>> without conversion.
>>>>
>>>> Signed-off-by: Hans Verkuil <Hans Verkuil@cisco.com>
>>>> Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
>>>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
>>>> ---
>>>> This is v3 of the RFC suggested originaly by Hans Verkuil:
>>>>
>>>>    https://patchwork.linuxtv.org/patch/28847/
>>>>
>>>> And then a v2 from Philipp Zabel:
>>>>
>>>>    https://patchwork.kernel.org/project/linux-media/list/?series=15483
>>>>
>>>> changes in v3:
>>>> I added the API to subdevices as well and added fixes according to
>>>> comments from Hans.
>>>> I also added a usecase for the new API for the rkisp1 driver.
>>>>
>>>> changes in v2 (reported by Philipp Zabel):
>>>>    - convert to rst
>>>>    - split V4L2_PIX_FMT_FLAG_REQUEST_CSC into four separate flags for
>>>>      colorspace, ycbcr_enc/hsv_enc, quantization, and xfer_func
>>>>    - let driver set flags to indicate supported features
>>>>
>>>> [1] https://patchwork.linuxtv.org/patch/28847/
>>>>
>>>>    .../media/v4l/pixfmt-reserved.rst             |  6 +++
>>>>    .../media/v4l/pixfmt-v4l2-mplane.rst          | 16 ++-----
>>>>    .../userspace-api/media/v4l/pixfmt-v4l2.rst   | 36 +++++++++++++---
>>>>    .../media/v4l/subdev-formats.rst              | 42 +++++++++++++++++--
>>>>    .../media/v4l/vidioc-enum-fmt.rst             | 18 ++++++++
>>>>    .../v4l/vidioc-subdev-enum-mbus-code.rst      | 23 ++++++++++
>>>>    drivers/media/v4l2-core/v4l2-ioctl.c          | 19 ++++++++-
>>>>    drivers/media/v4l2-core/v4l2-subdev.c         |  7 ++++
>>>>    include/uapi/linux/v4l2-mediabus.h            |  5 ++-
>>>>    include/uapi/linux/v4l2-subdev.h              |  5 ++-
>>>>    include/uapi/linux/videodev2.h                |  4 ++
>>>>    11 files changed, 158 insertions(+), 23 deletions(-)
>>>>
>>>> diff --git a/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst b/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst
>>>> index 59b9e7238f90..fa8dada69f8c 100644
>>>> --- a/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst
>>>> +++ b/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst
>>>
>>> Why on earth did the documentation of these pixel format flags end up in
>>> pixfmt-reserved.rst?
>>>
>>> Can you make a patch that moves this to pixfmt-v4l2.rst?
>>>
>>>> @@ -280,3 +280,9 @@ please make a proposal on the linux-media mailing list.
>>>>    	by RGBA values (128, 192, 255, 128), the same pixel described with
>>>>    	premultiplied colors would be described by RGBA values (64, 96,
>>>>    	128, 128)
>>>> +    * - ``V4L2_PIX_FMT_FLAG_HAS_CSC``
>>>> +      - 0x00000002
>>>> +      - Set by the application. It is only used for capture and is
>>>> +	ignored for output streams. If set, then request the driver to do
>>>> +	colorspace conversion from the received colorspace, only conversions
>>>> +	of Ycbcr encoding, HSV encoding and quantization are supported.
>>>
>>> I'd replace the part "If set..." with this:
>>>
>>> ----------
>>> If set, then request the driver to do colorspace conversion from the received
>>> colorspace to the requested colorspace values. If a colorimetry field
>>> (``colorspace``, ``xfer_func``, ``ycncr_enc`` or ``quantization``) is set to 0,
>>> then that colorimetry setting will remain unchanged from what was received.
>>> So to just change the quantization only the ``quantization`` field shall be set
>>> to a non-zero value (``V4L2_QUANTIZATION_FULL_RANGE`` or
>>> ``V4L2_QUANTIZATION_LIM_RANGE``) and all other colorimetry fields shall be set to 0.
>>>
>>> To check which conversions are supported by the hardware for the current pixel
>>> format, see :ref:`fmtdesc-flags`.
>>> ----------
>>>
>>>
>>>> diff --git a/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst b/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst
>>>> index 444b4082684c..66f3365d7b72 100644
>>>> --- a/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst
>>>> +++ b/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst
>>>> @@ -105,29 +105,21 @@ describing all planes of that format.
>>>>        * - __u8
>>>>          - ``ycbcr_enc``
>>>>          - Y'CbCr encoding, from enum :c:type:`v4l2_ycbcr_encoding`.
>>>> -        This information supplements the ``colorspace`` and must be set by
>>>> -	the driver for capture streams and by the application for output
>>>> -	streams, see :ref:`colorspaces`.
>>>> +	See struct :c:type:`v4l2_pix_format`.
>>>>        * - __u8
>>>>          - ``hsv_enc``
>>>>          - HSV encoding, from enum :c:type:`v4l2_hsv_encoding`.
>>>> -        This information supplements the ``colorspace`` and must be set by
>>>> -	the driver for capture streams and by the application for output
>>>> -	streams, see :ref:`colorspaces`.
>>>> +	See struct :c:type:`v4l2_pix_format`.
>>>>        * - }
>>>>          -
>>>>        * - __u8
>>>>          - ``quantization``
>>>>          - Quantization range, from enum :c:type:`v4l2_quantization`.
>>>> -        This information supplements the ``colorspace`` and must be set by
>>>> -	the driver for capture streams and by the application for output
>>>> -	streams, see :ref:`colorspaces`.
>>>> +	See struct :c:type:`v4l2_pix_format`.
>>>>        * - __u8
>>>>          - ``xfer_func``
>>>>          - Transfer function, from enum :c:type:`v4l2_xfer_func`.
>>>> -        This information supplements the ``colorspace`` and must be set by
>>>> -	the driver for capture streams and by the application for output
>>>> -	streams, see :ref:`colorspaces`.
>>>> +	See struct :c:type:`v4l2_pix_format`.
>>>>        * - __u8
>>>>          - ``reserved[7]``
>>>>          - Reserved for future extensions. Should be zeroed by drivers and
>>>> diff --git a/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst b/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst
>>>> index 759420a872d6..ce57718cd66b 100644
>>>> --- a/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst
>>>> +++ b/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst
>>>> @@ -110,8 +110,8 @@ Single-planar format structure
>>>>          - ``colorspace``
>>>>          - Image colorspace, from enum :c:type:`v4l2_colorspace`.
>>>>            This information supplements the ``pixelformat`` and must be set
>>>> -	by the driver for capture streams and by the application for
>>>> -	output streams, see :ref:`colorspaces`.
>>>> +	by the driver for capture streams and by the application for output
>>>> +	streams, see :ref:`colorspace`.
>>>
>>> This doesn't appear to change anything.
>>>
>>>>        * - __u32
>>>>          - ``priv``
>>>>          - This field indicates whether the remaining fields of the
>>>> @@ -148,13 +148,31 @@ Single-planar format structure
>>>>          - Y'CbCr encoding, from enum :c:type:`v4l2_ycbcr_encoding`.
>>>>            This information supplements the ``colorspace`` and must be set by
>>>>    	the driver for capture streams and by the application for output
>>>> -	streams, see :ref:`colorspaces`.
>>>> +	streams, see :ref:`colorspaces`. If the application sets the
>>>> +	flag ``V4L2_PIX_FMT_FLAG_HAS_CSC`` then the application can set
>>>> +	this field for a capture stream to request a specific Y'CbCr encoding
>>>> +	for the captured image data. The driver will attempt to do the
>>>> +	conversion to the specified Y'CbCr encoding or return the encoding it
>>>> +	will use if it can't do the conversion. This field is ignored for
>>>> +	non-Y'CbCr pixelformats. The driver indicates that ycbcr_enc conversion
>>>> +	is supported by setting the flag V4L2_FMT_FLAG_CSC_YCBCR_ENC on the
>>>> +	on the corresponding struct :c:type:`v4l2_fmtdesc` during enumeration.
>>>
>>> on the on the -> in the
>>>
>>>> +	See :ref:`fmtdesc-flags`
>>>>        * - __u32
>>>>          - ``hsv_enc``
>>>>          - HSV encoding, from enum :c:type:`v4l2_hsv_encoding`.
>>>>            This information supplements the ``colorspace`` and must be set by
>>>>    	the driver for capture streams and by the application for output
>>>> -	streams, see :ref:`colorspaces`.
>>>> +	streams, see :ref:`colorspaces`. If the application sets the flag
>>>> +	``V4L2_PIX_FMT_FLAG_HAS_CSC`` then the application can set this
>>>> +	field for a capture stream to request a specific HSV encoding for the
>>>> +	captured image data. The driver will attempt to do the conversion to
>>>> +	the specified HSV encoding or return the encoding it will use if it
>>>> +	can't do the conversion. This field is ignored for non-HSV
>>>> +	pixelformats. The driver indicates that hsv_enc conversion
>>>> +	is supported by setting the flag V4L2_FMT_FLAG_CSC_HSV_ENC on the
>>>> +	on the corresponding struct :c:type:`v4l2_fmtdesc` during enumeration.
>>>
>>> Ditto.
>>>
>>>> +	See :ref:`fmtdesc-flags`
>>>>        * - }
>>>>          -
>>>>        * - __u32
>>>> @@ -162,7 +180,15 @@ Single-planar format structure
>>>>          - Quantization range, from enum :c:type:`v4l2_quantization`.
>>>>            This information supplements the ``colorspace`` and must be set by
>>>>    	the driver for capture streams and by the application for output
>>>> -	streams, see :ref:`colorspaces`.
>>>> +	streams, see :ref:`colorspaces`. If the application sets the flag
>>>> +	``V4L2_PIX_FMT_FLAG_HAS_CSC`` then the application can set
>>>> +	this field for a capture stream to request a specific quantization
>>>> +	range for the captured image data. The driver will attempt to do the
>>>> +	conversion to the specified quantization range or return the
>>>> +	quantization it will use if it can't do the conversion.  The driver
>>>> +	indicates that quantization conversion is supported by setting the flag
>>>> +	V4L2_FMT_FLAG_CSC_QUANTIZATION on the on the corresponding struct
>>>
>>> Ditto
>>>
>>>> +	:c:type:`v4l2_fmtdesc` during enumeration. See :ref:`fmtdesc-flags`
>>>>        * - __u32
>>>>          - ``xfer_func``
>>>>          - Transfer function, from enum :c:type:`v4l2_xfer_func`.
>>>> diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
>>>> index 9a4d61b0d76f..f1d4ca29a3e8 100644
>>>> --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
>>>> +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
>>>> @@ -49,13 +49,33 @@ Media Bus Formats
>>>>          - Y'CbCr encoding, from enum :c:type:`v4l2_ycbcr_encoding`.
>>>>            This information supplements the ``colorspace`` and must be set by
>>>>    	the driver for capture streams and by the application for output
>>>> -	streams, see :ref:`colorspaces`.
>>>> +	streams, see :ref:`colorspaces`. If the application sets the
>>>> +	flag ``V4L2_MBUS_FRAMEFMT_HAS_CSC`` the the application can set
>>>> +	this field for a capture stream to request a specific Y'CbCr encoding
>>>> +	for the mbus data. The driver will attempt to do the
>>>
>>> mbus -> media bus
>>>
>>>> +	conversion to the specified Y'CbCr encoding or return the encoding it
>>>> +	will use if it can't do the conversion. This field is ignored for
>>>> +	non-Y'CbCr pixelformats. The driver indicates that ycbcr_enc conversion
>>>
>>> This is a media bus format, not a pixelformat.
>>>
>>>> +	is supported by setting the flag V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC on
>>>
>>> on -> in
>>>
>>>> +	the corresponding struct c:type:`v4l2_subdev_mbus_code_enum` during
>>>> +	enumeration. See :ref:`vidioc-subdev-enum-mbus-code.rst <v4l2-subdev-mbus-code-flags>`
>>>> +
>>>>        * - __u16
>>>>          - ``quantization``
>>>>          - Quantization range, from enum :c:type:`v4l2_quantization`.
>>>>            This information supplements the ``colorspace`` and must be set by
>>>>    	the driver for capture streams and by the application for output
>>>> -	streams, see :ref:`colorspaces`.
>>>> +	streams, see :ref:`colorspaces`. If the application sets the
>>>> +	flag ``V4L2_MBUS_FRAMEFMT_HAS_CSC`` the the application can set
>>>> +	this field for a capture stream to request a specific quantization
>>>> +	encoding for the mbus data. The driver will attempt to do the
>>>
>>> mbus -> media bus
>>>
>>>> +	conversion to the specified quantization or return the quantization it
>>>> +	will use if it can't do the conversion. The driver indicates that
>>>> +	quantization conversion is supported by setting the flag
>>>> +	V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION on the corresponding struct
>>>
>>> on -> in
>>>
>>>> +	c:type:`v4l2_subdev_mbus_code_enum` during enumeration.
>>>> +	See :ref:`vidioc-subdev-enum-mbus-code.rst <v4l2-subdev-mbus-code-flags>`
>>>> +
>>>>        * - __u16
>>>>          - ``xfer_func``
>>>>          - Transfer function, from enum :c:type:`v4l2_xfer_func`.
>>>> @@ -63,10 +83,26 @@ Media Bus Formats
>>>>    	the driver for capture streams and by the application for output
>>>>    	streams, see :ref:`colorspaces`.
>>>>        * - __u16
>>>> -      - ``reserved``\ [11]
>>>> +      - ``flags``
>>>> +      - flags see:  :ref:v4l2-mbus-framefmt-flags
>>>
>>> flags see: -> See:
>>>
>>>> +    * - __u16
>>>> +      - ``reserved``\ [10]
>>>>          - Reserved for future extensions. Applications and drivers must set
>>>>    	the array to zero.
>>>>    
>>>> +.. _v4l2-mbus-framefmt-flags:
>>>> +
>>>> +.. flat-table:: v4l2_mbus_framefmt Flags
>>>> +    :header-rows:  0
>>>> +    :stub-columns: 0
>>>> +    :widths:       3 1 4
>>>> +
>>>> +    * - ``V4L2_MBUS_FRAMEFMT_HAS_CSC``
>>>> +      - 0x0001
>>>> +      - Set by the application. It is only used for capture and is
>>>> +	ignored for output streams. If set, then request the subdevice to do
>>>> +	colorspace conversion from the received colorspace, only conversions
>>>> +	of Ycbcr encoding, and quantization are supported.
>>>
>>> Replace by a similar text as for V4L2_PIX_FMT_FLAG_HAS_CSC.
>>>
>>>>    
>>>>    
>>>>    .. _v4l2-mbus-pixelcode:
>>>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
>>>> index 7e3142e11d77..125f074543af 100644
>>>> --- a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
>>>> +++ b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
>>>> @@ -145,6 +145,24 @@ formats in preference order, where preferred formats are returned before
>>>>    	parameters are detected. This flag can only be used in combination
>>>>    	with the ``V4L2_FMT_FLAG_COMPRESSED`` flag, since this applies to
>>>>    	compressed formats only. It is also only applies to stateful codecs.
>>>> +    * - ``V4L2_FMT_FLAG_CSC_YCBCR_ENC``
>>>> +      - 0x0010
>>>> +      - The driver allows the application to try to change the default
>>>> +	ycbcr encoding. This flag is relevant only for capture devices.
>>>> +	The application can ask to configure the ycbcr_enc of the capture device
>>>> +	when calling the :c:func:`VIDIOC_S_FMT` ioctl.
>>>
>>> add: ...ioctl with ``V4L2_FMT_FLAG_REQUEST_CSC`` set. (this should probably be
>>> a reference to the REQUEST_CSC flag documentation).
>>>
>>>> +    * - ``V4L2_FMT_FLAG_CSC_HSV_ENC``
>>>> +      - 0x0010
>>>> +      - The driver allows the application to try to change the default
>>>> +	hsv encoding. This flag is relevant only for capture devices.
>>>> +	The application can ask to configure the hsv_enc of the capture device
>>>> +	when calling the :c:func:`VIDIOC_S_FMT` ioctl.
>>>
>>> Ditto. Also: hsv -> HSV.
>>>
>>>> +    * - ``V4L2_FMT_FLAG_CSC_QUANTIZATION``
>>>> +      - 0x0020
>>>> +      - The driver allows the application to try to change the default
>>>> +	quantization. This flag is relevant only for capture devices.
>>>> +	The application can ask to configure the quantization of the capture device
>>>> +	when calling the :c:func:`VIDIOC_S_FMT` ioctl.
>>>
>>> Ditto.
>>>
>>>>    
>>>>    
>>>>    Return Value
>>>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
>>>> index 35b8607203a4..4ad87cb74f57 100644
>>>> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
>>>> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
>>>> @@ -78,11 +78,34 @@ information about the try formats.
>>>>          - ``which``
>>>>          - Media bus format codes to be enumerated, from enum
>>>>    	:ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
>>>> +    * - __u32
>>>> +      - ``flags``
>>>> +      - see :ref:`v4l2-subdev-mbus-code-flags`
>>>
>>> see -> See
>>>
>>>>        * - __u32
>>>>          - ``reserved``\ [8]
>>>>          - Reserved for future extensions. Applications and drivers must set
>>>>    	the array to zero.
>>>>    
>>>> +.. _v4l2-subdev-mbus-code-flags:
>>>> +
>>>> +
>>>> +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{7.7cm}|
>>>> +
>>>> +.. flat-table:: flags in struct v4l2_subdev_mbus_code_enum
>>>> +    :header-rows:  0
>>>> +    :stub-columns: 0
>>>> +    :widths:       1 1 2
>>>> +
>>>> +    * - V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC
>>>> +      - 0x00000001
>>>> +      - The driver supports setting the ycbcr encoding by the application
>>>> +	when calling the VIDIOC_SUBDEV_S_FMT ioctl. see :ref:`v4l2-mbus-format`
>>>
>>> see -> See
>>>
>>>> +	on how to do this.
>>>> +    * - V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION
>>>> +      - 0x00000002
>>>> +      - The driver supports setting the quantization by the application
>>>> +	when calling the VIDIOC_SUBDEV_S_FMT ioctl. see :ref:`v4l2-mbus-format`
>>>
>>> see -> See
>>>
>>>> +	on how to do this.
>>>>    
>>>>    Return Value
>>>>    ============
>>>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
>>>> index b2ef8e60ea7d..3c7ffb6b15cb 100644
>>>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
>>>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
>>>> @@ -1029,6 +1029,15 @@ static void v4l_sanitize_format(struct v4l2_format *fmt)
>>>>    		fmt->fmt.pix_mp.num_planes = min_t(u32, fmt->fmt.pix_mp.num_planes,
>>>>    					       VIDEO_MAX_PLANES);
>>>>    
>>>> +	if (fmt->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE) {
>>>> +		if (fmt->fmt.pix_mp.flags & V4L2_PIX_FMT_FLAG_HAS_CSC)
>>>> +			return;
>>>> +		fmt->fmt.pix_mp.colorspace = V4L2_COLORSPACE_DEFAULT;
>>>> +		fmt->fmt.pix_mp.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>>>> +		fmt->fmt.pix_mp.quantization = V4L2_QUANTIZATION_DEFAULT;
>>>> +		fmt->fmt.pix_mp.xfer_func = V4L2_XFER_FUNC_DEFAULT;
>>>> +	}
>>>> +
>>>>    	/*
>>>>    	 * The v4l2_pix_format structure has been extended with fields that were
>>>>    	 * not previously required to be set to zero by applications. The priv
>>>> @@ -1043,8 +1052,16 @@ static void v4l_sanitize_format(struct v4l2_format *fmt)
>>>>    	    fmt->type != V4L2_BUF_TYPE_VIDEO_OUTPUT)
>>>>    		return;
>>>>    
>>>> -	if (fmt->fmt.pix.priv == V4L2_PIX_FMT_PRIV_MAGIC)
>>>> +	if (fmt->fmt.pix.priv == V4L2_PIX_FMT_PRIV_MAGIC) {
>>>> +		if (fmt->type != V4L2_BUF_TYPE_VIDEO_CAPTURE ||
>>>> +		    fmt->fmt.pix.flags & V4L2_PIX_FMT_FLAG_HAS_CSC)
>>>> +			return;
>>>> +		fmt->fmt.pix.colorspace = V4L2_COLORSPACE_DEFAULT;
>>>> +		fmt->fmt.pix.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>>>> +		fmt->fmt.pix.quantization = V4L2_QUANTIZATION_DEFAULT;
>>>> +		fmt->fmt.pix.xfer_func = V4L2_XFER_FUNC_DEFAULT;
>>>>    		return;
>>>> +	}
>>>>    
>>>>    	fmt->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>>>>    
>>>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>>>> index a376b351135f..51e009936aad 100644
>>>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>>>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>>>> @@ -477,6 +477,13 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>>>>    	case VIDIOC_SUBDEV_S_FMT: {
>>>>    		struct v4l2_subdev_format *format = arg;
>>>>    
>>>> +		if (!(format->format.flags & V4L2_MBUS_FRAMEFMT_HAS_CSC)) {
>>>> +			format->format.colorspace = V4L2_COLORSPACE_DEFAULT;
>>>> +			format->format.xfer_func = V4L2_XFER_FUNC_DEFAULT;
>>>> +			format->format.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>>>> +			format->format.quantization = V4L2_QUANTIZATION_DEFAULT;
>>>> +		}
I wonder if those 3 code chunks added to v4l2-ioctl.c and v4l2-subdev.c are actually
needed. They set the colorspace fields on capture devices to 0 if the HAS_CSC flag is not set.
But I don't see the reason to do it, if the capture driver do not support CSC it will just ignore
the values of those fields that came from userspace which is what is already done.

Also, it could be that a driver does not support CSC and userspace set the SET_CSC flag anyway, in this
case those fields will not be set to defalut although they should.
What do you think?

Thanks,
Dafna

>>>> +
>>>>    		memset(format->reserved, 0, sizeof(format->reserved));
>>>>    		memset(format->format.reserved, 0, sizeof(format->format.reserved));
>>>>    		return v4l2_subdev_call(sd, pad, set_fmt, subdev_fh->pad, format);
>>>> diff --git a/include/uapi/linux/v4l2-mediabus.h b/include/uapi/linux/v4l2-mediabus.h
>>>> index 123a231001a8..89ff0ad6a631 100644
>>>> --- a/include/uapi/linux/v4l2-mediabus.h
>>>> +++ b/include/uapi/linux/v4l2-mediabus.h
>>>> @@ -16,6 +16,8 @@
>>>>    #include <linux/types.h>
>>>>    #include <linux/videodev2.h>
>>>>    
>>>> +#define V4L2_MBUS_FRAMEFMT_HAS_CSC	0x0001
>>>> +
>>>>    /**
>>>>     * struct v4l2_mbus_framefmt - frame format on the media bus
>>>>     * @width:	image width
>>>> @@ -36,7 +38,8 @@ struct v4l2_mbus_framefmt {
>>>>    	__u16			ycbcr_enc;
>>>>    	__u16			quantization;
>>>>    	__u16			xfer_func;
>>>> -	__u16			reserved[11];
>>>> +	__u16			flags;
>>>> +	__u16			reserved[10];
>>>>    };
>>>>    
>>>>    #ifndef __KERNEL__
>>>> diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
>>>> index 03970ce30741..4410b26a7158 100644
>>>> --- a/include/uapi/linux/v4l2-subdev.h
>>>> +++ b/include/uapi/linux/v4l2-subdev.h
>>>> @@ -65,6 +65,8 @@ struct v4l2_subdev_crop {
>>>>    	__u32 reserved[8];
>>>>    };
>>>>    
>>>> +#define V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC	0x00000001
>>>
>>> I think it is better to add an HSV_ENC alias here as well.
>>>
>>> It's for readability only, since talking about YCBCR_ENC when the mediabus format
>>> is a HSV format is just weird.
>>
>> I see that in 'v4l2_pix_format' the {ycbcr_enc, hsv_enc} are a union
>> but in 'v4l2_mbus_framefmt' there is only the field ycbcr_enc.
>> I wonder why is that, since I see there is one HSV media bus format
>> V4L2_MBUS_FROM_MEDIA_BUS_FMT(AHSV8888_1X32).
>>
>> So I guess this is why I didn't add a V4L2_SUBDEV_MBUS_CODE_CSC_HSV_ENC define.
>> I also didn't add this flag to the docs. Should I?
> 
> I think this can be done in a separate (final) patch where a union is added for
> v4l2_mbus_framefmt together with a V4L2_SUBDEV_MBUS_CODE_CSC_HSV_ENC define.
> 
> These are effectively aliases to make code that works with HSV easier to read,
> so they do not change functionality.
> 
> Regards,
> 
> 	Hans
> 
>>
>> Thanks,
>> Dafna
>>
>>>
>>>> +#define V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION	0x00000002
>>>>    /**
>>>>     * struct v4l2_subdev_mbus_code_enum - Media bus format enumeration
>>>>     * @pad: pad number, as reported by the media API
>>>> @@ -77,7 +79,8 @@ struct v4l2_subdev_mbus_code_enum {
>>>>    	__u32 index;
>>>>    	__u32 code;
>>>>    	__u32 which;
>>>> -	__u32 reserved[8];
>>>> +	__u32 flags;
>>>> +	__u32 reserved[7];
>>>>    };
>>>>    
>>>>    /**
>>>> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
>>>> index 9817b7e2c968..adc9dd1080b8 100644
>>>> --- a/include/uapi/linux/videodev2.h
>>>> +++ b/include/uapi/linux/videodev2.h
>>>> @@ -772,6 +772,7 @@ struct v4l2_pix_format {
>>>>    
>>>>    /* Flags */
>>>>    #define V4L2_PIX_FMT_FLAG_PREMUL_ALPHA	0x00000001
>>>> +#define V4L2_PIX_FMT_FLAG_HAS_CSC	0x00000002
>>>>    
>>>>    /*
>>>>     *	F O R M A T   E N U M E R A T I O N
>>>> @@ -789,6 +790,9 @@ struct v4l2_fmtdesc {
>>>>    #define V4L2_FMT_FLAG_EMULATED			0x0002
>>>>    #define V4L2_FMT_FLAG_CONTINUOUS_BYTESTREAM	0x0004
>>>>    #define V4L2_FMT_FLAG_DYN_RESOLUTION		0x0008
>>>> +#define V4L2_FMT_FLAG_CSC_YCBCR_ENC		0x0010
>>>> +#define V4L2_FMT_FLAG_CSC_HSV_ENC		0x0010
>>>> +#define V4L2_FMT_FLAG_CSC_QUANTIZATION		0x0020
>>>>    
>>>>    	/* Frame Size and frame rate enumeration */
>>>>    /*
>>>>
>>>
>>> I think this is quite good, it's mainly doc improvements and renaming HAS_CSC to REQUEST_CSC.
>>>
>>> That said, I do want to see this supported in at least vivid (vimc would be nice
>>> too!). And v4l2-ctl needs a patch to support this.
>>>
>>> A compliance test for v4l2-compliance would be nice too, but it is not a prerequisite.
>>> I'm not entirely sure what would be the best way to test this. Perhaps a vivid-specific
>>> test might be easiest (i.e., don't test it for all drivers, but just for vivid).
>>>
>>> Regards,
>>>
>>> 	Hans
>>>
> 

Re: [RFC v3 1/2] v4l2: add support for colorspace conversion for video capture

[Hans Verkuil]

On 26/05/2020 14:36, Dafna Hirschfeld wrote:
> 
> 
> On 25.05.20 16:25, Hans Verkuil wrote:
>> On 25/05/2020 15:56, Dafna Hirschfeld wrote:
>>> Hi
>>>
>>> On 07.05.20 15:00, Hans Verkuil wrote:
>>>> On 16/04/2020 16:56, Dafna Hirschfeld wrote:
>>>>> From: Philipp Zabel <p.zabel@pengutronix.de>
>>>>>
>>>>> For video capture it is the driver that reports the colorspace,
>>>>> Y'CbCr/HSV encoding, quantization range and transfer function
>>>>> used by the video, and there is no way to request something
>>>>> different, even though many HDTV receivers have some sort of
>>>>> colorspace conversion capabilities.
>>>>>
>>>>> For output video this feature already exists since the application
>>>>> specifies this information for the video format it will send out, and
>>>>> the transmitter will enable any available CSC if a format conversion has
>>>>> to be performed in order to match the capabilities of the sink.
>>>>>
>>>>> For video capture we propose adding new pix_format flag:
>>>>> V4L2_PIX_FMT_FLAG_HAS_CSC. The flag is set by the application,
>>>>
>>>> That should be V4L2_PIX_FMT_FLAG_REQUEST_CSC, since the application actively
>>>> requests the use of the CSC.
>>> Maybe change to V4L2_PIX_FMT_FLAG_ASK_FOR_CSC so people won't think
>>> it is related to the request API ?
>>
>> How about _SET_CSC? I think that's even better than REQUEST_CSC.
>>
>>>>
>>>>> the driver will interpret the ycbcr_enc/hsv_enc, and quantization fields
>>>>> as the requested colorspace information and will attempt to
>>>>> do the conversion it supports.
>>>>>
>>>>> Drivers set the flags
>>>>> V4L2_FMT_FLAG_CSC_YCBCR_ENC,
>>>>> V4L2_FMT_FLAG_CSC_HSV_ENC,
>>>>> V4L2_FMT_FLAG_CSC_HSV_QUANTIZATION,
>>>>
>>>> That last define should be 'V4L2_FMT_FLAG_CSC_QUANTIZATION' without 'HSV_'.
>>>>
>>>>> in the flags field of the struct v4l2_fmtdesc during enumeration to
>>>>> indicate that they support colorspace conversion for the respective field.
>>>>> Currently the conversion of the fields colorspace, xfer_func is not
>>>>> supported since there are no drivers that support it.
>>>>>
>>>>> The same API is added for the subdevices. With the flag V4L2_MBUS_FRAMEFMT_HAS_CSC
>>>>
>>>> Should also be REQUEST_CSC.
>>>>
>>>>> set by the application in the VIDIOC_SUBDEV_S_FMT ioctl and the flags
>>>>> V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC, V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION
>>>>> set by the driver in the VIDIOC_SUBDEV_ENUM_MBUS_CODE ioctl.
>>>>>
>>>>> For subdevices, new 'flags' fields were added to the structs
>>>>> v4l2_subdev_mbus_code_enum, v4l2_mbus_framefmt which are borrowed from the 'reserved' field
>>>>>
>>>>> Drivers do not have to actually look at the flags: if the flags are not
>>>>> set, then the colorspace, ycbcr_enc and quantization fields are set to
>>>>> the default values by the core, i.e. just pass on the received format
>>>>> without conversion.
>>>>>
>>>>> Signed-off-by: Hans Verkuil <Hans Verkuil@cisco.com>
>>>>> Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
>>>>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
>>>>> ---
>>>>> This is v3 of the RFC suggested originaly by Hans Verkuil:
>>>>>
>>>>>    https://patchwork.linuxtv.org/patch/28847/
>>>>>
>>>>> And then a v2 from Philipp Zabel:
>>>>>
>>>>>    https://patchwork.kernel.org/project/linux-media/list/?series=15483
>>>>>
>>>>> changes in v3:
>>>>> I added the API to subdevices as well and added fixes according to
>>>>> comments from Hans.
>>>>> I also added a usecase for the new API for the rkisp1 driver.
>>>>>
>>>>> changes in v2 (reported by Philipp Zabel):
>>>>>    - convert to rst
>>>>>    - split V4L2_PIX_FMT_FLAG_REQUEST_CSC into four separate flags for
>>>>>      colorspace, ycbcr_enc/hsv_enc, quantization, and xfer_func
>>>>>    - let driver set flags to indicate supported features
>>>>>
>>>>> [1] https://patchwork.linuxtv.org/patch/28847/
>>>>>
>>>>>    .../media/v4l/pixfmt-reserved.rst             |  6 +++
>>>>>    .../media/v4l/pixfmt-v4l2-mplane.rst          | 16 ++-----
>>>>>    .../userspace-api/media/v4l/pixfmt-v4l2.rst   | 36 +++++++++++++---
>>>>>    .../media/v4l/subdev-formats.rst              | 42 +++++++++++++++++--
>>>>>    .../media/v4l/vidioc-enum-fmt.rst             | 18 ++++++++
>>>>>    .../v4l/vidioc-subdev-enum-mbus-code.rst      | 23 ++++++++++
>>>>>    drivers/media/v4l2-core/v4l2-ioctl.c          | 19 ++++++++-
>>>>>    drivers/media/v4l2-core/v4l2-subdev.c         |  7 ++++
>>>>>    include/uapi/linux/v4l2-mediabus.h            |  5 ++-
>>>>>    include/uapi/linux/v4l2-subdev.h              |  5 ++-
>>>>>    include/uapi/linux/videodev2.h                |  4 ++
>>>>>    11 files changed, 158 insertions(+), 23 deletions(-)
>>>>>
>>>>> diff --git a/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst b/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst
>>>>> index 59b9e7238f90..fa8dada69f8c 100644
>>>>> --- a/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst
>>>>> +++ b/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst
>>>>
>>>> Why on earth did the documentation of these pixel format flags end up in
>>>> pixfmt-reserved.rst?
>>>>
>>>> Can you make a patch that moves this to pixfmt-v4l2.rst?
>>>>
>>>>> @@ -280,3 +280,9 @@ please make a proposal on the linux-media mailing list.
>>>>>    	by RGBA values (128, 192, 255, 128), the same pixel described with
>>>>>    	premultiplied colors would be described by RGBA values (64, 96,
>>>>>    	128, 128)
>>>>> +    * - ``V4L2_PIX_FMT_FLAG_HAS_CSC``
>>>>> +      - 0x00000002
>>>>> +      - Set by the application. It is only used for capture and is
>>>>> +	ignored for output streams. If set, then request the driver to do
>>>>> +	colorspace conversion from the received colorspace, only conversions
>>>>> +	of Ycbcr encoding, HSV encoding and quantization are supported.
>>>>
>>>> I'd replace the part "If set..." with this:
>>>>
>>>> ----------
>>>> If set, then request the driver to do colorspace conversion from the received
>>>> colorspace to the requested colorspace values. If a colorimetry field
>>>> (``colorspace``, ``xfer_func``, ``ycncr_enc`` or ``quantization``) is set to 0,
>>>> then that colorimetry setting will remain unchanged from what was received.
>>>> So to just change the quantization only the ``quantization`` field shall be set
>>>> to a non-zero value (``V4L2_QUANTIZATION_FULL_RANGE`` or
>>>> ``V4L2_QUANTIZATION_LIM_RANGE``) and all other colorimetry fields shall be set to 0.
>>>>
>>>> To check which conversions are supported by the hardware for the current pixel
>>>> format, see :ref:`fmtdesc-flags`.
>>>> ----------
>>>>
>>>>
>>>>> diff --git a/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst b/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst
>>>>> index 444b4082684c..66f3365d7b72 100644
>>>>> --- a/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst
>>>>> +++ b/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst
>>>>> @@ -105,29 +105,21 @@ describing all planes of that format.
>>>>>        * - __u8
>>>>>          - ``ycbcr_enc``
>>>>>          - Y'CbCr encoding, from enum :c:type:`v4l2_ycbcr_encoding`.
>>>>> -        This information supplements the ``colorspace`` and must be set by
>>>>> -	the driver for capture streams and by the application for output
>>>>> -	streams, see :ref:`colorspaces`.
>>>>> +	See struct :c:type:`v4l2_pix_format`.
>>>>>        * - __u8
>>>>>          - ``hsv_enc``
>>>>>          - HSV encoding, from enum :c:type:`v4l2_hsv_encoding`.
>>>>> -        This information supplements the ``colorspace`` and must be set by
>>>>> -	the driver for capture streams and by the application for output
>>>>> -	streams, see :ref:`colorspaces`.
>>>>> +	See struct :c:type:`v4l2_pix_format`.
>>>>>        * - }
>>>>>          -
>>>>>        * - __u8
>>>>>          - ``quantization``
>>>>>          - Quantization range, from enum :c:type:`v4l2_quantization`.
>>>>> -        This information supplements the ``colorspace`` and must be set by
>>>>> -	the driver for capture streams and by the application for output
>>>>> -	streams, see :ref:`colorspaces`.
>>>>> +	See struct :c:type:`v4l2_pix_format`.
>>>>>        * - __u8
>>>>>          - ``xfer_func``
>>>>>          - Transfer function, from enum :c:type:`v4l2_xfer_func`.
>>>>> -        This information supplements the ``colorspace`` and must be set by
>>>>> -	the driver for capture streams and by the application for output
>>>>> -	streams, see :ref:`colorspaces`.
>>>>> +	See struct :c:type:`v4l2_pix_format`.
>>>>>        * - __u8
>>>>>          - ``reserved[7]``
>>>>>          - Reserved for future extensions. Should be zeroed by drivers and
>>>>> diff --git a/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst b/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst
>>>>> index 759420a872d6..ce57718cd66b 100644
>>>>> --- a/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst
>>>>> +++ b/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst
>>>>> @@ -110,8 +110,8 @@ Single-planar format structure
>>>>>          - ``colorspace``
>>>>>          - Image colorspace, from enum :c:type:`v4l2_colorspace`.
>>>>>            This information supplements the ``pixelformat`` and must be set
>>>>> -	by the driver for capture streams and by the application for
>>>>> -	output streams, see :ref:`colorspaces`.
>>>>> +	by the driver for capture streams and by the application for output
>>>>> +	streams, see :ref:`colorspace`.
>>>>
>>>> This doesn't appear to change anything.
>>>>
>>>>>        * - __u32
>>>>>          - ``priv``
>>>>>          - This field indicates whether the remaining fields of the
>>>>> @@ -148,13 +148,31 @@ Single-planar format structure
>>>>>          - Y'CbCr encoding, from enum :c:type:`v4l2_ycbcr_encoding`.
>>>>>            This information supplements the ``colorspace`` and must be set by
>>>>>    	the driver for capture streams and by the application for output
>>>>> -	streams, see :ref:`colorspaces`.
>>>>> +	streams, see :ref:`colorspaces`. If the application sets the
>>>>> +	flag ``V4L2_PIX_FMT_FLAG_HAS_CSC`` then the application can set
>>>>> +	this field for a capture stream to request a specific Y'CbCr encoding
>>>>> +	for the captured image data. The driver will attempt to do the
>>>>> +	conversion to the specified Y'CbCr encoding or return the encoding it
>>>>> +	will use if it can't do the conversion. This field is ignored for
>>>>> +	non-Y'CbCr pixelformats. The driver indicates that ycbcr_enc conversion
>>>>> +	is supported by setting the flag V4L2_FMT_FLAG_CSC_YCBCR_ENC on the
>>>>> +	on the corresponding struct :c:type:`v4l2_fmtdesc` during enumeration.
>>>>
>>>> on the on the -> in the
>>>>
>>>>> +	See :ref:`fmtdesc-flags`
>>>>>        * - __u32
>>>>>          - ``hsv_enc``
>>>>>          - HSV encoding, from enum :c:type:`v4l2_hsv_encoding`.
>>>>>            This information supplements the ``colorspace`` and must be set by
>>>>>    	the driver for capture streams and by the application for output
>>>>> -	streams, see :ref:`colorspaces`.
>>>>> +	streams, see :ref:`colorspaces`. If the application sets the flag
>>>>> +	``V4L2_PIX_FMT_FLAG_HAS_CSC`` then the application can set this
>>>>> +	field for a capture stream to request a specific HSV encoding for the
>>>>> +	captured image data. The driver will attempt to do the conversion to
>>>>> +	the specified HSV encoding or return the encoding it will use if it
>>>>> +	can't do the conversion. This field is ignored for non-HSV
>>>>> +	pixelformats. The driver indicates that hsv_enc conversion
>>>>> +	is supported by setting the flag V4L2_FMT_FLAG_CSC_HSV_ENC on the
>>>>> +	on the corresponding struct :c:type:`v4l2_fmtdesc` during enumeration.
>>>>
>>>> Ditto.
>>>>
>>>>> +	See :ref:`fmtdesc-flags`
>>>>>        * - }
>>>>>          -
>>>>>        * - __u32
>>>>> @@ -162,7 +180,15 @@ Single-planar format structure
>>>>>          - Quantization range, from enum :c:type:`v4l2_quantization`.
>>>>>            This information supplements the ``colorspace`` and must be set by
>>>>>    	the driver for capture streams and by the application for output
>>>>> -	streams, see :ref:`colorspaces`.
>>>>> +	streams, see :ref:`colorspaces`. If the application sets the flag
>>>>> +	``V4L2_PIX_FMT_FLAG_HAS_CSC`` then the application can set
>>>>> +	this field for a capture stream to request a specific quantization
>>>>> +	range for the captured image data. The driver will attempt to do the
>>>>> +	conversion to the specified quantization range or return the
>>>>> +	quantization it will use if it can't do the conversion.  The driver
>>>>> +	indicates that quantization conversion is supported by setting the flag
>>>>> +	V4L2_FMT_FLAG_CSC_QUANTIZATION on the on the corresponding struct
>>>>
>>>> Ditto
>>>>
>>>>> +	:c:type:`v4l2_fmtdesc` during enumeration. See :ref:`fmtdesc-flags`
>>>>>        * - __u32
>>>>>          - ``xfer_func``
>>>>>          - Transfer function, from enum :c:type:`v4l2_xfer_func`.
>>>>> diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
>>>>> index 9a4d61b0d76f..f1d4ca29a3e8 100644
>>>>> --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
>>>>> +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
>>>>> @@ -49,13 +49,33 @@ Media Bus Formats
>>>>>          - Y'CbCr encoding, from enum :c:type:`v4l2_ycbcr_encoding`.
>>>>>            This information supplements the ``colorspace`` and must be set by
>>>>>    	the driver for capture streams and by the application for output
>>>>> -	streams, see :ref:`colorspaces`.
>>>>> +	streams, see :ref:`colorspaces`. If the application sets the
>>>>> +	flag ``V4L2_MBUS_FRAMEFMT_HAS_CSC`` the the application can set
>>>>> +	this field for a capture stream to request a specific Y'CbCr encoding
>>>>> +	for the mbus data. The driver will attempt to do the
>>>>
>>>> mbus -> media bus
>>>>
>>>>> +	conversion to the specified Y'CbCr encoding or return the encoding it
>>>>> +	will use if it can't do the conversion. This field is ignored for
>>>>> +	non-Y'CbCr pixelformats. The driver indicates that ycbcr_enc conversion
>>>>
>>>> This is a media bus format, not a pixelformat.
>>>>
>>>>> +	is supported by setting the flag V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC on
>>>>
>>>> on -> in
>>>>
>>>>> +	the corresponding struct c:type:`v4l2_subdev_mbus_code_enum` during
>>>>> +	enumeration. See :ref:`vidioc-subdev-enum-mbus-code.rst <v4l2-subdev-mbus-code-flags>`
>>>>> +
>>>>>        * - __u16
>>>>>          - ``quantization``
>>>>>          - Quantization range, from enum :c:type:`v4l2_quantization`.
>>>>>            This information supplements the ``colorspace`` and must be set by
>>>>>    	the driver for capture streams and by the application for output
>>>>> -	streams, see :ref:`colorspaces`.
>>>>> +	streams, see :ref:`colorspaces`. If the application sets the
>>>>> +	flag ``V4L2_MBUS_FRAMEFMT_HAS_CSC`` the the application can set
>>>>> +	this field for a capture stream to request a specific quantization
>>>>> +	encoding for the mbus data. The driver will attempt to do the
>>>>
>>>> mbus -> media bus
>>>>
>>>>> +	conversion to the specified quantization or return the quantization it
>>>>> +	will use if it can't do the conversion. The driver indicates that
>>>>> +	quantization conversion is supported by setting the flag
>>>>> +	V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION on the corresponding struct
>>>>
>>>> on -> in
>>>>
>>>>> +	c:type:`v4l2_subdev_mbus_code_enum` during enumeration.
>>>>> +	See :ref:`vidioc-subdev-enum-mbus-code.rst <v4l2-subdev-mbus-code-flags>`
>>>>> +
>>>>>        * - __u16
>>>>>          - ``xfer_func``
>>>>>          - Transfer function, from enum :c:type:`v4l2_xfer_func`.
>>>>> @@ -63,10 +83,26 @@ Media Bus Formats
>>>>>    	the driver for capture streams and by the application for output
>>>>>    	streams, see :ref:`colorspaces`.
>>>>>        * - __u16
>>>>> -      - ``reserved``\ [11]
>>>>> +      - ``flags``
>>>>> +      - flags see:  :ref:v4l2-mbus-framefmt-flags
>>>>
>>>> flags see: -> See:
>>>>
>>>>> +    * - __u16
>>>>> +      - ``reserved``\ [10]
>>>>>          - Reserved for future extensions. Applications and drivers must set
>>>>>    	the array to zero.
>>>>>    
>>>>> +.. _v4l2-mbus-framefmt-flags:
>>>>> +
>>>>> +.. flat-table:: v4l2_mbus_framefmt Flags
>>>>> +    :header-rows:  0
>>>>> +    :stub-columns: 0
>>>>> +    :widths:       3 1 4
>>>>> +
>>>>> +    * - ``V4L2_MBUS_FRAMEFMT_HAS_CSC``
>>>>> +      - 0x0001
>>>>> +      - Set by the application. It is only used for capture and is
>>>>> +	ignored for output streams. If set, then request the subdevice to do
>>>>> +	colorspace conversion from the received colorspace, only conversions
>>>>> +	of Ycbcr encoding, and quantization are supported.
>>>>
>>>> Replace by a similar text as for V4L2_PIX_FMT_FLAG_HAS_CSC.
>>>>
>>>>>    
>>>>>    
>>>>>    .. _v4l2-mbus-pixelcode:
>>>>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
>>>>> index 7e3142e11d77..125f074543af 100644
>>>>> --- a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
>>>>> +++ b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
>>>>> @@ -145,6 +145,24 @@ formats in preference order, where preferred formats are returned before
>>>>>    	parameters are detected. This flag can only be used in combination
>>>>>    	with the ``V4L2_FMT_FLAG_COMPRESSED`` flag, since this applies to
>>>>>    	compressed formats only. It is also only applies to stateful codecs.
>>>>> +    * - ``V4L2_FMT_FLAG_CSC_YCBCR_ENC``
>>>>> +      - 0x0010
>>>>> +      - The driver allows the application to try to change the default
>>>>> +	ycbcr encoding. This flag is relevant only for capture devices.
>>>>> +	The application can ask to configure the ycbcr_enc of the capture device
>>>>> +	when calling the :c:func:`VIDIOC_S_FMT` ioctl.
>>>>
>>>> add: ...ioctl with ``V4L2_FMT_FLAG_REQUEST_CSC`` set. (this should probably be
>>>> a reference to the REQUEST_CSC flag documentation).
>>>>
>>>>> +    * - ``V4L2_FMT_FLAG_CSC_HSV_ENC``
>>>>> +      - 0x0010
>>>>> +      - The driver allows the application to try to change the default
>>>>> +	hsv encoding. This flag is relevant only for capture devices.
>>>>> +	The application can ask to configure the hsv_enc of the capture device
>>>>> +	when calling the :c:func:`VIDIOC_S_FMT` ioctl.
>>>>
>>>> Ditto. Also: hsv -> HSV.
>>>>
>>>>> +    * - ``V4L2_FMT_FLAG_CSC_QUANTIZATION``
>>>>> +      - 0x0020
>>>>> +      - The driver allows the application to try to change the default
>>>>> +	quantization. This flag is relevant only for capture devices.
>>>>> +	The application can ask to configure the quantization of the capture device
>>>>> +	when calling the :c:func:`VIDIOC_S_FMT` ioctl.
>>>>
>>>> Ditto.
>>>>
>>>>>    
>>>>>    
>>>>>    Return Value
>>>>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
>>>>> index 35b8607203a4..4ad87cb74f57 100644
>>>>> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
>>>>> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
>>>>> @@ -78,11 +78,34 @@ information about the try formats.
>>>>>          - ``which``
>>>>>          - Media bus format codes to be enumerated, from enum
>>>>>    	:ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
>>>>> +    * - __u32
>>>>> +      - ``flags``
>>>>> +      - see :ref:`v4l2-subdev-mbus-code-flags`
>>>>
>>>> see -> See
>>>>
>>>>>        * - __u32
>>>>>          - ``reserved``\ [8]
>>>>>          - Reserved for future extensions. Applications and drivers must set
>>>>>    	the array to zero.
>>>>>    
>>>>> +.. _v4l2-subdev-mbus-code-flags:
>>>>> +
>>>>> +
>>>>> +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{7.7cm}|
>>>>> +
>>>>> +.. flat-table:: flags in struct v4l2_subdev_mbus_code_enum
>>>>> +    :header-rows:  0
>>>>> +    :stub-columns: 0
>>>>> +    :widths:       1 1 2
>>>>> +
>>>>> +    * - V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC
>>>>> +      - 0x00000001
>>>>> +      - The driver supports setting the ycbcr encoding by the application
>>>>> +	when calling the VIDIOC_SUBDEV_S_FMT ioctl. see :ref:`v4l2-mbus-format`
>>>>
>>>> see -> See
>>>>
>>>>> +	on how to do this.
>>>>> +    * - V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION
>>>>> +      - 0x00000002
>>>>> +      - The driver supports setting the quantization by the application
>>>>> +	when calling the VIDIOC_SUBDEV_S_FMT ioctl. see :ref:`v4l2-mbus-format`
>>>>
>>>> see -> See
>>>>
>>>>> +	on how to do this.
>>>>>    
>>>>>    Return Value
>>>>>    ============
>>>>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
>>>>> index b2ef8e60ea7d..3c7ffb6b15cb 100644
>>>>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
>>>>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
>>>>> @@ -1029,6 +1029,15 @@ static void v4l_sanitize_format(struct v4l2_format *fmt)
>>>>>    		fmt->fmt.pix_mp.num_planes = min_t(u32, fmt->fmt.pix_mp.num_planes,
>>>>>    					       VIDEO_MAX_PLANES);
>>>>>    
>>>>> +	if (fmt->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE) {
>>>>> +		if (fmt->fmt.pix_mp.flags & V4L2_PIX_FMT_FLAG_HAS_CSC)
>>>>> +			return;
>>>>> +		fmt->fmt.pix_mp.colorspace = V4L2_COLORSPACE_DEFAULT;
>>>>> +		fmt->fmt.pix_mp.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>>>>> +		fmt->fmt.pix_mp.quantization = V4L2_QUANTIZATION_DEFAULT;
>>>>> +		fmt->fmt.pix_mp.xfer_func = V4L2_XFER_FUNC_DEFAULT;
>>>>> +	}
>>>>> +
>>>>>    	/*
>>>>>    	 * The v4l2_pix_format structure has been extended with fields that were
>>>>>    	 * not previously required to be set to zero by applications. The priv
>>>>> @@ -1043,8 +1052,16 @@ static void v4l_sanitize_format(struct v4l2_format *fmt)
>>>>>    	    fmt->type != V4L2_BUF_TYPE_VIDEO_OUTPUT)
>>>>>    		return;
>>>>>    
>>>>> -	if (fmt->fmt.pix.priv == V4L2_PIX_FMT_PRIV_MAGIC)
>>>>> +	if (fmt->fmt.pix.priv == V4L2_PIX_FMT_PRIV_MAGIC) {
>>>>> +		if (fmt->type != V4L2_BUF_TYPE_VIDEO_CAPTURE ||
>>>>> +		    fmt->fmt.pix.flags & V4L2_PIX_FMT_FLAG_HAS_CSC)
>>>>> +			return;
>>>>> +		fmt->fmt.pix.colorspace = V4L2_COLORSPACE_DEFAULT;
>>>>> +		fmt->fmt.pix.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>>>>> +		fmt->fmt.pix.quantization = V4L2_QUANTIZATION_DEFAULT;
>>>>> +		fmt->fmt.pix.xfer_func = V4L2_XFER_FUNC_DEFAULT;
>>>>>    		return;
>>>>> +	}
>>>>>    
>>>>>    	fmt->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>>>>>    
>>>>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>>>>> index a376b351135f..51e009936aad 100644
>>>>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>>>>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>>>>> @@ -477,6 +477,13 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>>>>>    	case VIDIOC_SUBDEV_S_FMT: {
>>>>>    		struct v4l2_subdev_format *format = arg;
>>>>>    
>>>>> +		if (!(format->format.flags & V4L2_MBUS_FRAMEFMT_HAS_CSC)) {
>>>>> +			format->format.colorspace = V4L2_COLORSPACE_DEFAULT;
>>>>> +			format->format.xfer_func = V4L2_XFER_FUNC_DEFAULT;
>>>>> +			format->format.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>>>>> +			format->format.quantization = V4L2_QUANTIZATION_DEFAULT;
>>>>> +		}
> I wonder if those 3 code chunks added to v4l2-ioctl.c and v4l2-subdev.c are actually
> needed. They set the colorspace fields on capture devices to 0 if the HAS_CSC flag is not set.
> But I don't see the reason to do it, if the capture driver do not support CSC it will just ignore
> the values of those fields that came from userspace which is what is already done.
> Also, it could be that a driver does not support CSC and userspace set the SET_CSC flag anyway, in this
> case those fields will not be set to defalut although they should.
> What do you think?

I agree. That code doesn't make sense.

I think it is safe to drop this.

Regards,

	Hans

> 
> Thanks,
> Dafna
> 
>>>>> +
>>>>>    		memset(format->reserved, 0, sizeof(format->reserved));
>>>>>    		memset(format->format.reserved, 0, sizeof(format->format.reserved));
>>>>>    		return v4l2_subdev_call(sd, pad, set_fmt, subdev_fh->pad, format);
>>>>> diff --git a/include/uapi/linux/v4l2-mediabus.h b/include/uapi/linux/v4l2-mediabus.h
>>>>> index 123a231001a8..89ff0ad6a631 100644
>>>>> --- a/include/uapi/linux/v4l2-mediabus.h
>>>>> +++ b/include/uapi/linux/v4l2-mediabus.h
>>>>> @@ -16,6 +16,8 @@
>>>>>    #include <linux/types.h>
>>>>>    #include <linux/videodev2.h>
>>>>>    
>>>>> +#define V4L2_MBUS_FRAMEFMT_HAS_CSC	0x0001
>>>>> +
>>>>>    /**
>>>>>     * struct v4l2_mbus_framefmt - frame format on the media bus
>>>>>     * @width:	image width
>>>>> @@ -36,7 +38,8 @@ struct v4l2_mbus_framefmt {
>>>>>    	__u16			ycbcr_enc;
>>>>>    	__u16			quantization;
>>>>>    	__u16			xfer_func;
>>>>> -	__u16			reserved[11];
>>>>> +	__u16			flags;
>>>>> +	__u16			reserved[10];
>>>>>    };
>>>>>    
>>>>>    #ifndef __KERNEL__
>>>>> diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
>>>>> index 03970ce30741..4410b26a7158 100644
>>>>> --- a/include/uapi/linux/v4l2-subdev.h
>>>>> +++ b/include/uapi/linux/v4l2-subdev.h
>>>>> @@ -65,6 +65,8 @@ struct v4l2_subdev_crop {
>>>>>    	__u32 reserved[8];
>>>>>    };
>>>>>    
>>>>> +#define V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC	0x00000001
>>>>
>>>> I think it is better to add an HSV_ENC alias here as well.
>>>>
>>>> It's for readability only, since talking about YCBCR_ENC when the mediabus format
>>>> is a HSV format is just weird.
>>>
>>> I see that in 'v4l2_pix_format' the {ycbcr_enc, hsv_enc} are a union
>>> but in 'v4l2_mbus_framefmt' there is only the field ycbcr_enc.
>>> I wonder why is that, since I see there is one HSV media bus format
>>> V4L2_MBUS_FROM_MEDIA_BUS_FMT(AHSV8888_1X32).
>>>
>>> So I guess this is why I didn't add a V4L2_SUBDEV_MBUS_CODE_CSC_HSV_ENC define.
>>> I also didn't add this flag to the docs. Should I?
>>
>> I think this can be done in a separate (final) patch where a union is added for
>> v4l2_mbus_framefmt together with a V4L2_SUBDEV_MBUS_CODE_CSC_HSV_ENC define.
>>
>> These are effectively aliases to make code that works with HSV easier to read,
>> so they do not change functionality.
>>
>> Regards,
>>
>> 	Hans
>>
>>>
>>> Thanks,
>>> Dafna
>>>
>>>>
>>>>> +#define V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION	0x00000002
>>>>>    /**
>>>>>     * struct v4l2_subdev_mbus_code_enum - Media bus format enumeration
>>>>>     * @pad: pad number, as reported by the media API
>>>>> @@ -77,7 +79,8 @@ struct v4l2_subdev_mbus_code_enum {
>>>>>    	__u32 index;
>>>>>    	__u32 code;
>>>>>    	__u32 which;
>>>>> -	__u32 reserved[8];
>>>>> +	__u32 flags;
>>>>> +	__u32 reserved[7];
>>>>>    };
>>>>>    
>>>>>    /**
>>>>> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
>>>>> index 9817b7e2c968..adc9dd1080b8 100644
>>>>> --- a/include/uapi/linux/videodev2.h
>>>>> +++ b/include/uapi/linux/videodev2.h
>>>>> @@ -772,6 +772,7 @@ struct v4l2_pix_format {
>>>>>    
>>>>>    /* Flags */
>>>>>    #define V4L2_PIX_FMT_FLAG_PREMUL_ALPHA	0x00000001
>>>>> +#define V4L2_PIX_FMT_FLAG_HAS_CSC	0x00000002
>>>>>    
>>>>>    /*
>>>>>     *	F O R M A T   E N U M E R A T I O N
>>>>> @@ -789,6 +790,9 @@ struct v4l2_fmtdesc {
>>>>>    #define V4L2_FMT_FLAG_EMULATED			0x0002
>>>>>    #define V4L2_FMT_FLAG_CONTINUOUS_BYTESTREAM	0x0004
>>>>>    #define V4L2_FMT_FLAG_DYN_RESOLUTION		0x0008
>>>>> +#define V4L2_FMT_FLAG_CSC_YCBCR_ENC		0x0010
>>>>> +#define V4L2_FMT_FLAG_CSC_HSV_ENC		0x0010
>>>>> +#define V4L2_FMT_FLAG_CSC_QUANTIZATION		0x0020
>>>>>    
>>>>>    	/* Frame Size and frame rate enumeration */
>>>>>    /*
>>>>>
>>>>
>>>> I think this is quite good, it's mainly doc improvements and renaming HAS_CSC to REQUEST_CSC.
>>>>
>>>> That said, I do want to see this supported in at least vivid (vimc would be nice
>>>> too!). And v4l2-ctl needs a patch to support this.
>>>>
>>>> A compliance test for v4l2-compliance would be nice too, but it is not a prerequisite.
>>>> I'm not entirely sure what would be the best way to test this. Perhaps a vivid-specific
>>>> test might be easiest (i.e., don't test it for all drivers, but just for vivid).
>>>>
>>>> Regards,
>>>>
>>>> 	Hans
>>>>
>>

Re: [RFC v3 1/2] v4l2: add support for colorspace conversion for video capture

[Tomasz Figa]

Hi Dafna,

On Thu, Apr 16, 2020 at 04:56:04PM +0200, Dafna Hirschfeld wrote:
> From: Philipp Zabel <p.zabel@pengutronix.de>
> 
> For video capture it is the driver that reports the colorspace,
> Y'CbCr/HSV encoding, quantization range and transfer function
> used by the video, and there is no way to request something
> different, even though many HDTV receivers have some sort of
> colorspace conversion capabilities.
> 

Thanks for working on this! Please see my comments inline.

> For output video this feature already exists since the application
> specifies this information for the video format it will send out, and
> the transmitter will enable any available CSC if a format conversion has
> to be performed in order to match the capabilities of the sink.
> 
> For video capture we propose adding new pix_format flag:
> V4L2_PIX_FMT_FLAG_HAS_CSC. The flag is set by the application,
> the driver will interpret the ycbcr_enc/hsv_enc, and quantization fields
> as the requested colorspace information and will attempt to
> do the conversion it supports.
> 
> Drivers set the flags
> V4L2_FMT_FLAG_CSC_YCBCR_ENC,
> V4L2_FMT_FLAG_CSC_HSV_ENC,
> V4L2_FMT_FLAG_CSC_HSV_QUANTIZATION,

Do we need this level of granularity? The drivers would ignore
unsupported encoding/quantization settings and reset them to supported
ones anyway, so if one doesn't support changing given parameter, it
would just return back the original stream parameter.

> in the flags field of the struct v4l2_fmtdesc during enumeration to
> indicate that they support colorspace conversion for the respective field.
> Currently the conversion of the fields colorspace, xfer_func is not
> supported since there are no drivers that support it.
> 
> The same API is added for the subdevices. With the flag V4L2_MBUS_FRAMEFMT_HAS_CSC
> set by the application in the VIDIOC_SUBDEV_S_FMT ioctl and the flags
> V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC, V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION
> set by the driver in the VIDIOC_SUBDEV_ENUM_MBUS_CODE ioctl.
> 
> For subdevices, new 'flags' fields were added to the structs
> v4l2_subdev_mbus_code_enum, v4l2_mbus_framefmt which are borrowed from the 'reserved' field
> 
> Drivers do not have to actually look at the flags: if the flags are not
> set, then the colorspace, ycbcr_enc and quantization fields are set to
> the default values by the core, i.e. just pass on the received format
> without conversion.
> 
> Signed-off-by: Hans Verkuil <Hans Verkuil@cisco.com>

Something wrong with the email address here.

> Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
> ---
> This is v3 of the RFC suggested originaly by Hans Verkuil:
> 
>  https://patchwork.linuxtv.org/patch/28847/
> 
> And then a v2 from Philipp Zabel:
> 
>  https://patchwork.kernel.org/project/linux-media/list/?series=15483
> 
> changes in v3:
> I added the API to subdevices as well and added fixes according to
> comments from Hans.
> I also added a usecase for the new API for the rkisp1 driver.
> 
> changes in v2 (reported by Philipp Zabel):
>  - convert to rst
>  - split V4L2_PIX_FMT_FLAG_REQUEST_CSC into four separate flags for
>    colorspace, ycbcr_enc/hsv_enc, quantization, and xfer_func
>  - let driver set flags to indicate supported features
> 
> [1] https://patchwork.linuxtv.org/patch/28847/
> 
>  .../media/v4l/pixfmt-reserved.rst             |  6 +++
>  .../media/v4l/pixfmt-v4l2-mplane.rst          | 16 ++-----
>  .../userspace-api/media/v4l/pixfmt-v4l2.rst   | 36 +++++++++++++---
>  .../media/v4l/subdev-formats.rst              | 42 +++++++++++++++++--
>  .../media/v4l/vidioc-enum-fmt.rst             | 18 ++++++++
>  .../v4l/vidioc-subdev-enum-mbus-code.rst      | 23 ++++++++++
>  drivers/media/v4l2-core/v4l2-ioctl.c          | 19 ++++++++-
>  drivers/media/v4l2-core/v4l2-subdev.c         |  7 ++++
>  include/uapi/linux/v4l2-mediabus.h            |  5 ++-
>  include/uapi/linux/v4l2-subdev.h              |  5 ++-
>  include/uapi/linux/videodev2.h                |  4 ++
>  11 files changed, 158 insertions(+), 23 deletions(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst b/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst
> index 59b9e7238f90..fa8dada69f8c 100644
> --- a/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst
> +++ b/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst
> @@ -280,3 +280,9 @@ please make a proposal on the linux-media mailing list.
>  	by RGBA values (128, 192, 255, 128), the same pixel described with
>  	premultiplied colors would be described by RGBA values (64, 96,
>  	128, 128)
> +    * - ``V4L2_PIX_FMT_FLAG_HAS_CSC``
> +      - 0x00000002
> +      - Set by the application. It is only used for capture and is
> +	ignored for output streams. If set, then request the driver to do
> +	colorspace conversion from the received colorspace, only conversions

Should this be:

  colorspace. Only conversions

?

> +	of Ycbcr encoding, HSV encoding and quantization are supported.

YCbCr?

> diff --git a/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst b/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst
> index 444b4082684c..66f3365d7b72 100644
> --- a/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst
> +++ b/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst
> @@ -105,29 +105,21 @@ describing all planes of that format.
>      * - __u8
>        - ``ycbcr_enc``
>        - Y'CbCr encoding, from enum :c:type:`v4l2_ycbcr_encoding`.
> -        This information supplements the ``colorspace`` and must be set by
> -	the driver for capture streams and by the application for output
> -	streams, see :ref:`colorspaces`.
> +	See struct :c:type:`v4l2_pix_format`.
>      * - __u8
>        - ``hsv_enc``
>        - HSV encoding, from enum :c:type:`v4l2_hsv_encoding`.
> -        This information supplements the ``colorspace`` and must be set by
> -	the driver for capture streams and by the application for output
> -	streams, see :ref:`colorspaces`.
> +	See struct :c:type:`v4l2_pix_format`.
>      * - }
>        -
>      * - __u8
>        - ``quantization``
>        - Quantization range, from enum :c:type:`v4l2_quantization`.
> -        This information supplements the ``colorspace`` and must be set by
> -	the driver for capture streams and by the application for output
> -	streams, see :ref:`colorspaces`.
> +	See struct :c:type:`v4l2_pix_format`.
>      * - __u8
>        - ``xfer_func``
>        - Transfer function, from enum :c:type:`v4l2_xfer_func`.
> -        This information supplements the ``colorspace`` and must be set by
> -	the driver for capture streams and by the application for output
> -	streams, see :ref:`colorspaces`.
> +	See struct :c:type:`v4l2_pix_format`.
>      * - __u8
>        - ``reserved[7]``
>        - Reserved for future extensions. Should be zeroed by drivers and
> diff --git a/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst b/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst
> index 759420a872d6..ce57718cd66b 100644
> --- a/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst
> +++ b/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst
> @@ -110,8 +110,8 @@ Single-planar format structure
>        - ``colorspace``
>        - Image colorspace, from enum :c:type:`v4l2_colorspace`.
>          This information supplements the ``pixelformat`` and must be set
> -	by the driver for capture streams and by the application for
> -	output streams, see :ref:`colorspaces`.
> +	by the driver for capture streams and by the application for output
> +	streams, see :ref:`colorspace`.

Is it expected that the colorspace itself can't be converted? Also is
the change of the reference name expected?

>      * - __u32
>        - ``priv``
>        - This field indicates whether the remaining fields of the
> @@ -148,13 +148,31 @@ Single-planar format structure
>        - Y'CbCr encoding, from enum :c:type:`v4l2_ycbcr_encoding`.
>          This information supplements the ``colorspace`` and must be set by
>  	the driver for capture streams and by the application for output
> -	streams, see :ref:`colorspaces`.
> +	streams, see :ref:`colorspaces`. If the application sets the
> +	flag ``V4L2_PIX_FMT_FLAG_HAS_CSC`` then the application can set
> +	this field for a capture stream to request a specific Y'CbCr encoding
> +	for the captured image data. The driver will attempt to do the
> +	conversion to the specified Y'CbCr encoding or return the encoding it
> +	will use if it can't do the conversion. This field is ignored for

nit: The driver will attempt to do the conversion when the streaming
starts, so that's not an entirely correct description. How about

"[...] captured image data. If the driver cannot handle requested
conversion, it will return another, supported encoding."

?

> +	non-Y'CbCr pixelformats. The driver indicates that ycbcr_enc conversion
> +	is supported by setting the flag V4L2_FMT_FLAG_CSC_YCBCR_ENC on the
> +	on the corresponding struct :c:type:`v4l2_fmtdesc` during enumeration.
> +	See :ref:`fmtdesc-flags`
>      * - __u32
>        - ``hsv_enc``
>        - HSV encoding, from enum :c:type:`v4l2_hsv_encoding`.
>          This information supplements the ``colorspace`` and must be set by
>  	the driver for capture streams and by the application for output
> -	streams, see :ref:`colorspaces`.
> +	streams, see :ref:`colorspaces`. If the application sets the flag
> +	``V4L2_PIX_FMT_FLAG_HAS_CSC`` then the application can set this
> +	field for a capture stream to request a specific HSV encoding for the
> +	captured image data. The driver will attempt to do the conversion to
> +	the specified HSV encoding or return the encoding it will use if it
> +	can't do the conversion. This field is ignored for non-HSV

Ditto.

> +	pixelformats. The driver indicates that hsv_enc conversion
> +	is supported by setting the flag V4L2_FMT_FLAG_CSC_HSV_ENC on the
> +	on the corresponding struct :c:type:`v4l2_fmtdesc` during enumeration.
> +	See :ref:`fmtdesc-flags`
>      * - }
>        -
>      * - __u32
> @@ -162,7 +180,15 @@ Single-planar format structure
>        - Quantization range, from enum :c:type:`v4l2_quantization`.
>          This information supplements the ``colorspace`` and must be set by
>  	the driver for capture streams and by the application for output
> -	streams, see :ref:`colorspaces`.
> +	streams, see :ref:`colorspaces`. If the application sets the flag
> +	``V4L2_PIX_FMT_FLAG_HAS_CSC`` then the application can set
> +	this field for a capture stream to request a specific quantization
> +	range for the captured image data. The driver will attempt to do the
> +	conversion to the specified quantization range or return the
> +	quantization it will use if it can't do the conversion.  The driver

Ditto.

> +	indicates that quantization conversion is supported by setting the flag
> +	V4L2_FMT_FLAG_CSC_QUANTIZATION on the on the corresponding struct
> +	:c:type:`v4l2_fmtdesc` during enumeration. See :ref:`fmtdesc-flags`
>      * - __u32
>        - ``xfer_func``
>        - Transfer function, from enum :c:type:`v4l2_xfer_func`.
> diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> index 9a4d61b0d76f..f1d4ca29a3e8 100644
> --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
> +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
> @@ -49,13 +49,33 @@ Media Bus Formats
>        - Y'CbCr encoding, from enum :c:type:`v4l2_ycbcr_encoding`.
>          This information supplements the ``colorspace`` and must be set by
>  	the driver for capture streams and by the application for output
> -	streams, see :ref:`colorspaces`.
> +	streams, see :ref:`colorspaces`. If the application sets the
> +	flag ``V4L2_MBUS_FRAMEFMT_HAS_CSC`` the the application can set
> +	this field for a capture stream to request a specific Y'CbCr encoding
> +	for the mbus data. The driver will attempt to do the
> +	conversion to the specified Y'CbCr encoding or return the encoding it
> +	will use if it can't do the conversion. This field is ignored for

Ditto.

> +	non-Y'CbCr pixelformats. The driver indicates that ycbcr_enc conversion
> +	is supported by setting the flag V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC on
> +	the corresponding struct c:type:`v4l2_subdev_mbus_code_enum` during
> +	enumeration. See :ref:`vidioc-subdev-enum-mbus-code.rst <v4l2-subdev-mbus-code-flags>`
> +
>      * - __u16
>        - ``quantization``
>        - Quantization range, from enum :c:type:`v4l2_quantization`.
>          This information supplements the ``colorspace`` and must be set by
>  	the driver for capture streams and by the application for output
> -	streams, see :ref:`colorspaces`.
> +	streams, see :ref:`colorspaces`. If the application sets the
> +	flag ``V4L2_MBUS_FRAMEFMT_HAS_CSC`` the the application can set
> +	this field for a capture stream to request a specific quantization
> +	encoding for the mbus data. The driver will attempt to do the
> +	conversion to the specified quantization or return the quantization it
> +	will use if it can't do the conversion. The driver indicates that

Ditto.

> +	quantization conversion is supported by setting the flag
> +	V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION on the corresponding struct
> +	c:type:`v4l2_subdev_mbus_code_enum` during enumeration.
> +	See :ref:`vidioc-subdev-enum-mbus-code.rst <v4l2-subdev-mbus-code-flags>`
> +
>      * - __u16
>        - ``xfer_func``
>        - Transfer function, from enum :c:type:`v4l2_xfer_func`.
> @@ -63,10 +83,26 @@ Media Bus Formats
>  	the driver for capture streams and by the application for output
>  	streams, see :ref:`colorspaces`.
>      * - __u16
> -      - ``reserved``\ [11]
> +      - ``flags``
> +      - flags see:  :ref:v4l2-mbus-framefmt-flags
> +    * - __u16
> +      - ``reserved``\ [10]
>        - Reserved for future extensions. Applications and drivers must set
>  	the array to zero.
>  
> +.. _v4l2-mbus-framefmt-flags:
> +
> +.. flat-table:: v4l2_mbus_framefmt Flags
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths:       3 1 4
> +
> +    * - ``V4L2_MBUS_FRAMEFMT_HAS_CSC``
> +      - 0x0001
> +      - Set by the application. It is only used for capture and is
> +	ignored for output streams. If set, then request the subdevice to do
> +	colorspace conversion from the received colorspace, only conversions
> +	of Ycbcr encoding, and quantization are supported.

YCbCr?

>  
>  
>  .. _v4l2-mbus-pixelcode:
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> index 7e3142e11d77..125f074543af 100644
> --- a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> +++ b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
> @@ -145,6 +145,24 @@ formats in preference order, where preferred formats are returned before
>  	parameters are detected. This flag can only be used in combination
>  	with the ``V4L2_FMT_FLAG_COMPRESSED`` flag, since this applies to
>  	compressed formats only. It is also only applies to stateful codecs.
> +    * - ``V4L2_FMT_FLAG_CSC_YCBCR_ENC``
> +      - 0x0010
> +      - The driver allows the application to try to change the default
> +	ycbcr encoding. This flag is relevant only for capture devices.

YCbCr

> +	The application can ask to configure the ycbcr_enc of the capture device
> +	when calling the :c:func:`VIDIOC_S_FMT` ioctl.
> +    * - ``V4L2_FMT_FLAG_CSC_HSV_ENC``
> +      - 0x0010
> +      - The driver allows the application to try to change the default
> +	hsv encoding. This flag is relevant only for capture devices.
> +	The application can ask to configure the hsv_enc of the capture device
> +	when calling the :c:func:`VIDIOC_S_FMT` ioctl.
> +    * - ``V4L2_FMT_FLAG_CSC_QUANTIZATION``
> +      - 0x0020
> +      - The driver allows the application to try to change the default
> +	quantization. This flag is relevant only for capture devices.
> +	The application can ask to configure the quantization of the capture device
> +	when calling the :c:func:`VIDIOC_S_FMT` ioctl.
>  
>  
>  Return Value
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
> index 35b8607203a4..4ad87cb74f57 100644
> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
> @@ -78,11 +78,34 @@ information about the try formats.
>        - ``which``
>        - Media bus format codes to be enumerated, from enum
>  	:ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
> +    * - __u32
> +      - ``flags``
> +      - see :ref:`v4l2-subdev-mbus-code-flags`
>      * - __u32
>        - ``reserved``\ [8]
>        - Reserved for future extensions. Applications and drivers must set
>  	the array to zero.
>  
> +.. _v4l2-subdev-mbus-code-flags:
> +
> +
> +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{7.7cm}|
> +
> +.. flat-table:: flags in struct v4l2_subdev_mbus_code_enum
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths:       1 1 2
> +
> +    * - V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC
> +      - 0x00000001
> +      - The driver supports setting the ycbcr encoding by the application
> +	when calling the VIDIOC_SUBDEV_S_FMT ioctl. see :ref:`v4l2-mbus-format`
> +	on how to do this.
> +    * - V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION
> +      - 0x00000002
> +      - The driver supports setting the quantization by the application
> +	when calling the VIDIOC_SUBDEV_S_FMT ioctl. see :ref:`v4l2-mbus-format`
> +	on how to do this.
>  
>  Return Value
>  ============
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index b2ef8e60ea7d..3c7ffb6b15cb 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -1029,6 +1029,15 @@ static void v4l_sanitize_format(struct v4l2_format *fmt)
>  		fmt->fmt.pix_mp.num_planes = min_t(u32, fmt->fmt.pix_mp.num_planes,
>  					       VIDEO_MAX_PLANES);
>  
> +	if (fmt->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE) {
> +		if (fmt->fmt.pix_mp.flags & V4L2_PIX_FMT_FLAG_HAS_CSC)
> +			return;

This return statement might end up being confusing in the future,
because one might add further fixups below. Perhaps instead, the inner
condition negated could be added to the outer one?

> +		fmt->fmt.pix_mp.colorspace = V4L2_COLORSPACE_DEFAULT;
> +		fmt->fmt.pix_mp.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> +		fmt->fmt.pix_mp.quantization = V4L2_QUANTIZATION_DEFAULT;
> +		fmt->fmt.pix_mp.xfer_func = V4L2_XFER_FUNC_DEFAULT;
> +	}
> +
>  	/*
>  	 * The v4l2_pix_format structure has been extended with fields that were
>  	 * not previously required to be set to zero by applications. The priv
> @@ -1043,8 +1052,16 @@ static void v4l_sanitize_format(struct v4l2_format *fmt)
>  	    fmt->type != V4L2_BUF_TYPE_VIDEO_OUTPUT)
>  		return;
>  
> -	if (fmt->fmt.pix.priv == V4L2_PIX_FMT_PRIV_MAGIC)
> +	if (fmt->fmt.pix.priv == V4L2_PIX_FMT_PRIV_MAGIC) {
> +		if (fmt->type != V4L2_BUF_TYPE_VIDEO_CAPTURE ||
> +		    fmt->fmt.pix.flags & V4L2_PIX_FMT_FLAG_HAS_CSC)
> +			return;
> +		fmt->fmt.pix.colorspace = V4L2_COLORSPACE_DEFAULT;
> +		fmt->fmt.pix.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> +		fmt->fmt.pix.quantization = V4L2_QUANTIZATION_DEFAULT;
> +		fmt->fmt.pix.xfer_func = V4L2_XFER_FUNC_DEFAULT;
>  		return;
> +	}
>  
>  	fmt->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>  
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index a376b351135f..51e009936aad 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -477,6 +477,13 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  	case VIDIOC_SUBDEV_S_FMT: {
>  		struct v4l2_subdev_format *format = arg;
>  
> +		if (!(format->format.flags & V4L2_MBUS_FRAMEFMT_HAS_CSC)) {
> +			format->format.colorspace = V4L2_COLORSPACE_DEFAULT;
> +			format->format.xfer_func = V4L2_XFER_FUNC_DEFAULT;
> +			format->format.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> +			format->format.quantization = V4L2_QUANTIZATION_DEFAULT;
> +		}

Wouldn't this break setting the colorspaces on the sink pads, for which
the flag isn't required? Actually there is some unfortunate statement in
the documentation related to this:

"This information supplements the colorspace and must be set by the
driver for capture streams and by the application for output streams,
see Colorspaces."

However, I don't think there is any notion of "capture" and "output" for
subdevices, right? Instead, the pad direction would have to be checked,
but AFAICT there is no access to this kind of information from this
wrapper.

> +
>  		memset(format->reserved, 0, sizeof(format->reserved));
>  		memset(format->format.reserved, 0, sizeof(format->format.reserved));
>  		return v4l2_subdev_call(sd, pad, set_fmt, subdev_fh->pad, format);
> diff --git a/include/uapi/linux/v4l2-mediabus.h b/include/uapi/linux/v4l2-mediabus.h
> index 123a231001a8..89ff0ad6a631 100644
> --- a/include/uapi/linux/v4l2-mediabus.h
> +++ b/include/uapi/linux/v4l2-mediabus.h
> @@ -16,6 +16,8 @@
>  #include <linux/types.h>
>  #include <linux/videodev2.h>
>  
> +#define V4L2_MBUS_FRAMEFMT_HAS_CSC	0x0001
> +
>  /**
>   * struct v4l2_mbus_framefmt - frame format on the media bus
>   * @width:	image width
> @@ -36,7 +38,8 @@ struct v4l2_mbus_framefmt {
>  	__u16			ycbcr_enc;
>  	__u16			quantization;
>  	__u16			xfer_func;
> -	__u16			reserved[11];
> +	__u16			flags;

Are we okay with a u16 for flags?

Best regards,
Tomasz

Re: [RFC v3 1/2] v4l2: add support for colorspace conversion for video capture

[Dafna Hirschfeld]

Hi,

On 04.06.20 19:39, Tomasz Figa wrote:
> Hi Dafna,
> 
> On Thu, Apr 16, 2020 at 04:56:04PM +0200, Dafna Hirschfeld wrote:
>> From: Philipp Zabel <p.zabel@pengutronix.de>
>>
>> For video capture it is the driver that reports the colorspace,
>> Y'CbCr/HSV encoding, quantization range and transfer function
>> used by the video, and there is no way to request something
>> different, even though many HDTV receivers have some sort of
>> colorspace conversion capabilities.
>>
> 
> Thanks for working on this! Please see my comments inline.
> 
>> For output video this feature already exists since the application
>> specifies this information for the video format it will send out, and
>> the transmitter will enable any available CSC if a format conversion has
>> to be performed in order to match the capabilities of the sink.
>>
>> For video capture we propose adding new pix_format flag:
>> V4L2_PIX_FMT_FLAG_HAS_CSC. The flag is set by the application,
>> the driver will interpret the ycbcr_enc/hsv_enc, and quantization fields
>> as the requested colorspace information and will attempt to
>> do the conversion it supports.
>>
>> Drivers set the flags
>> V4L2_FMT_FLAG_CSC_YCBCR_ENC,
>> V4L2_FMT_FLAG_CSC_HSV_ENC,
>> V4L2_FMT_FLAG_CSC_HSV_QUANTIZATION,
> 
> Do we need this level of granularity? The drivers would ignore
> unsupported encoding/quantization settings and reset them to supported
> ones anyway, so if one doesn't support changing given parameter, it
> would just return back the original stream parameter.

I think this granularity allows userspace to know ahead what is supported
and what is not so it doesn't have to guess.

> 
>> in the flags field of the struct v4l2_fmtdesc during enumeration to
>> indicate that they support colorspace conversion for the respective field.
>> Currently the conversion of the fields colorspace, xfer_func is not
>> supported since there are no drivers that support it.
>>
>> The same API is added for the subdevices. With the flag V4L2_MBUS_FRAMEFMT_HAS_CSC
>> set by the application in the VIDIOC_SUBDEV_S_FMT ioctl and the flags
>> V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC, V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION
>> set by the driver in the VIDIOC_SUBDEV_ENUM_MBUS_CODE ioctl.
>>
>> For subdevices, new 'flags' fields were added to the structs
>> v4l2_subdev_mbus_code_enum, v4l2_mbus_framefmt which are borrowed from the 'reserved' field
>>
>> Drivers do not have to actually look at the flags: if the flags are not
>> set, then the colorspace, ycbcr_enc and quantization fields are set to
>> the default values by the core, i.e. just pass on the received format
>> without conversion.
>>
>> Signed-off-by: Hans Verkuil <Hans Verkuil@cisco.com>
> 
> Something wrong with the email address here.
> 
>> Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
>> ---
>> This is v3 of the RFC suggested originaly by Hans Verkuil:
>>
>>   https://patchwork.linuxtv.org/patch/28847/
>>
>> And then a v2 from Philipp Zabel:
>>
>>   https://patchwork.kernel.org/project/linux-media/list/?series=15483
>>
>> changes in v3:
>> I added the API to subdevices as well and added fixes according to
>> comments from Hans.
>> I also added a usecase for the new API for the rkisp1 driver.
>>
>> changes in v2 (reported by Philipp Zabel):
>>   - convert to rst
>>   - split V4L2_PIX_FMT_FLAG_REQUEST_CSC into four separate flags for
>>     colorspace, ycbcr_enc/hsv_enc, quantization, and xfer_func
>>   - let driver set flags to indicate supported features
>>
>> [1] https://patchwork.linuxtv.org/patch/28847/
>>
>>   .../media/v4l/pixfmt-reserved.rst             |  6 +++
>>   .../media/v4l/pixfmt-v4l2-mplane.rst          | 16 ++-----
>>   .../userspace-api/media/v4l/pixfmt-v4l2.rst   | 36 +++++++++++++---
>>   .../media/v4l/subdev-formats.rst              | 42 +++++++++++++++++--
>>   .../media/v4l/vidioc-enum-fmt.rst             | 18 ++++++++
>>   .../v4l/vidioc-subdev-enum-mbus-code.rst      | 23 ++++++++++
>>   drivers/media/v4l2-core/v4l2-ioctl.c          | 19 ++++++++-
>>   drivers/media/v4l2-core/v4l2-subdev.c         |  7 ++++
>>   include/uapi/linux/v4l2-mediabus.h            |  5 ++-
>>   include/uapi/linux/v4l2-subdev.h              |  5 ++-
>>   include/uapi/linux/videodev2.h                |  4 ++
>>   11 files changed, 158 insertions(+), 23 deletions(-)
>>
>> diff --git a/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst b/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst
>> index 59b9e7238f90..fa8dada69f8c 100644
>> --- a/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst
>> +++ b/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst
>> @@ -280,3 +280,9 @@ please make a proposal on the linux-media mailing list.
>>   	by RGBA values (128, 192, 255, 128), the same pixel described with
>>   	premultiplied colors would be described by RGBA values (64, 96,
>>   	128, 128)
>> +    * - ``V4L2_PIX_FMT_FLAG_HAS_CSC``
>> +      - 0x00000002
>> +      - Set by the application. It is only used for capture and is
>> +	ignored for output streams. If set, then request the driver to do
>> +	colorspace conversion from the received colorspace, only conversions
> 
> Should this be:
> 
>    colorspace. Only conversions
> 
> ?
> 
>> +	of Ycbcr encoding, HSV encoding and quantization are supported.
> 
> YCbCr?

I raformulate in coming v4 this doc according to Verkuil's comments.

> 
>> diff --git a/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst b/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst
>> index 444b4082684c..66f3365d7b72 100644
>> --- a/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst
>> +++ b/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst
>> @@ -105,29 +105,21 @@ describing all planes of that format.
>>       * - __u8
>>         - ``ycbcr_enc``
>>         - Y'CbCr encoding, from enum :c:type:`v4l2_ycbcr_encoding`.
>> -        This information supplements the ``colorspace`` and must be set by
>> -	the driver for capture streams and by the application for output
>> -	streams, see :ref:`colorspaces`.
>> +	See struct :c:type:`v4l2_pix_format`.
>>       * - __u8
>>         - ``hsv_enc``
>>         - HSV encoding, from enum :c:type:`v4l2_hsv_encoding`.
>> -        This information supplements the ``colorspace`` and must be set by
>> -	the driver for capture streams and by the application for output
>> -	streams, see :ref:`colorspaces`.
>> +	See struct :c:type:`v4l2_pix_format`.
>>       * - }
>>         -
>>       * - __u8
>>         - ``quantization``
>>         - Quantization range, from enum :c:type:`v4l2_quantization`.
>> -        This information supplements the ``colorspace`` and must be set by
>> -	the driver for capture streams and by the application for output
>> -	streams, see :ref:`colorspaces`.
>> +	See struct :c:type:`v4l2_pix_format`.
>>       * - __u8
>>         - ``xfer_func``
>>         - Transfer function, from enum :c:type:`v4l2_xfer_func`.
>> -        This information supplements the ``colorspace`` and must be set by
>> -	the driver for capture streams and by the application for output
>> -	streams, see :ref:`colorspaces`.
>> +	See struct :c:type:`v4l2_pix_format`.
>>       * - __u8
>>         - ``reserved[7]``
>>         - Reserved for future extensions. Should be zeroed by drivers and
>> diff --git a/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst b/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst
>> index 759420a872d6..ce57718cd66b 100644
>> --- a/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst
>> +++ b/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst
>> @@ -110,8 +110,8 @@ Single-planar format structure
>>         - ``colorspace``
>>         - Image colorspace, from enum :c:type:`v4l2_colorspace`.
>>           This information supplements the ``pixelformat`` and must be set
>> -	by the driver for capture streams and by the application for
>> -	output streams, see :ref:`colorspaces`.
>> +	by the driver for capture streams and by the application for output
>> +	streams, see :ref:`colorspace`.
> 
> Is it expected that the colorspace itself can't be converted? Also is
> the change of the reference name expected?

This chunk will be removed in v4

> 
>>       * - __u32
>>         - ``priv``
>>         - This field indicates whether the remaining fields of the
>> @@ -148,13 +148,31 @@ Single-planar format structure
>>         - Y'CbCr encoding, from enum :c:type:`v4l2_ycbcr_encoding`.
>>           This information supplements the ``colorspace`` and must be set by
>>   	the driver for capture streams and by the application for output
>> -	streams, see :ref:`colorspaces`.
>> +	streams, see :ref:`colorspaces`. If the application sets the
>> +	flag ``V4L2_PIX_FMT_FLAG_HAS_CSC`` then the application can set
>> +	this field for a capture stream to request a specific Y'CbCr encoding
>> +	for the captured image data. The driver will attempt to do the
>> +	conversion to the specified Y'CbCr encoding or return the encoding it
>> +	will use if it can't do the conversion. This field is ignored for
> 
> nit: The driver will attempt to do the conversion when the streaming
> starts, so that's not an entirely correct description. How about
> 
> "[...] captured image data. If the driver cannot handle requested
> conversion, it will return another, supported encoding."
> 
> ?

I'll change that

> 
>> +	non-Y'CbCr pixelformats. The driver indicates that ycbcr_enc conversion
>> +	is supported by setting the flag V4L2_FMT_FLAG_CSC_YCBCR_ENC on the
>> +	on the corresponding struct :c:type:`v4l2_fmtdesc` during enumeration.
>> +	See :ref:`fmtdesc-flags`
>>       * - __u32
>>         - ``hsv_enc``
>>         - HSV encoding, from enum :c:type:`v4l2_hsv_encoding`.
>>           This information supplements the ``colorspace`` and must be set by
>>   	the driver for capture streams and by the application for output
>> -	streams, see :ref:`colorspaces`.
>> +	streams, see :ref:`colorspaces`. If the application sets the flag
>> +	``V4L2_PIX_FMT_FLAG_HAS_CSC`` then the application can set this
>> +	field for a capture stream to request a specific HSV encoding for the
>> +	captured image data. The driver will attempt to do the conversion to
>> +	the specified HSV encoding or return the encoding it will use if it
>> +	can't do the conversion. This field is ignored for non-HSV
> 
> Ditto.
> 
>> +	pixelformats. The driver indicates that hsv_enc conversion
>> +	is supported by setting the flag V4L2_FMT_FLAG_CSC_HSV_ENC on the
>> +	on the corresponding struct :c:type:`v4l2_fmtdesc` during enumeration.
>> +	See :ref:`fmtdesc-flags`
>>       * - }
>>         -
>>       * - __u32
>> @@ -162,7 +180,15 @@ Single-planar format structure
>>         - Quantization range, from enum :c:type:`v4l2_quantization`.
>>           This information supplements the ``colorspace`` and must be set by
>>   	the driver for capture streams and by the application for output
>> -	streams, see :ref:`colorspaces`.
>> +	streams, see :ref:`colorspaces`. If the application sets the flag
>> +	``V4L2_PIX_FMT_FLAG_HAS_CSC`` then the application can set
>> +	this field for a capture stream to request a specific quantization
>> +	range for the captured image data. The driver will attempt to do the
>> +	conversion to the specified quantization range or return the
>> +	quantization it will use if it can't do the conversion.  The driver
> 
> Ditto.
> 
>> +	indicates that quantization conversion is supported by setting the flag
>> +	V4L2_FMT_FLAG_CSC_QUANTIZATION on the on the corresponding struct
>> +	:c:type:`v4l2_fmtdesc` during enumeration. See :ref:`fmtdesc-flags`
>>       * - __u32
>>         - ``xfer_func``
>>         - Transfer function, from enum :c:type:`v4l2_xfer_func`.
>> diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst
>> index 9a4d61b0d76f..f1d4ca29a3e8 100644
>> --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst
>> +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst
>> @@ -49,13 +49,33 @@ Media Bus Formats
>>         - Y'CbCr encoding, from enum :c:type:`v4l2_ycbcr_encoding`.
>>           This information supplements the ``colorspace`` and must be set by
>>   	the driver for capture streams and by the application for output
>> -	streams, see :ref:`colorspaces`.
>> +	streams, see :ref:`colorspaces`. If the application sets the
>> +	flag ``V4L2_MBUS_FRAMEFMT_HAS_CSC`` the the application can set
>> +	this field for a capture stream to request a specific Y'CbCr encoding
>> +	for the mbus data. The driver will attempt to do the
>> +	conversion to the specified Y'CbCr encoding or return the encoding it
>> +	will use if it can't do the conversion. This field is ignored for
> 
> Ditto.
> 
>> +	non-Y'CbCr pixelformats. The driver indicates that ycbcr_enc conversion
>> +	is supported by setting the flag V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC on
>> +	the corresponding struct c:type:`v4l2_subdev_mbus_code_enum` during
>> +	enumeration. See :ref:`vidioc-subdev-enum-mbus-code.rst <v4l2-subdev-mbus-code-flags>`
>> +
>>       * - __u16
>>         - ``quantization``
>>         - Quantization range, from enum :c:type:`v4l2_quantization`.
>>           This information supplements the ``colorspace`` and must be set by
>>   	the driver for capture streams and by the application for output
>> -	streams, see :ref:`colorspaces`.
>> +	streams, see :ref:`colorspaces`. If the application sets the
>> +	flag ``V4L2_MBUS_FRAMEFMT_HAS_CSC`` the the application can set
>> +	this field for a capture stream to request a specific quantization
>> +	encoding for the mbus data. The driver will attempt to do the
>> +	conversion to the specified quantization or return the quantization it
>> +	will use if it can't do the conversion. The driver indicates that
> 
> Ditto.
> 
>> +	quantization conversion is supported by setting the flag
>> +	V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION on the corresponding struct
>> +	c:type:`v4l2_subdev_mbus_code_enum` during enumeration.
>> +	See :ref:`vidioc-subdev-enum-mbus-code.rst <v4l2-subdev-mbus-code-flags>`
>> +
>>       * - __u16
>>         - ``xfer_func``
>>         - Transfer function, from enum :c:type:`v4l2_xfer_func`.
>> @@ -63,10 +83,26 @@ Media Bus Formats
>>   	the driver for capture streams and by the application for output
>>   	streams, see :ref:`colorspaces`.
>>       * - __u16
>> -      - ``reserved``\ [11]
>> +      - ``flags``
>> +      - flags see:  :ref:v4l2-mbus-framefmt-flags
>> +    * - __u16
>> +      - ``reserved``\ [10]
>>         - Reserved for future extensions. Applications and drivers must set
>>   	the array to zero.
>>   
>> +.. _v4l2-mbus-framefmt-flags:
>> +
>> +.. flat-table:: v4l2_mbus_framefmt Flags
>> +    :header-rows:  0
>> +    :stub-columns: 0
>> +    :widths:       3 1 4
>> +
>> +    * - ``V4L2_MBUS_FRAMEFMT_HAS_CSC``
>> +      - 0x0001
>> +      - Set by the application. It is only used for capture and is
>> +	ignored for output streams. If set, then request the subdevice to do
>> +	colorspace conversion from the received colorspace, only conversions
>> +	of Ycbcr encoding, and quantization are supported.
> 
> YCbCr?
> 
>>   
>>   
>>   .. _v4l2-mbus-pixelcode:
>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
>> index 7e3142e11d77..125f074543af 100644
>> --- a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
>> +++ b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst
>> @@ -145,6 +145,24 @@ formats in preference order, where preferred formats are returned before
>>   	parameters are detected. This flag can only be used in combination
>>   	with the ``V4L2_FMT_FLAG_COMPRESSED`` flag, since this applies to
>>   	compressed formats only. It is also only applies to stateful codecs.
>> +    * - ``V4L2_FMT_FLAG_CSC_YCBCR_ENC``
>> +      - 0x0010
>> +      - The driver allows the application to try to change the default
>> +	ycbcr encoding. This flag is relevant only for capture devices.
> 
> YCbCr
> 
>> +	The application can ask to configure the ycbcr_enc of the capture device
>> +	when calling the :c:func:`VIDIOC_S_FMT` ioctl.
>> +    * - ``V4L2_FMT_FLAG_CSC_HSV_ENC``
>> +      - 0x0010
>> +      - The driver allows the application to try to change the default
>> +	hsv encoding. This flag is relevant only for capture devices.
>> +	The application can ask to configure the hsv_enc of the capture device
>> +	when calling the :c:func:`VIDIOC_S_FMT` ioctl.
>> +    * - ``V4L2_FMT_FLAG_CSC_QUANTIZATION``
>> +      - 0x0020
>> +      - The driver allows the application to try to change the default
>> +	quantization. This flag is relevant only for capture devices.
>> +	The application can ask to configure the quantization of the capture device
>> +	when calling the :c:func:`VIDIOC_S_FMT` ioctl.
>>   
>>   
>>   Return Value
>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
>> index 35b8607203a4..4ad87cb74f57 100644
>> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
>> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
>> @@ -78,11 +78,34 @@ information about the try formats.
>>         - ``which``
>>         - Media bus format codes to be enumerated, from enum
>>   	:ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
>> +    * - __u32
>> +      - ``flags``
>> +      - see :ref:`v4l2-subdev-mbus-code-flags`
>>       * - __u32
>>         - ``reserved``\ [8]
>>         - Reserved for future extensions. Applications and drivers must set
>>   	the array to zero.
>>   
>> +.. _v4l2-subdev-mbus-code-flags:
>> +
>> +
>> +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{7.7cm}|
>> +
>> +.. flat-table:: flags in struct v4l2_subdev_mbus_code_enum
>> +    :header-rows:  0
>> +    :stub-columns: 0
>> +    :widths:       1 1 2
>> +
>> +    * - V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC
>> +      - 0x00000001
>> +      - The driver supports setting the ycbcr encoding by the application
>> +	when calling the VIDIOC_SUBDEV_S_FMT ioctl. see :ref:`v4l2-mbus-format`
>> +	on how to do this.
>> +    * - V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION
>> +      - 0x00000002
>> +      - The driver supports setting the quantization by the application
>> +	when calling the VIDIOC_SUBDEV_S_FMT ioctl. see :ref:`v4l2-mbus-format`
>> +	on how to do this.
>>   
>>   Return Value
>>   ============
>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
>> index b2ef8e60ea7d..3c7ffb6b15cb 100644
>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
>> @@ -1029,6 +1029,15 @@ static void v4l_sanitize_format(struct v4l2_format *fmt)
>>   		fmt->fmt.pix_mp.num_planes = min_t(u32, fmt->fmt.pix_mp.num_planes,
>>   					       VIDEO_MAX_PLANES);
>>   
>> +	if (fmt->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE) {
>> +		if (fmt->fmt.pix_mp.flags & V4L2_PIX_FMT_FLAG_HAS_CSC)
>> +			return;
> 
> This return statement might end up being confusing in the future,
> because one might add further fixups below. Perhaps instead, the inner
> condition negated could be added to the outer one?
> 
>> +		fmt->fmt.pix_mp.colorspace = V4L2_COLORSPACE_DEFAULT;
>> +		fmt->fmt.pix_mp.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>> +		fmt->fmt.pix_mp.quantization = V4L2_QUANTIZATION_DEFAULT;
>> +		fmt->fmt.pix_mp.xfer_func = V4L2_XFER_FUNC_DEFAULT;
>> +	}
>> +
>>   	/*
>>   	 * The v4l2_pix_format structure has been extended with fields that were
>>   	 * not previously required to be set to zero by applications. The priv
>> @@ -1043,8 +1052,16 @@ static void v4l_sanitize_format(struct v4l2_format *fmt)
>>   	    fmt->type != V4L2_BUF_TYPE_VIDEO_OUTPUT)
>>   		return;
>>   
>> -	if (fmt->fmt.pix.priv == V4L2_PIX_FMT_PRIV_MAGIC)
>> +	if (fmt->fmt.pix.priv == V4L2_PIX_FMT_PRIV_MAGIC) {
>> +		if (fmt->type != V4L2_BUF_TYPE_VIDEO_CAPTURE ||
>> +		    fmt->fmt.pix.flags & V4L2_PIX_FMT_FLAG_HAS_CSC)
>> +			return;
>> +		fmt->fmt.pix.colorspace = V4L2_COLORSPACE_DEFAULT;
>> +		fmt->fmt.pix.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>> +		fmt->fmt.pix.quantization = V4L2_QUANTIZATION_DEFAULT;
>> +		fmt->fmt.pix.xfer_func = V4L2_XFER_FUNC_DEFAULT;
>>   		return;
>> +	}
>>   
>>   	fmt->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>>   
>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>> index a376b351135f..51e009936aad 100644
>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>> @@ -477,6 +477,13 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>>   	case VIDIOC_SUBDEV_S_FMT: {
>>   		struct v4l2_subdev_format *format = arg;
>>   
>> +		if (!(format->format.flags & V4L2_MBUS_FRAMEFMT_HAS_CSC)) {
>> +			format->format.colorspace = V4L2_COLORSPACE_DEFAULT;
>> +			format->format.xfer_func = V4L2_XFER_FUNC_DEFAULT;
>> +			format->format.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>> +			format->format.quantization = V4L2_QUANTIZATION_DEFAULT;
>> +		}
> 
> Wouldn't this break setting the colorspaces on the sink pads, for which
> the flag isn't required? Actually there is some unfortunate statement in
> the documentation related to this:
> 
> "This information supplements the colorspace and must be set by the
> driver for capture streams and by the application for output streams,
> see Colorspaces."
> 
> However, I don't think there is any notion of "capture" and "output" for
> subdevices, right? Instead, the pad direction would have to be checked,
> but AFAICT there is no access to this kind of information from this
> wrapper.

Actually in coming v4 there is no new code added accept of the new fields and
macros of the API. I think there is no need to change any code.


> 
>> +
>>   		memset(format->reserved, 0, sizeof(format->reserved));
>>   		memset(format->format.reserved, 0, sizeof(format->format.reserved));
>>   		return v4l2_subdev_call(sd, pad, set_fmt, subdev_fh->pad, format);
>> diff --git a/include/uapi/linux/v4l2-mediabus.h b/include/uapi/linux/v4l2-mediabus.h
>> index 123a231001a8..89ff0ad6a631 100644
>> --- a/include/uapi/linux/v4l2-mediabus.h
>> +++ b/include/uapi/linux/v4l2-mediabus.h
>> @@ -16,6 +16,8 @@
>>   #include <linux/types.h>
>>   #include <linux/videodev2.h>
>>   
>> +#define V4L2_MBUS_FRAMEFMT_HAS_CSC	0x0001
>> +
>>   /**
>>    * struct v4l2_mbus_framefmt - frame format on the media bus
>>    * @width:	image width
>> @@ -36,7 +38,8 @@ struct v4l2_mbus_framefmt {
>>   	__u16			ycbcr_enc;
>>   	__u16			quantization;
>>   	__u16			xfer_func;
>> -	__u16			reserved[11];
>> +	__u16			flags;
> 
> Are we okay with a u16 for flags?

I am fine with it, though don't mind changing it if there are objections.

Thanks,
Dafna

> 
> Best regards,
> Tomasz
> 

Re: [RFC v3 1/2] v4l2: add support for colorspace conversion for video capture

[Tomasz Figa]

On Fri, Jun 05, 2020 at 12:11:33PM +0200, Dafna Hirschfeld wrote:
> Hi,
> 
> On 04.06.20 19:39, Tomasz Figa wrote:
> > Hi Dafna,
> > 
> > On Thu, Apr 16, 2020 at 04:56:04PM +0200, Dafna Hirschfeld wrote:
> > > From: Philipp Zabel <p.zabel@pengutronix.de>
> > > 
> > > For video capture it is the driver that reports the colorspace,
> > > Y'CbCr/HSV encoding, quantization range and transfer function
> > > used by the video, and there is no way to request something
> > > different, even though many HDTV receivers have some sort of
> > > colorspace conversion capabilities.
> > > 
> > 
> > Thanks for working on this! Please see my comments inline.
> > 
> > > For output video this feature already exists since the application
> > > specifies this information for the video format it will send out, and
> > > the transmitter will enable any available CSC if a format conversion has
> > > to be performed in order to match the capabilities of the sink.
> > > 
> > > For video capture we propose adding new pix_format flag:
> > > V4L2_PIX_FMT_FLAG_HAS_CSC. The flag is set by the application,
> > > the driver will interpret the ycbcr_enc/hsv_enc, and quantization fields
> > > as the requested colorspace information and will attempt to
> > > do the conversion it supports.
> > > 
> > > Drivers set the flags
> > > V4L2_FMT_FLAG_CSC_YCBCR_ENC,
> > > V4L2_FMT_FLAG_CSC_HSV_ENC,
> > > V4L2_FMT_FLAG_CSC_HSV_QUANTIZATION,
> > 
> > Do we need this level of granularity? The drivers would ignore
> > unsupported encoding/quantization settings and reset them to supported
> > ones anyway, so if one doesn't support changing given parameter, it
> > would just return back the original stream parameter.
> 
> I think this granularity allows userspace to know ahead what is supported
> and what is not so it doesn't have to guess.
>

The userspace needs to guess anyway, because there is no way to
enumerate the supported target parameters. For example, even if the
CSC_YCBCR_ENC bit is set, only only DEFAULT, 601 and BT2020 could be
supported. If the userspace wants to get the 709 encoding, it needs to
explicitly try setting it and see if the TRY_FMT/S_FMT operation accepts
the setting.

[snip]
> > > diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> > > index a376b351135f..51e009936aad 100644
> > > --- a/drivers/media/v4l2-core/v4l2-subdev.c
> > > +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> > > @@ -477,6 +477,13 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
> > >   	case VIDIOC_SUBDEV_S_FMT: {
> > >   		struct v4l2_subdev_format *format = arg;
> > > +		if (!(format->format.flags & V4L2_MBUS_FRAMEFMT_HAS_CSC)) {
> > > +			format->format.colorspace = V4L2_COLORSPACE_DEFAULT;
> > > +			format->format.xfer_func = V4L2_XFER_FUNC_DEFAULT;
> > > +			format->format.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> > > +			format->format.quantization = V4L2_QUANTIZATION_DEFAULT;
> > > +		}
> > 
> > Wouldn't this break setting the colorspaces on the sink pads, for which
> > the flag isn't required? Actually there is some unfortunate statement in
> > the documentation related to this:
> > 
> > "This information supplements the colorspace and must be set by the
> > driver for capture streams and by the application for output streams,
> > see Colorspaces."
> > 
> > However, I don't think there is any notion of "capture" and "output" for
> > subdevices, right? Instead, the pad direction would have to be checked,
> > but AFAICT there is no access to this kind of information from this
> > wrapper.
> 
> Actually in coming v4 there is no new code added accept of the new fields and
> macros of the API. I think there is no need to change any code.
> 
> 

Agreed.

> > 
> > > +
> > >   		memset(format->reserved, 0, sizeof(format->reserved));
> > >   		memset(format->format.reserved, 0, sizeof(format->format.reserved));
> > >   		return v4l2_subdev_call(sd, pad, set_fmt, subdev_fh->pad, format);
> > > diff --git a/include/uapi/linux/v4l2-mediabus.h b/include/uapi/linux/v4l2-mediabus.h
> > > index 123a231001a8..89ff0ad6a631 100644
> > > --- a/include/uapi/linux/v4l2-mediabus.h
> > > +++ b/include/uapi/linux/v4l2-mediabus.h
> > > @@ -16,6 +16,8 @@
> > >   #include <linux/types.h>
> > >   #include <linux/videodev2.h>
> > > +#define V4L2_MBUS_FRAMEFMT_HAS_CSC	0x0001
> > > +
> > >   /**
> > >    * struct v4l2_mbus_framefmt - frame format on the media bus
> > >    * @width:	image width
> > > @@ -36,7 +38,8 @@ struct v4l2_mbus_framefmt {
> > >   	__u16			ycbcr_enc;
> > >   	__u16			quantization;
> > >   	__u16			xfer_func;
> > > -	__u16			reserved[11];
> > > +	__u16			flags;
> > 
> > Are we okay with a u16 for flags?
> 
> I am fine with it, though don't mind changing it if there are objections.
> 

I'd suggest making it a u32 to be a bit more future-proof.

Best regards,
Tomasz

Re: [RFC v3 1/2] v4l2: add support for colorspace conversion for video capture

[Dafna Hirschfeld]

On 10.06.20 15:34, Tomasz Figa wrote:
> On Fri, Jun 05, 2020 at 12:11:33PM +0200, Dafna Hirschfeld wrote:
>> Hi,
>>
>> On 04.06.20 19:39, Tomasz Figa wrote:
>>> Hi Dafna,
>>>
>>> On Thu, Apr 16, 2020 at 04:56:04PM +0200, Dafna Hirschfeld wrote:
>>>> From: Philipp Zabel <p.zabel@pengutronix.de>
>>>>
>>>> For video capture it is the driver that reports the colorspace,
>>>> Y'CbCr/HSV encoding, quantization range and transfer function
>>>> used by the video, and there is no way to request something
>>>> different, even though many HDTV receivers have some sort of
>>>> colorspace conversion capabilities.
>>>>
>>>
>>> Thanks for working on this! Please see my comments inline.
>>>
>>>> For output video this feature already exists since the application
>>>> specifies this information for the video format it will send out, and
>>>> the transmitter will enable any available CSC if a format conversion has
>>>> to be performed in order to match the capabilities of the sink.
>>>>
>>>> For video capture we propose adding new pix_format flag:
>>>> V4L2_PIX_FMT_FLAG_HAS_CSC. The flag is set by the application,
>>>> the driver will interpret the ycbcr_enc/hsv_enc, and quantization fields
>>>> as the requested colorspace information and will attempt to
>>>> do the conversion it supports.
>>>>
>>>> Drivers set the flags
>>>> V4L2_FMT_FLAG_CSC_YCBCR_ENC,
>>>> V4L2_FMT_FLAG_CSC_HSV_ENC,
>>>> V4L2_FMT_FLAG_CSC_HSV_QUANTIZATION,
>>>
>>> Do we need this level of granularity? The drivers would ignore
>>> unsupported encoding/quantization settings and reset them to supported
>>> ones anyway, so if one doesn't support changing given parameter, it
>>> would just return back the original stream parameter.
>>
>> I think this granularity allows userspace to know ahead what is supported
>> and what is not so it doesn't have to guess.
>>
> 
> The userspace needs to guess anyway, because there is no way to
> enumerate the supported target parameters. For example, even if the
> CSC_YCBCR_ENC bit is set, only only DEFAULT, 601 and BT2020 could be
> supported. If the userspace wants to get the 709 encoding, it needs to
> explicitly try setting it and see if the TRY_FMT/S_FMT operation accepts
> the setting.

yes, indeed, Hans Verkuil suggested those flags. Maybe it is indeed enough
to have one flag.

> 
> [snip]
>>>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>>>> index a376b351135f..51e009936aad 100644
>>>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>>>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>>>> @@ -477,6 +477,13 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>>>>    	case VIDIOC_SUBDEV_S_FMT: {
>>>>    		struct v4l2_subdev_format *format = arg;
>>>> +		if (!(format->format.flags & V4L2_MBUS_FRAMEFMT_HAS_CSC)) {
>>>> +			format->format.colorspace = V4L2_COLORSPACE_DEFAULT;
>>>> +			format->format.xfer_func = V4L2_XFER_FUNC_DEFAULT;
>>>> +			format->format.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>>>> +			format->format.quantization = V4L2_QUANTIZATION_DEFAULT;
>>>> +		}
>>>
>>> Wouldn't this break setting the colorspaces on the sink pads, for which
>>> the flag isn't required? Actually there is some unfortunate statement in
>>> the documentation related to this:
>>>
>>> "This information supplements the colorspace and must be set by the
>>> driver for capture streams and by the application for output streams,
>>> see Colorspaces."
>>>
>>> However, I don't think there is any notion of "capture" and "output" for
>>> subdevices, right? Instead, the pad direction would have to be checked,
>>> but AFAICT there is no access to this kind of information from this
>>> wrapper.
>>
>> Actually in coming v4 there is no new code added accept of the new fields and
>> macros of the API. I think there is no need to change any code.
>>
>>
> 
> Agreed.
> 
>>>
>>>> +
>>>>    		memset(format->reserved, 0, sizeof(format->reserved));
>>>>    		memset(format->format.reserved, 0, sizeof(format->format.reserved));
>>>>    		return v4l2_subdev_call(sd, pad, set_fmt, subdev_fh->pad, format);
>>>> diff --git a/include/uapi/linux/v4l2-mediabus.h b/include/uapi/linux/v4l2-mediabus.h
>>>> index 123a231001a8..89ff0ad6a631 100644
>>>> --- a/include/uapi/linux/v4l2-mediabus.h
>>>> +++ b/include/uapi/linux/v4l2-mediabus.h
>>>> @@ -16,6 +16,8 @@
>>>>    #include <linux/types.h>
>>>>    #include <linux/videodev2.h>
>>>> +#define V4L2_MBUS_FRAMEFMT_HAS_CSC	0x0001
>>>> +
>>>>    /**
>>>>     * struct v4l2_mbus_framefmt - frame format on the media bus
>>>>     * @width:	image width
>>>> @@ -36,7 +38,8 @@ struct v4l2_mbus_framefmt {
>>>>    	__u16			ycbcr_enc;
>>>>    	__u16			quantization;
>>>>    	__u16			xfer_func;
>>>> -	__u16			reserved[11];
>>>> +	__u16			flags;
>>>
>>> Are we okay with a u16 for flags?
>>
>> I am fine with it, though don't mind changing it if there are objections.
>>
> 
> I'd suggest making it a u32 to be a bit more future-proof.

ok, I see that just changing the type to __u32 and the reserved array
to 'reserved[9]' increases the struct size from 48 to 52 because of padding.
There are two ways to solve it,
- move the flags field to be just above 'ycbcr_enc'
- change reserve to 'reserve[8]'

Is moving fields order in a struct ok? If so it save us 2 bytes.

Thanks,
Dafna

> 
> Best regards,
> Tomasz
> 

Re: [RFC v3 1/2] v4l2: add support for colorspace conversion for video capture

[Tomasz Figa]

On Tue, Jun 30, 2020 at 6:11 PM Dafna Hirschfeld
<dafna.hirschfeld@collabora.com> wrote:
>
>
>
> On 10.06.20 15:34, Tomasz Figa wrote:
> > On Fri, Jun 05, 2020 at 12:11:33PM +0200, Dafna Hirschfeld wrote:
> >> Hi,
> >>
> >> On 04.06.20 19:39, Tomasz Figa wrote:
> >>> Hi Dafna,
> >>>
> >>> On Thu, Apr 16, 2020 at 04:56:04PM +0200, Dafna Hirschfeld wrote:
> >>>> From: Philipp Zabel <p.zabel@pengutronix.de>
> >>>>
> >>>> For video capture it is the driver that reports the colorspace,
> >>>> Y'CbCr/HSV encoding, quantization range and transfer function
> >>>> used by the video, and there is no way to request something
> >>>> different, even though many HDTV receivers have some sort of
> >>>> colorspace conversion capabilities.
> >>>>
> >>>
> >>> Thanks for working on this! Please see my comments inline.
> >>>
> >>>> For output video this feature already exists since the application
> >>>> specifies this information for the video format it will send out, and
> >>>> the transmitter will enable any available CSC if a format conversion has
> >>>> to be performed in order to match the capabilities of the sink.
> >>>>
> >>>> For video capture we propose adding new pix_format flag:
> >>>> V4L2_PIX_FMT_FLAG_HAS_CSC. The flag is set by the application,
> >>>> the driver will interpret the ycbcr_enc/hsv_enc, and quantization fields
> >>>> as the requested colorspace information and will attempt to
> >>>> do the conversion it supports.
> >>>>
> >>>> Drivers set the flags
> >>>> V4L2_FMT_FLAG_CSC_YCBCR_ENC,
> >>>> V4L2_FMT_FLAG_CSC_HSV_ENC,
> >>>> V4L2_FMT_FLAG_CSC_HSV_QUANTIZATION,
> >>>
> >>> Do we need this level of granularity? The drivers would ignore
> >>> unsupported encoding/quantization settings and reset them to supported
> >>> ones anyway, so if one doesn't support changing given parameter, it
> >>> would just return back the original stream parameter.
> >>
> >> I think this granularity allows userspace to know ahead what is supported
> >> and what is not so it doesn't have to guess.
> >>
> >
> > The userspace needs to guess anyway, because there is no way to
> > enumerate the supported target parameters. For example, even if the
> > CSC_YCBCR_ENC bit is set, only only DEFAULT, 601 and BT2020 could be
> > supported. If the userspace wants to get the 709 encoding, it needs to
> > explicitly try setting it and see if the TRY_FMT/S_FMT operation accepts
> > the setting.
>
> yes, indeed, Hans Verkuil suggested those flags. Maybe it is indeed enough
> to have one flag.
>

Hans, what's your thought on this?

> >
> > [snip]
> >>>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> >>>> index a376b351135f..51e009936aad 100644
> >>>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> >>>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> >>>> @@ -477,6 +477,13 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
> >>>>            case VIDIOC_SUBDEV_S_FMT: {
> >>>>                    struct v4l2_subdev_format *format = arg;
> >>>> +          if (!(format->format.flags & V4L2_MBUS_FRAMEFMT_HAS_CSC)) {
> >>>> +                  format->format.colorspace = V4L2_COLORSPACE_DEFAULT;
> >>>> +                  format->format.xfer_func = V4L2_XFER_FUNC_DEFAULT;
> >>>> +                  format->format.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> >>>> +                  format->format.quantization = V4L2_QUANTIZATION_DEFAULT;
> >>>> +          }
> >>>
> >>> Wouldn't this break setting the colorspaces on the sink pads, for which
> >>> the flag isn't required? Actually there is some unfortunate statement in
> >>> the documentation related to this:
> >>>
> >>> "This information supplements the colorspace and must be set by the
> >>> driver for capture streams and by the application for output streams,
> >>> see Colorspaces."
> >>>
> >>> However, I don't think there is any notion of "capture" and "output" for
> >>> subdevices, right? Instead, the pad direction would have to be checked,
> >>> but AFAICT there is no access to this kind of information from this
> >>> wrapper.
> >>
> >> Actually in coming v4 there is no new code added accept of the new fields and
> >> macros of the API. I think there is no need to change any code.
> >>
> >>
> >
> > Agreed.
> >
> >>>
> >>>> +
> >>>>                    memset(format->reserved, 0, sizeof(format->reserved));
> >>>>                    memset(format->format.reserved, 0, sizeof(format->format.reserved));
> >>>>                    return v4l2_subdev_call(sd, pad, set_fmt, subdev_fh->pad, format);
> >>>> diff --git a/include/uapi/linux/v4l2-mediabus.h b/include/uapi/linux/v4l2-mediabus.h
> >>>> index 123a231001a8..89ff0ad6a631 100644
> >>>> --- a/include/uapi/linux/v4l2-mediabus.h
> >>>> +++ b/include/uapi/linux/v4l2-mediabus.h
> >>>> @@ -16,6 +16,8 @@
> >>>>    #include <linux/types.h>
> >>>>    #include <linux/videodev2.h>
> >>>> +#define V4L2_MBUS_FRAMEFMT_HAS_CSC        0x0001
> >>>> +
> >>>>    /**
> >>>>     * struct v4l2_mbus_framefmt - frame format on the media bus
> >>>>     * @width:      image width
> >>>> @@ -36,7 +38,8 @@ struct v4l2_mbus_framefmt {
> >>>>            __u16                   ycbcr_enc;
> >>>>            __u16                   quantization;
> >>>>            __u16                   xfer_func;
> >>>> -  __u16                   reserved[11];
> >>>> +  __u16                   flags;
> >>>
> >>> Are we okay with a u16 for flags?
> >>
> >> I am fine with it, though don't mind changing it if there are objections.
> >>
> >
> > I'd suggest making it a u32 to be a bit more future-proof.
>
> ok, I see that just changing the type to __u32 and the reserved array
> to 'reserved[9]' increases the struct size from 48 to 52 because of padding.
> There are two ways to solve it,
> - move the flags field to be just above 'ycbcr_enc'
> - change reserve to 'reserve[8]'
>
> Is moving fields order in a struct ok? If so it save us 2 bytes.

Since the structure is a part of the stable UAPI, we can't reorder the
fields. Similarly, we can't change the struct size, because it's
embedded in the ioctl code. (Although there are ways around it, not
currently implemented by V4L2.) That leaves us only the second option
- changing reserved to [8].

Best regards,
Tomasz

Re: [RFC v3 1/2] v4l2: add support for colorspace conversion for video capture

[Laurent Pinchart]

Hi Tomasz,

On Tue, Jun 30, 2020 at 06:31:19PM +0200, Tomasz Figa wrote:
> On Tue, Jun 30, 2020 at 6:11 PM Dafna Hirschfeld wrote:
> > On 10.06.20 15:34, Tomasz Figa wrote:
> >> On Fri, Jun 05, 2020 at 12:11:33PM +0200, Dafna Hirschfeld wrote:
> >>> On 04.06.20 19:39, Tomasz Figa wrote:
> >>>> On Thu, Apr 16, 2020 at 04:56:04PM +0200, Dafna Hirschfeld wrote:
> >>>>> From: Philipp Zabel <p.zabel@pengutronix.de>
> >>>>>
> >>>>> For video capture it is the driver that reports the colorspace,
> >>>>> Y'CbCr/HSV encoding, quantization range and transfer function
> >>>>> used by the video, and there is no way to request something
> >>>>> different, even though many HDTV receivers have some sort of
> >>>>> colorspace conversion capabilities.
> >>>>
> >>>> Thanks for working on this! Please see my comments inline.
> >>>>
> >>>>> For output video this feature already exists since the application
> >>>>> specifies this information for the video format it will send out, and
> >>>>> the transmitter will enable any available CSC if a format conversion has
> >>>>> to be performed in order to match the capabilities of the sink.
> >>>>>
> >>>>> For video capture we propose adding new pix_format flag:
> >>>>> V4L2_PIX_FMT_FLAG_HAS_CSC. The flag is set by the application,
> >>>>> the driver will interpret the ycbcr_enc/hsv_enc, and quantization fields
> >>>>> as the requested colorspace information and will attempt to
> >>>>> do the conversion it supports.
> >>>>>
> >>>>> Drivers set the flags
> >>>>> V4L2_FMT_FLAG_CSC_YCBCR_ENC,
> >>>>> V4L2_FMT_FLAG_CSC_HSV_ENC,
> >>>>> V4L2_FMT_FLAG_CSC_HSV_QUANTIZATION,
> >>>>
> >>>> Do we need this level of granularity? The drivers would ignore
> >>>> unsupported encoding/quantization settings and reset them to supported
> >>>> ones anyway, so if one doesn't support changing given parameter, it
> >>>> would just return back the original stream parameter.
> >>>
> >>> I think this granularity allows userspace to know ahead what is supported
> >>> and what is not so it doesn't have to guess.
> >>>
> >>
> >> The userspace needs to guess anyway, because there is no way to
> >> enumerate the supported target parameters. For example, even if the
> >> CSC_YCBCR_ENC bit is set, only only DEFAULT, 601 and BT2020 could be
> >> supported. If the userspace wants to get the 709 encoding, it needs to
> >> explicitly try setting it and see if the TRY_FMT/S_FMT operation accepts
> >> the setting.
> >
> > yes, indeed, Hans Verkuil suggested those flags. Maybe it is indeed enough
> > to have one flag.
> >
> 
> Hans, what's your thought on this?
> 
> >>
> >> [snip]
> >>>>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> >>>>> index a376b351135f..51e009936aad 100644
> >>>>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> >>>>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> >>>>> @@ -477,6 +477,13 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
> >>>>>            case VIDIOC_SUBDEV_S_FMT: {
> >>>>>                    struct v4l2_subdev_format *format = arg;
> >>>>> +          if (!(format->format.flags & V4L2_MBUS_FRAMEFMT_HAS_CSC)) {
> >>>>> +                  format->format.colorspace = V4L2_COLORSPACE_DEFAULT;
> >>>>> +                  format->format.xfer_func = V4L2_XFER_FUNC_DEFAULT;
> >>>>> +                  format->format.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> >>>>> +                  format->format.quantization = V4L2_QUANTIZATION_DEFAULT;
> >>>>> +          }
> >>>>
> >>>> Wouldn't this break setting the colorspaces on the sink pads, for which
> >>>> the flag isn't required? Actually there is some unfortunate statement in
> >>>> the documentation related to this:
> >>>>
> >>>> "This information supplements the colorspace and must be set by the
> >>>> driver for capture streams and by the application for output streams,
> >>>> see Colorspaces."
> >>>>
> >>>> However, I don't think there is any notion of "capture" and "output" for
> >>>> subdevices, right? Instead, the pad direction would have to be checked,
> >>>> but AFAICT there is no access to this kind of information from this
> >>>> wrapper.
> >>>
> >>> Actually in coming v4 there is no new code added accept of the new fields and
> >>> macros of the API. I think there is no need to change any code.
> >>
> >> Agreed.
> >>
> >>>>> +
> >>>>>                    memset(format->reserved, 0, sizeof(format->reserved));
> >>>>>                    memset(format->format.reserved, 0, sizeof(format->format.reserved));
> >>>>>                    return v4l2_subdev_call(sd, pad, set_fmt, subdev_fh->pad, format);
> >>>>> diff --git a/include/uapi/linux/v4l2-mediabus.h b/include/uapi/linux/v4l2-mediabus.h
> >>>>> index 123a231001a8..89ff0ad6a631 100644
> >>>>> --- a/include/uapi/linux/v4l2-mediabus.h
> >>>>> +++ b/include/uapi/linux/v4l2-mediabus.h
> >>>>> @@ -16,6 +16,8 @@
> >>>>>    #include <linux/types.h>
> >>>>>    #include <linux/videodev2.h>
> >>>>> +#define V4L2_MBUS_FRAMEFMT_HAS_CSC        0x0001
> >>>>> +
> >>>>>    /**
> >>>>>     * struct v4l2_mbus_framefmt - frame format on the media bus
> >>>>>     * @width:      image width
> >>>>> @@ -36,7 +38,8 @@ struct v4l2_mbus_framefmt {
> >>>>>            __u16                   ycbcr_enc;
> >>>>>            __u16                   quantization;
> >>>>>            __u16                   xfer_func;
> >>>>> -  __u16                   reserved[11];
> >>>>> +  __u16                   flags;
> >>>>
> >>>> Are we okay with a u16 for flags?
> >>>
> >>> I am fine with it, though don't mind changing it if there are objections.
> >>>
> >>
> >> I'd suggest making it a u32 to be a bit more future-proof.
> >
> > ok, I see that just changing the type to __u32 and the reserved array
> > to 'reserved[9]' increases the struct size from 48 to 52 because of padding.
> > There are two ways to solve it,
> > - move the flags field to be just above 'ycbcr_enc'
> > - change reserve to 'reserve[8]'
> >
> > Is moving fields order in a struct ok? If so it save us 2 bytes.
> 
> Since the structure is a part of the stable UAPI, we can't reorder the
> fields. Similarly, we can't change the struct size, because it's
> embedded in the ioctl code. (Although there are ways around it, not
> currently implemented by V4L2.) That leaves us only the second option
> - changing reserved to [8].

You can also possibly do

	__u16			ycbcr_enc;
	__u16			quantization;
	__u16			xfer_func;
	__u16			reserved2;
	__u32			flags;
	__u16			reserved[8];

to explicitly show there's a hole.

-- 
Regards,

Laurent Pinchart

Re: [RFC v3 1/2] v4l2: add support for colorspace conversion for video capture

[Tomasz Figa]

On Tue, Jun 30, 2020 at 7:43 PM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> Hi Tomasz,
>
> On Tue, Jun 30, 2020 at 06:31:19PM +0200, Tomasz Figa wrote:
> > On Tue, Jun 30, 2020 at 6:11 PM Dafna Hirschfeld wrote:
> > > On 10.06.20 15:34, Tomasz Figa wrote:
> > >> On Fri, Jun 05, 2020 at 12:11:33PM +0200, Dafna Hirschfeld wrote:
> > >>> On 04.06.20 19:39, Tomasz Figa wrote:
> > >>>> On Thu, Apr 16, 2020 at 04:56:04PM +0200, Dafna Hirschfeld wrote:
> > >>>>> From: Philipp Zabel <p.zabel@pengutronix.de>
> > >>>>>
> > >>>>> For video capture it is the driver that reports the colorspace,
> > >>>>> Y'CbCr/HSV encoding, quantization range and transfer function
> > >>>>> used by the video, and there is no way to request something
> > >>>>> different, even though many HDTV receivers have some sort of
> > >>>>> colorspace conversion capabilities.
> > >>>>
> > >>>> Thanks for working on this! Please see my comments inline.
> > >>>>
> > >>>>> For output video this feature already exists since the application
> > >>>>> specifies this information for the video format it will send out, and
> > >>>>> the transmitter will enable any available CSC if a format conversion has
> > >>>>> to be performed in order to match the capabilities of the sink.
> > >>>>>
> > >>>>> For video capture we propose adding new pix_format flag:
> > >>>>> V4L2_PIX_FMT_FLAG_HAS_CSC. The flag is set by the application,
> > >>>>> the driver will interpret the ycbcr_enc/hsv_enc, and quantization fields
> > >>>>> as the requested colorspace information and will attempt to
> > >>>>> do the conversion it supports.
> > >>>>>
> > >>>>> Drivers set the flags
> > >>>>> V4L2_FMT_FLAG_CSC_YCBCR_ENC,
> > >>>>> V4L2_FMT_FLAG_CSC_HSV_ENC,
> > >>>>> V4L2_FMT_FLAG_CSC_HSV_QUANTIZATION,
> > >>>>
> > >>>> Do we need this level of granularity? The drivers would ignore
> > >>>> unsupported encoding/quantization settings and reset them to supported
> > >>>> ones anyway, so if one doesn't support changing given parameter, it
> > >>>> would just return back the original stream parameter.
> > >>>
> > >>> I think this granularity allows userspace to know ahead what is supported
> > >>> and what is not so it doesn't have to guess.
> > >>>
> > >>
> > >> The userspace needs to guess anyway, because there is no way to
> > >> enumerate the supported target parameters. For example, even if the
> > >> CSC_YCBCR_ENC bit is set, only only DEFAULT, 601 and BT2020 could be
> > >> supported. If the userspace wants to get the 709 encoding, it needs to
> > >> explicitly try setting it and see if the TRY_FMT/S_FMT operation accepts
> > >> the setting.
> > >
> > > yes, indeed, Hans Verkuil suggested those flags. Maybe it is indeed enough
> > > to have one flag.
> > >
> >
> > Hans, what's your thought on this?
> >
> > >>
> > >> [snip]
> > >>>>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> > >>>>> index a376b351135f..51e009936aad 100644
> > >>>>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> > >>>>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> > >>>>> @@ -477,6 +477,13 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
> > >>>>>            case VIDIOC_SUBDEV_S_FMT: {
> > >>>>>                    struct v4l2_subdev_format *format = arg;
> > >>>>> +          if (!(format->format.flags & V4L2_MBUS_FRAMEFMT_HAS_CSC)) {
> > >>>>> +                  format->format.colorspace = V4L2_COLORSPACE_DEFAULT;
> > >>>>> +                  format->format.xfer_func = V4L2_XFER_FUNC_DEFAULT;
> > >>>>> +                  format->format.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
> > >>>>> +                  format->format.quantization = V4L2_QUANTIZATION_DEFAULT;
> > >>>>> +          }
> > >>>>
> > >>>> Wouldn't this break setting the colorspaces on the sink pads, for which
> > >>>> the flag isn't required? Actually there is some unfortunate statement in
> > >>>> the documentation related to this:
> > >>>>
> > >>>> "This information supplements the colorspace and must be set by the
> > >>>> driver for capture streams and by the application for output streams,
> > >>>> see Colorspaces."
> > >>>>
> > >>>> However, I don't think there is any notion of "capture" and "output" for
> > >>>> subdevices, right? Instead, the pad direction would have to be checked,
> > >>>> but AFAICT there is no access to this kind of information from this
> > >>>> wrapper.
> > >>>
> > >>> Actually in coming v4 there is no new code added accept of the new fields and
> > >>> macros of the API. I think there is no need to change any code.
> > >>
> > >> Agreed.
> > >>
> > >>>>> +
> > >>>>>                    memset(format->reserved, 0, sizeof(format->reserved));
> > >>>>>                    memset(format->format.reserved, 0, sizeof(format->format.reserved));
> > >>>>>                    return v4l2_subdev_call(sd, pad, set_fmt, subdev_fh->pad, format);
> > >>>>> diff --git a/include/uapi/linux/v4l2-mediabus.h b/include/uapi/linux/v4l2-mediabus.h
> > >>>>> index 123a231001a8..89ff0ad6a631 100644
> > >>>>> --- a/include/uapi/linux/v4l2-mediabus.h
> > >>>>> +++ b/include/uapi/linux/v4l2-mediabus.h
> > >>>>> @@ -16,6 +16,8 @@
> > >>>>>    #include <linux/types.h>
> > >>>>>    #include <linux/videodev2.h>
> > >>>>> +#define V4L2_MBUS_FRAMEFMT_HAS_CSC        0x0001
> > >>>>> +
> > >>>>>    /**
> > >>>>>     * struct v4l2_mbus_framefmt - frame format on the media bus
> > >>>>>     * @width:      image width
> > >>>>> @@ -36,7 +38,8 @@ struct v4l2_mbus_framefmt {
> > >>>>>            __u16                   ycbcr_enc;
> > >>>>>            __u16                   quantization;
> > >>>>>            __u16                   xfer_func;
> > >>>>> -  __u16                   reserved[11];
> > >>>>> +  __u16                   flags;
> > >>>>
> > >>>> Are we okay with a u16 for flags?
> > >>>
> > >>> I am fine with it, though don't mind changing it if there are objections.
> > >>>
> > >>
> > >> I'd suggest making it a u32 to be a bit more future-proof.
> > >
> > > ok, I see that just changing the type to __u32 and the reserved array
> > > to 'reserved[9]' increases the struct size from 48 to 52 because of padding.
> > > There are two ways to solve it,
> > > - move the flags field to be just above 'ycbcr_enc'
> > > - change reserve to 'reserve[8]'
> > >
> > > Is moving fields order in a struct ok? If so it save us 2 bytes.
> >
> > Since the structure is a part of the stable UAPI, we can't reorder the
> > fields. Similarly, we can't change the struct size, because it's
> > embedded in the ioctl code. (Although there are ways around it, not
> > currently implemented by V4L2.) That leaves us only the second option
> > - changing reserved to [8].
>
> You can also possibly do
>
>         __u16                   ycbcr_enc;
>         __u16                   quantization;
>         __u16                   xfer_func;
>         __u16                   reserved2;
>         __u32                   flags;
>         __u16                   reserved[8];
>
> to explicitly show there's a hole.

Good point. I didn't realize that there actually was a hole. Thought
that xfer_func ended at a 32-bit boundary.

Perhaps when changing this, we could make it __u32 reserved[4]? Or
would that have some compatibility concerns?

Best regards,
Tomasz