Linux-Media Archive on lore.kernel.org
 help / color / Atom feed
From: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
To: Tomasz Figa <tfiga@chromium.org>
Cc: linux-media@vger.kernel.org, helen.koike@collabora.com,
	ezequiel@collabora.com, hverkuil@xs4all.nl, kernel@collabora.com,
	dafna3@gmail.com, laurent.pinchart@ideasonboard.com,
	linux-rockchip@lists.infradead.org, sakari.ailus@linux.intel.com
Subject: Re: [RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad
Date: Thu, 4 Jun 2020 21:11:20 +0200
Message-ID: <87f11ebf-aecf-7eee-f18c-98dd4e8d692b@collabora.com> (raw)
In-Reply-To: <20200604175443.GB76282@chromium.org>

Hi

On 04.06.20 19:54, Tomasz Figa wrote:
> On Thu, Apr 16, 2020 at 04:56:05PM +0200, Dafna Hirschfeld wrote:
>> The isp entity has a hardware support to force full range quantization
>> for YUV formats. Use the new API to indicate userspace that
>> quantization conversion is supported by adding the flag
>> V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION during media code enumeration.
>> Then uppon s_fmt on the video source pad, we assign the
>> quantization from userspace for YUV formats.
>> Also in the capture and resizer entities we retrieve the colorspace
>> from the isp entity.
>>
>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
>> ---
>>   drivers/staging/media/rkisp1/rkisp1-capture.c | 28 ++++++-------
>>   drivers/staging/media/rkisp1/rkisp1-common.h  |  2 +
>>   drivers/staging/media/rkisp1/rkisp1-isp.c     | 39 +++++++++++++++++--
>>   drivers/staging/media/rkisp1/rkisp1-resizer.c | 13 +++++++
>>   4 files changed, 65 insertions(+), 17 deletions(-)
>>
>> diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c
>> index fbf62399fe3d..aca0f93bc772 100644
>> --- a/drivers/staging/media/rkisp1/rkisp1-capture.c
>> +++ b/drivers/staging/media/rkisp1/rkisp1-capture.c
>> @@ -1066,14 +1066,13 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
>>   			   const struct v4l2_format_info **fmt_info)
>>   {
>>   	const struct rkisp1_capture_config *config = cap->config;
>> -	struct rkisp1_capture *other_cap =
>> -			&cap->rkisp1->capture_devs[cap->id ^ 1];
>>   	const struct rkisp1_capture_fmt_cfg *fmt;
>>   	const struct v4l2_format_info *info;
>>   	const unsigned int max_widths[] = { RKISP1_RSZ_MP_SRC_MAX_WIDTH,
>>   					    RKISP1_RSZ_SP_SRC_MAX_WIDTH };
>>   	const unsigned int max_heights[] = { RKISP1_RSZ_MP_SRC_MAX_HEIGHT,
>>   					     RKISP1_RSZ_SP_SRC_MAX_HEIGHT};
>> +	struct v4l2_subdev_format isp_sd_fmt;
>>   
>>   	fmt = rkisp1_find_fmt_cfg(cap, pixm->pixelformat);
>>   	if (!fmt) {
>> @@ -1081,24 +1080,27 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
>>   		pixm->pixelformat = fmt->fourcc;
>>   	}
>>   
>> +	rkisp1_get_isp_src_fmt(cap->rkisp1, &isp_sd_fmt);
>> +	pixm->field = isp_sd_fmt.format.field;
>> +	pixm->colorspace = isp_sd_fmt.format.colorspace;
>> +	pixm->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
>> +	pixm->xfer_func = isp_sd_fmt.format.xfer_func;
>> +
>> +	/*
>> +	 * isp has a feature to set full range quantization for yuv formats.
> 
> How about "select between limited and full range for YUV formats"?
> 
>> +	 * so we need to get the format from the isp.
>> +	 */
>> +	pixm->quantization = isp_sd_fmt.format.quantization;
>> +	if (!v4l2_is_format_yuv(cap->pix.info))
>> +		pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>> +
>>   	pixm->width = clamp_t(u32, pixm->width,
>>   			      RKISP1_RSZ_SRC_MIN_WIDTH, max_widths[cap->id]);
>>   	pixm->height = clamp_t(u32, pixm->height,
>>   			       RKISP1_RSZ_SRC_MIN_HEIGHT, max_heights[cap->id]);
>>   
>> -	pixm->field = V4L2_FIELD_NONE;
>> -	pixm->colorspace = V4L2_COLORSPACE_DEFAULT;
>> -	pixm->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>> -
>>   	info = rkisp1_fill_pixfmt(pixm, cap->id);
>>   
>> -	/* can not change quantization when stream-on */
>> -	if (other_cap->is_streaming)
>> -		pixm->quantization = other_cap->pix.fmt.quantization;
>> -	/* output full range by default, take effect in params */
>> -	else if (!pixm->quantization ||
>> -		 pixm->quantization > V4L2_QUANTIZATION_LIM_RANGE)
>> -		pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>   
>>   	if (fmt_cfg)
>>   		*fmt_cfg = fmt;
>> diff --git a/drivers/staging/media/rkisp1/rkisp1-common.h b/drivers/staging/media/rkisp1/rkisp1-common.h
>> index 2d7b7e078636..7a5576fa14c9 100644
>> --- a/drivers/staging/media/rkisp1/rkisp1-common.h
>> +++ b/drivers/staging/media/rkisp1/rkisp1-common.h
>> @@ -300,6 +300,8 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
>>   			struct v4l2_device *v4l2_dev);
>>   void rkisp1_isp_unregister(struct rkisp1_device *rkisp1);
>>   
>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
>> +			   struct v4l2_subdev_format *sd_fmt);
>>   const struct rkisp1_isp_mbus_info *rkisp1_isp_mbus_info_get(u32 mbus_code);
>>   
>>   irqreturn_t rkisp1_isp_isr(struct rkisp1_device *rkisp1);
>> diff --git a/drivers/staging/media/rkisp1/rkisp1-isp.c b/drivers/staging/media/rkisp1/rkisp1-isp.c
>> index dee8e96f3900..6fdf5ed0b6b1 100644
>> --- a/drivers/staging/media/rkisp1/rkisp1-isp.c
>> +++ b/drivers/staging/media/rkisp1/rkisp1-isp.c
>> @@ -613,6 +613,10 @@ static int rkisp1_isp_enum_mbus_code(struct v4l2_subdev *sd,
>>   
>>   		if (code->index == pos - 1) {
>>   			code->code = fmt->mbus_code;
>> +			if (fmt->pixel_enc == V4L2_PIXEL_ENC_YUV &&
>> +			    dir == RKISP1_DIR_SRC)
>> +				code->flags =
>> +					V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION;
>>   			return 0;
>>   		}
>>   	}
>> @@ -639,12 +643,21 @@ static int rkisp1_isp_init_config(struct v4l2_subdev *sd,
>>   	sink_crop->height = RKISP1_DEFAULT_HEIGHT;
>>   	sink_crop->left = 0;
>>   	sink_crop->top = 0;
>> +	sink_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>> +	sink_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(sink_fmt->colorspace);
>> +	sink_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(sink_fmt->colorspace);
>> +	sink_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>> +
>>   
>>   	src_fmt = v4l2_subdev_get_try_format(sd, cfg,
>>   					     RKISP1_ISP_PAD_SOURCE_VIDEO);
>>   	*src_fmt = *sink_fmt;
>>   	src_fmt->code = RKISP1_DEF_SRC_PAD_FMT;
>> -	src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>> +	src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>> +	src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
>> +	src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
>> +	src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
>> +
>>   
>>   	src_crop = v4l2_subdev_get_try_crop(sd, cfg,
>>   					    RKISP1_ISP_PAD_SOURCE_VIDEO);
>> @@ -687,10 +700,17 @@ static void rkisp1_isp_set_src_fmt(struct rkisp1_isp *isp,
>>   		isp->src_fmt = mbus_info;
>>   	src_fmt->width  = src_crop->width;
>>   	src_fmt->height = src_crop->height;
>> -	src_fmt->quantization = format->quantization;
>> -	/* full range by default */
>> -	if (!src_fmt->quantization)
>> +
>> +	src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>> +	src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
>> +	src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
>> +
>> +	if (mbus_info->pixel_enc == V4L2_PIXEL_ENC_BAYER)
>>   		src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>> +	else if (format->quantization == V4L2_QUANTIZATION_DEFAULT)
>> +		src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
>> +	else
>> +		src_fmt->quantization = format->quantization;
>>   
>>   	*format = *src_fmt;
>>   }
>> @@ -1068,6 +1088,17 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
>>   	return ret;
>>   }
>>   
>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
>> +			   struct v4l2_subdev_format *sd_fmt)
>> +{
>> +	struct rkisp1_isp *isp = &rkisp1->isp;
>> +
>> +	sd_fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
>> +	sd_fmt->pad = RKISP1_ISP_PAD_SOURCE_VIDEO;
>> +
>> +	return v4l2_subdev_call(&isp->sd, pad, get_fmt, NULL, sd_fmt);
> 
> Do we need to get through the external API to access data that is
> driver-internal anyway?
> 
>> +}
>> +
>>   void rkisp1_isp_unregister(struct rkisp1_device *rkisp1)
>>   {
>>   	struct v4l2_subdev *sd = &rkisp1->isp.sd;
>> diff --git a/drivers/staging/media/rkisp1/rkisp1-resizer.c b/drivers/staging/media/rkisp1/rkisp1-resizer.c
>> index 7b6b7ddd4169..8705b133de68 100644
>> --- a/drivers/staging/media/rkisp1/rkisp1-resizer.c
>> +++ b/drivers/staging/media/rkisp1/rkisp1-resizer.c
>> @@ -525,6 +525,7 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
>>   	const struct rkisp1_isp_mbus_info *mbus_info;
>>   	struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
>>   	struct v4l2_rect *sink_crop;
>> +	struct v4l2_subdev_format isp_sd_fmt;
>>   
>>   	sink_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SINK, which);
>>   	src_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SRC, which);
>> @@ -539,8 +540,20 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
>>   	if (which == V4L2_SUBDEV_FORMAT_ACTIVE)
>>   		rsz->pixel_enc = mbus_info->pixel_enc;
>>   
>> +	rkisp1_get_isp_src_fmt(rsz->rkisp1, &isp_sd_fmt);
>> +
> 
> Is this necessary? My understanding was that in the subdev model, it was
> the userspace responsibility to propagate any configuration changes through
> the graph.
> 
> Also, doing this only here wouldn't fully maintain the
> consistency of the state. For example, if one sets the ISP subdev format
> first, then the resizer subdev and then the ISP subdev again, wouldn't the
> resizer subdev end up with a wrong format?

yes, this is indeed a bug, I am preparing v4 now.
What I thought to do is adding quantization conversion
support also on ther resizer and capture entities.
Then in the 'link_validation' callbacks, there
is a validation that the quantization fields matches.

Thanks,
Dafna

> 
> Best regards,
> Tomasz
> 
>> +	sink_fmt->field = isp_sd_fmt.format.field;
>> +	sink_fmt->colorspace = isp_sd_fmt.format.colorspace;
>> +	sink_fmt->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
>> +	sink_fmt->xfer_func = isp_sd_fmt.format.xfer_func;
>> +
>>   	/* Propagete to source pad */
>>   	src_fmt->code = sink_fmt->code;
>> +	src_fmt->field = sink_fmt->field;
>> +	src_fmt->colorspace = sink_fmt->colorspace;
>> +	src_fmt->ycbcr_enc = sink_fmt->ycbcr_enc;
>> +	src_fmt->xfer_func = sink_fmt->xfer_func;
>> +	src_fmt->quantization = sink_fmt->quantization;
>>   
>>   	sink_fmt->width = clamp_t(u32, format->width,
>>   				  rsz->config->min_rsz_width,
>> -- 
>> 2.17.1
>>

  reply index

Thread overview: 41+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-04-16 14:56 [RFC v3 1/2] v4l2: add support for colorspace conversion for video capture Dafna Hirschfeld
2020-04-16 14:56 ` [RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad Dafna Hirschfeld
2020-06-04 17:54   ` Tomasz Figa
2020-06-04 19:11     ` Dafna Hirschfeld [this message]
2020-06-10 12:08       ` Tomasz Figa
2020-06-18 17:27         ` Tomasz Figa
2020-06-18 18:26           ` Dafna Hirschfeld
2020-06-18 18:49             ` Tomasz Figa
2020-06-18 18:54               ` Dafna Hirschfeld
2020-06-18 19:25                 ` Tomasz Figa
2020-06-26  0:15                   ` Helen Koike
2020-06-28 13:10                     ` Tomasz Figa
2020-06-28 13:15                       ` Laurent Pinchart
2020-06-28 14:59                         ` Tomasz Figa
2020-06-28 19:34                           ` Dafna Hirschfeld
2020-06-29 15:50                             ` Helen Koike
2020-06-29 23:08                               ` Laurent Pinchart
2020-06-30  9:36                                 ` Dafna Hirschfeld
2020-07-01 12:52                                   ` Laurent Pinchart
2020-07-01 14:41                                     ` Dafna Hirschfeld
2020-07-02 18:43                                       ` Laurent Pinchart
2020-06-28 20:27                           ` Laurent Pinchart
2020-06-28 21:33                             ` Tomasz Figa
2020-06-28 21:37                               ` Laurent Pinchart
2020-06-29 12:08                                 ` Dafna Hirschfeld
2020-06-29 12:21                                   ` Laurent Pinchart
2020-06-29 12:29                                     ` Dafna Hirschfeld
2020-06-29 12:33                                       ` Laurent Pinchart
2020-06-26  0:01             ` Helen Koike
2020-05-07 13:00 ` [RFC v3 1/2] v4l2: add support for colorspace conversion for video capture Hans Verkuil
2020-05-25 13:56   ` Dafna Hirschfeld
2020-05-25 14:25     ` Hans Verkuil
2020-05-26 12:36       ` Dafna Hirschfeld
2020-05-26 13:47         ` Hans Verkuil
2020-06-04 17:39 ` Tomasz Figa
2020-06-05 10:11   ` Dafna Hirschfeld
2020-06-10 13:34     ` Tomasz Figa
2020-06-30 16:11       ` Dafna Hirschfeld
2020-06-30 16:31         ` Tomasz Figa
2020-06-30 17:43           ` Laurent Pinchart
2020-06-30 18:02             ` Tomasz Figa

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=87f11ebf-aecf-7eee-f18c-98dd4e8d692b@collabora.com \
    --to=dafna.hirschfeld@collabora.com \
    --cc=dafna3@gmail.com \
    --cc=ezequiel@collabora.com \
    --cc=helen.koike@collabora.com \
    --cc=hverkuil@xs4all.nl \
    --cc=kernel@collabora.com \
    --cc=laurent.pinchart@ideasonboard.com \
    --cc=linux-media@vger.kernel.org \
    --cc=linux-rockchip@lists.infradead.org \
    --cc=sakari.ailus@linux.intel.com \
    --cc=tfiga@chromium.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-Media Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-media/0 linux-media/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-media linux-media/ https://lore.kernel.org/linux-media \
		linux-media@vger.kernel.org
	public-inbox-index linux-media

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-media


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git