From: Helen Koike <helen.koike@collabora.com>
To: Tomasz Figa <tfiga@chromium.org>,
Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
Cc: Linux Media Mailing List <linux-media@vger.kernel.org>,
Ezequiel Garcia <ezequiel@collabora.com>,
Hans Verkuil <hverkuil@xs4all.nl>,
kernel@collabora.com, Dafna Hirschfeld <dafna3@gmail.com>,
Laurent Pinchart <laurent.pinchart@ideasonboard.com>,
"open list:ARM/Rockchip SoC..."
<linux-rockchip@lists.infradead.org>,
Sakari Ailus <sakari.ailus@linux.intel.com>
Subject: Re: [RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad
Date: Thu, 25 Jun 2020 21:15:28 -0300 [thread overview]
Message-ID: <67cdd3be-ee2a-3b88-5d17-d9d1d17b2603@collabora.com> (raw)
In-Reply-To: <20200618192535.GD73379@chromium.org>
On 6/18/20 4:25 PM, Tomasz Figa wrote:
> On Thu, Jun 18, 2020 at 08:54:34PM +0200, Dafna Hirschfeld wrote:
>>
>>
>> On 18.06.20 20:49, Tomasz Figa wrote:
>>> On Thu, Jun 18, 2020 at 08:26:33PM +0200, Dafna Hirschfeld wrote:
>>>>
>>>>
>>>> On 18.06.20 19:27, Tomasz Figa wrote:
>>>>> On Wed, Jun 10, 2020 at 2:08 PM Tomasz Figa <tfiga@chromium.org> wrote:
>>>>>>
>>>>>> On Thu, Jun 4, 2020 at 9:11 PM Dafna Hirschfeld
>>>>>> <dafna.hirschfeld@collabora.com> wrote:
>>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> On 04.06.20 19:54, Tomasz Figa wrote:
>>>>>>>> On Thu, Apr 16, 2020 at 04:56:05PM +0200, Dafna Hirschfeld wrote:
>>>>>>>>> The isp entity has a hardware support to force full range quantization
>>>>>>>>> for YUV formats. Use the new API to indicate userspace that
>>>>>>>>> quantization conversion is supported by adding the flag
>>>>>>>>> V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION during media code enumeration.
>>>>>>>>> Then uppon s_fmt on the video source pad, we assign the
>>>>>>>>> quantization from userspace for YUV formats.
>>>>>>>>> Also in the capture and resizer entities we retrieve the colorspace
>>>>>>>>> from the isp entity.
>>>>>>>>>
>>>>>>>>> Signed-off-by: Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
>>>>>>>>> ---
>>>>>>>>> drivers/staging/media/rkisp1/rkisp1-capture.c | 28 ++++++-------
>>>>>>>>> drivers/staging/media/rkisp1/rkisp1-common.h | 2 +
>>>>>>>>> drivers/staging/media/rkisp1/rkisp1-isp.c | 39 +++++++++++++++++--
>>>>>>>>> drivers/staging/media/rkisp1/rkisp1-resizer.c | 13 +++++++
>>>>>>>>> 4 files changed, 65 insertions(+), 17 deletions(-)
>>>>>>>>>
>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>>>>> index fbf62399fe3d..aca0f93bc772 100644
>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-capture.c
>>>>>>>>> @@ -1066,14 +1066,13 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
>>>>>>>>> const struct v4l2_format_info **fmt_info)
>>>>>>>>> {
>>>>>>>>> const struct rkisp1_capture_config *config = cap->config;
>>>>>>>>> - struct rkisp1_capture *other_cap =
>>>>>>>>> - &cap->rkisp1->capture_devs[cap->id ^ 1];
>>>>>>>>> const struct rkisp1_capture_fmt_cfg *fmt;
>>>>>>>>> const struct v4l2_format_info *info;
>>>>>>>>> const unsigned int max_widths[] = { RKISP1_RSZ_MP_SRC_MAX_WIDTH,
>>>>>>>>> RKISP1_RSZ_SP_SRC_MAX_WIDTH };
>>>>>>>>> const unsigned int max_heights[] = { RKISP1_RSZ_MP_SRC_MAX_HEIGHT,
>>>>>>>>> RKISP1_RSZ_SP_SRC_MAX_HEIGHT};
>>>>>>>>> + struct v4l2_subdev_format isp_sd_fmt;
>>>>>>>>>
>>>>>>>>> fmt = rkisp1_find_fmt_cfg(cap, pixm->pixelformat);
>>>>>>>>> if (!fmt) {
>>>>>>>>> @@ -1081,24 +1080,27 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap,
>>>>>>>>> pixm->pixelformat = fmt->fourcc;
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> + rkisp1_get_isp_src_fmt(cap->rkisp1, &isp_sd_fmt);
>>>>>>>>> + pixm->field = isp_sd_fmt.format.field;
>>>>>>>>> + pixm->colorspace = isp_sd_fmt.format.colorspace;
>>>>>>>>> + pixm->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc;
>>>>>>>>> + pixm->xfer_func = isp_sd_fmt.format.xfer_func;
>>>>>>>>> +
>>>>>>>>> + /*
>>>>>>>>> + * isp has a feature to set full range quantization for yuv formats.
>>>>>>>>
>>>>>>>> How about "select between limited and full range for YUV formats"?
>>>>>>>>
>>>>>>>>> + * so we need to get the format from the isp.
>>>>>>>>> + */
>>>>>>>>> + pixm->quantization = isp_sd_fmt.format.quantization;
>>>>>>>>> + if (!v4l2_is_format_yuv(cap->pix.info))
>>>>>>>>> + pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>> +
>>>>>>>>> pixm->width = clamp_t(u32, pixm->width,
>>>>>>>>> RKISP1_RSZ_SRC_MIN_WIDTH, max_widths[cap->id]);
>>>>>>>>> pixm->height = clamp_t(u32, pixm->height,
>>>>>>>>> RKISP1_RSZ_SRC_MIN_HEIGHT, max_heights[cap->id]);
>>>>>>>>>
>>>>>>>>> - pixm->field = V4L2_FIELD_NONE;
>>>>>>>>> - pixm->colorspace = V4L2_COLORSPACE_DEFAULT;
>>>>>>>>> - pixm->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
>>>>>>>>> -
>>>>>>>>> info = rkisp1_fill_pixfmt(pixm, cap->id);
>>>>>>>>>
>>>>>>>>> - /* can not change quantization when stream-on */
>>>>>>>>> - if (other_cap->is_streaming)
>>>>>>>>> - pixm->quantization = other_cap->pix.fmt.quantization;
>>>>>>>>> - /* output full range by default, take effect in params */
>>>>>>>>> - else if (!pixm->quantization ||
>>>>>>>>> - pixm->quantization > V4L2_QUANTIZATION_LIM_RANGE)
>>>>>>>>> - pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>>
>>>>>>>>> if (fmt_cfg)
>>>>>>>>> *fmt_cfg = fmt;
>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-common.h b/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>>>>> index 2d7b7e078636..7a5576fa14c9 100644
>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-common.h
>>>>>>>>> @@ -300,6 +300,8 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
>>>>>>>>> struct v4l2_device *v4l2_dev);
>>>>>>>>> void rkisp1_isp_unregister(struct rkisp1_device *rkisp1);
>>>>>>>>>
>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
>>>>>>>>> + struct v4l2_subdev_format *sd_fmt);
>>>>>>>>> const struct rkisp1_isp_mbus_info *rkisp1_isp_mbus_info_get(u32 mbus_code);
>>>>>>>>>
>>>>>>>>> irqreturn_t rkisp1_isp_isr(struct rkisp1_device *rkisp1);
>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-isp.c b/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>>>>> index dee8e96f3900..6fdf5ed0b6b1 100644
>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-isp.c
>>>>>>>>> @@ -613,6 +613,10 @@ static int rkisp1_isp_enum_mbus_code(struct v4l2_subdev *sd,
>>>>>>>>>
>>>>>>>>> if (code->index == pos - 1) {
>>>>>>>>> code->code = fmt->mbus_code;
>>>>>>>>> + if (fmt->pixel_enc == V4L2_PIXEL_ENC_YUV &&
>>>>>>>>> + dir == RKISP1_DIR_SRC)
>>>>>>>>> + code->flags =
>>>>>>>>> + V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION;
>>>>>>>>> return 0;
>>>>>>>>> }
>>>>>>>>> }
>>>>>>>>> @@ -639,12 +643,21 @@ static int rkisp1_isp_init_config(struct v4l2_subdev *sd,
>>>>>>>>> sink_crop->height = RKISP1_DEFAULT_HEIGHT;
>>>>>>>>> sink_crop->left = 0;
>>>>>>>>> sink_crop->top = 0;
>>>>>>>>> + sink_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>>>>> + sink_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(sink_fmt->colorspace);
>>>>>>>>> + sink_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(sink_fmt->colorspace);
>>>>>>>>> + sink_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>> +
>>>>>>>>>
>>>>>>>>> src_fmt = v4l2_subdev_get_try_format(sd, cfg,
>>>>>>>>> RKISP1_ISP_PAD_SOURCE_VIDEO);
>>>>>>>>> *src_fmt = *sink_fmt;
>>>>>>>>> src_fmt->code = RKISP1_DEF_SRC_PAD_FMT;
>>>>>>>>> - src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>> + src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>>>>> + src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
>>>>>>>>> + src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
>>>>>>>>> + src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
>>>>>>>>> +
>>>>>>>>>
>>>>>>>>> src_crop = v4l2_subdev_get_try_crop(sd, cfg,
>>>>>>>>> RKISP1_ISP_PAD_SOURCE_VIDEO);
>>>>>>>>> @@ -687,10 +700,17 @@ static void rkisp1_isp_set_src_fmt(struct rkisp1_isp *isp,
>>>>>>>>> isp->src_fmt = mbus_info;
>>>>>>>>> src_fmt->width = src_crop->width;
>>>>>>>>> src_fmt->height = src_crop->height;
>>>>>>>>> - src_fmt->quantization = format->quantization;
>>>>>>>>> - /* full range by default */
>>>>>>>>> - if (!src_fmt->quantization)
>>>>>>>>> +
>>>>>>>>> + src_fmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>>>>> + src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace);
>>>>>>>>> + src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace);
>>>>>>>>> +
>>>>>>>>> + if (mbus_info->pixel_enc == V4L2_PIXEL_ENC_BAYER)
>>>>>>>>> src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE;
>>>>>>>>> + else if (format->quantization == V4L2_QUANTIZATION_DEFAULT)
>>>>>>>>> + src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE;
>>>>>>>>> + else
>>>>>>>>> + src_fmt->quantization = format->quantization;
>>>>>>>>>
>>>>>>>>> *format = *src_fmt;
>>>>>>>>> }
>>>>>>>>> @@ -1068,6 +1088,17 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1,
>>>>>>>>> return ret;
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1,
>>>>>>>>> + struct v4l2_subdev_format *sd_fmt)
>>>>>>>>> +{
>>>>>>>>> + struct rkisp1_isp *isp = &rkisp1->isp;
>>>>>>>>> +
>>>>>>>>> + sd_fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
>>>>>>>>> + sd_fmt->pad = RKISP1_ISP_PAD_SOURCE_VIDEO;
>>>>>>>>> +
>>>>>>>>> + return v4l2_subdev_call(&isp->sd, pad, get_fmt, NULL, sd_fmt);
>>>>>>>>
>>>>>>>> Do we need to get through the external API to access data that is
>>>>>>>> driver-internal anyway?
>>>>>>>>
>>>>>>>>> +}
>>>>>>>>> +
>>>>>>>>> void rkisp1_isp_unregister(struct rkisp1_device *rkisp1)
>>>>>>>>> {
>>>>>>>>> struct v4l2_subdev *sd = &rkisp1->isp.sd;
>>>>>>>>> diff --git a/drivers/staging/media/rkisp1/rkisp1-resizer.c b/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>>>>> index 7b6b7ddd4169..8705b133de68 100644
>>>>>>>>> --- a/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>>>>> +++ b/drivers/staging/media/rkisp1/rkisp1-resizer.c
>>>>>>>>> @@ -525,6 +525,7 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
>>>>>>>>> const struct rkisp1_isp_mbus_info *mbus_info;
>>>>>>>>> struct v4l2_mbus_framefmt *sink_fmt, *src_fmt;
>>>>>>>>> struct v4l2_rect *sink_crop;
>>>>>>>>> + struct v4l2_subdev_format isp_sd_fmt;
>>>>>>>>>
>>>>>>>>> sink_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SINK, which);
>>>>>>>>> src_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SRC, which);
>>>>>>>>> @@ -539,8 +540,20 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz,
>>>>>>>>> if (which == V4L2_SUBDEV_FORMAT_ACTIVE)
>>>>>>>>> rsz->pixel_enc = mbus_info->pixel_enc;
>>>>>>>>>
>>>>>>>>> + rkisp1_get_isp_src_fmt(rsz->rkisp1, &isp_sd_fmt);
>>>>>>>>> +
>>>>>>>>
>>>>>>>> Is this necessary? My understanding was that in the subdev model, it was
>>>>>>>> the userspace responsibility to propagate any configuration changes through
>>>>>>>> the graph.
>>>>>>>>
>>>>>>>> Also, doing this only here wouldn't fully maintain the
>>>>>>>> consistency of the state. For example, if one sets the ISP subdev format
>>>>>>>> first, then the resizer subdev and then the ISP subdev again, wouldn't the
>>>>>>>> resizer subdev end up with a wrong format?
>>>>>>>
>>>>>>> yes, this is indeed a bug, I am preparing v4 now.
>>>>>>> What I thought to do is adding quantization conversion
>>>>>>> support also on ther resizer and capture entities.
>>>>>>> Then in the 'link_validation' callbacks, there
>>>>>>> is a validation that the quantization fields matches.
>>>>>>
>>>>>> My understanding is that, if we have the following topology
>>>>>>
>>>>>> [ ISP ] -> [ Resizer ] -> [ Video node ]
>>>>>>
>>>>>> then the ISP source pad would have the csc capability, while resizer
>>>>>> and video node would just accept whatever is configured on their sink
>>>>>> pads (no need for csc capability for that) and propagate that to their
>>>>>> outputs, i.e. resizer source pad and video node CAPTURE format.
>>>>>>
>>>>>> Is this what you were going to implement?
>>>> Hi, I sent a v4 where the CSC capability is set on the reiszer and capture as well.
>>>> I can send a v5 implementing it the way you suggest. Currently the doc says that the colorspace fields
>>>> must be set by the driver for capture streams. This implies that userspace can set it
>>>> only if the CSC is supported.
>>>
>>> Why would the userspace have to set it for the capture stream on the
>>> resizer and video nodes? Couldn't the userspace set it to DEFAULT and
>>> then the driver override to whatever it received on the corresponding
>>> sink?
According to the documentation [1]:
"the default quantization encoding as defined by the colorspace"
Then, for instance, the default for V4L2_COLORSPACE_SRGB is limited range [2].
Unless we change the definition of DEFAULT to let it accept whatever it receives on the corresponding sink.
Then the way I understand is:
[sensor]->[isp]->[resizer]->[capture]
Userspace should set quantization on:
* sensor source pad
* isp sink pad
* resizer sink pad
* capture node
The remaining source pads would get values propagated from their sink pads.
[1] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-defs.html#c.v4l2_quantization
[2] https://linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/colorspaces-details.html#col-srgb
Regards,
Helen
>> What do you mean "received on the corresponding sink"? Received from whom?
>>
>
> The resizer entity has both a sink and a source, right?
>
> The userspace is responsible for setting the right colorspace on the
> resizer sink.
>
> The driver is responsible for propagating that colorspace information to
> the resizer source. Similarly, it is also responsible for updating the
> state of the corresponding video node>
> Best regards,
> Tomasz
>
next prev parent reply other threads:[~2020-06-26 0:15 UTC|newest]
Thread overview: 44+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-04-16 14:56 [RFC v3 1/2] v4l2: add support for colorspace conversion for video capture Dafna Hirschfeld
2020-04-16 14:56 ` [RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad Dafna Hirschfeld
2020-06-04 17:54 ` Tomasz Figa
2020-06-04 19:11 ` Dafna Hirschfeld
2020-06-10 12:08 ` Tomasz Figa
2020-06-18 17:27 ` Tomasz Figa
2020-06-18 18:26 ` Dafna Hirschfeld
2020-06-18 18:49 ` Tomasz Figa
2020-06-18 18:54 ` Dafna Hirschfeld
2020-06-18 19:25 ` Tomasz Figa
2020-06-26 0:15 ` Helen Koike [this message]
2020-06-28 13:10 ` Tomasz Figa
2020-06-28 13:15 ` Laurent Pinchart
2020-06-28 14:59 ` Tomasz Figa
2020-06-28 19:34 ` Dafna Hirschfeld
2020-06-29 15:50 ` Helen Koike
2020-06-29 23:08 ` Laurent Pinchart
2020-06-30 9:36 ` Dafna Hirschfeld
2020-07-01 12:52 ` Laurent Pinchart
2020-07-01 14:41 ` Dafna Hirschfeld
2020-07-02 18:43 ` Laurent Pinchart
2020-08-10 12:33 ` Dafna Hirschfeld
2020-08-10 13:15 ` Dafna Hirschfeld
2020-08-10 13:29 ` Laurent Pinchart
2020-06-28 20:27 ` Laurent Pinchart
2020-06-28 21:33 ` Tomasz Figa
2020-06-28 21:37 ` Laurent Pinchart
2020-06-29 12:08 ` Dafna Hirschfeld
2020-06-29 12:21 ` Laurent Pinchart
2020-06-29 12:29 ` Dafna Hirschfeld
2020-06-29 12:33 ` Laurent Pinchart
2020-06-26 0:01 ` Helen Koike
2020-05-07 13:00 ` [RFC v3 1/2] v4l2: add support for colorspace conversion for video capture Hans Verkuil
2020-05-25 13:56 ` Dafna Hirschfeld
2020-05-25 14:25 ` Hans Verkuil
2020-05-26 12:36 ` Dafna Hirschfeld
2020-05-26 13:47 ` Hans Verkuil
2020-06-04 17:39 ` Tomasz Figa
2020-06-05 10:11 ` Dafna Hirschfeld
2020-06-10 13:34 ` Tomasz Figa
2020-06-30 16:11 ` Dafna Hirschfeld
2020-06-30 16:31 ` Tomasz Figa
2020-06-30 17:43 ` Laurent Pinchart
2020-06-30 18:02 ` Tomasz Figa
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=67cdd3be-ee2a-3b88-5d17-d9d1d17b2603@collabora.com \
--to=helen.koike@collabora.com \
--cc=dafna.hirschfeld@collabora.com \
--cc=dafna3@gmail.com \
--cc=ezequiel@collabora.com \
--cc=hverkuil@xs4all.nl \
--cc=kernel@collabora.com \
--cc=laurent.pinchart@ideasonboard.com \
--cc=linux-media@vger.kernel.org \
--cc=linux-rockchip@lists.infradead.org \
--cc=sakari.ailus@linux.intel.com \
--cc=tfiga@chromium.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).